Goodbye, Prettify. Hello highlight.js! Swapping out our Syntax Highlighter

Question

Update 2020-09-24

This is now live network-wide.

Update

This is now live on Meta Stack Exchange and Meta Stack Overflow. Any bugs and feedback can be posted here as an answer.

---

I’m Ben and I’m a dev on the Teams team here at Stack Overflow - we're the team focused on building the private Teams experience on SO. I’ve recently been working on our post editing experience and I’d like to show off some preliminary work that’s coming to the network soon.

TL;DR

We’re switching our code block highlighting library from Google Prettify to highlight.js. All your favorite languages are still supported and you won’t need to change how you write posts at all. The only major change is how we render highlighted code blocks. In addition, we’re taking this opportunity to introduce our new highlighting theme as well. We’re rolling this out in stages, starting with MSE/MSO with other sites to follow. (See the FAQ at the bottom of this post for dates)

Some history on Prettify / code block highlighting

I tried to do some digging on when we first adopted Prettify, but it seems that its history goes allll the way back to site’s earliest days. The earliest reference I could find was from back in ‘08. I asked around internally too and the best answers I could get were along the lines of:

¯\_(ツ)_/¯ - Everyone

Ask Atwood - Dean

and

If I had to guess, it was something along the lines of "there's not a lot of options, and this is used by Google so it's probably fine" - Kevin

Eventually the wonderful Tim Post pointed me to Stack Overflow Podcast #11, aired June 2008, where Jeff and Joel talk about how incredible it was for the time and how Google uses it themselves for syntax highlighting in Google Code (RIP). They also put out a call for alternatives, which I’d have to assume came up short.

Why the change?

Google Prettify hasn't been under active development for a while, and was officially discontinued by Google in April, as you all have let us know repeatedly. This means that no new language syntaxes¹ are being supported and that existing language syntaxes aren’t getting updated to support all their new features. It’s time to move on to something that supports modern front-end workflows (such as providing an npm package, for starters) and continues to evolve to meet the needs of developers.

What’s changing about how I write posts?

Absolutely nothing :). There is absolutely no change to how posts are written. We still support all the Prettify language aliases you know and love, along with the new aliases from highlight.js. However, we are not adding support for any new languages at this time, instead choosing to keep the initial changeset simple and aiming for current feature parity instead. All the current markdown syntax is still supported, along with determining code highlighting from tags and site defaults.

So what is changing?

The “only” changes are visual. We are updating the client-side code block renderer that styles your code in posts (Questions, Answers, etc) and in the editor preview. Syntax autodetection when a language is not specified should be much better overall, along with syntax highlighting coverage in general. The biggest outward facing change for the typical user is going to be our new theme (see below for details).

Why highlight.js? Why not…

Why did we pick highlight.js over Prettify? Well, first off, you asked for it specifically. More convincingly, it’s open source, actively maintained, and overall just a solid product.

We’re extremely concerned about performance here at SO (both on the client and on the server), so we needed to ensure that this major change on our hottest page on the site didn’t negatively impact our users. There was some prior investigation into highlight.js's performance back in 2016, but I figured we should give it another shot.

In our internal performance benchmarks highlight.js scored better than Prettify consistently across all browsers (except macOS Safari 13.1, where it was actually a bit slower)². It is a tad heavier than Prettify³, weighing in at an extra ~17kB (over the wire) after including all the languages we support across the network. This extra weight gain was acceptable to us as a tradeoff for what we were getting in return.

Why did we pick highlight.js over other contenders? Simply put, it was the best option that served our needs. We needed a library that we could easily control for use in the browser (deferred loading, theming specific elements), while also being simple to consume via a npm package, not needing specific build steps or a special babel plugin to pull in only the parts we need. Additionally, we could run it on the server (via Node.js) to unify our syntax highlighting in our Stacks documentation, giving us a single syntax highlighter across our products. Also a major plus was the ability to tokenize the highlighting result for use in our new editor (stay tuned!).

What are some potential drawbacks?

The most obvious not-quite-a-drawback is that language autodetection is different from Prettify. In general, it will be much more accurate, but will possibly end up with a different result that what Prettify would give us. This isn’t so much a bad thing, as it is just a thing that might take some getting used to if you’re a Prettify power user.

As mentioned earlier, the overall code bundle size is a bit bigger too. The vast majority of users wouldn’t even notice the change, which would only affect the first fetch since the browser will cache the file locally for subsequent hits anyways.

The last item is a bit of a personal preference. highlight.js tends to not highlight punctuation, which makes it a bit less colorful than other highlighters. This is considered a feature. Not a deal breaker by any means, but something I should mention regardless.

Designing the new theme

To offer some insight into how the new theme was designed, I reached out to the author, principal design systems designer Aaron Shekey.

Since we’re upgrading, we wanted to take this opportunity to design a Stack Overflow-flavored theme that takes advantage of newer tech like CSS variables that are aware of both light and dark modes. While we’ve improved it over the years, it’s highly likely that the current production theme simply used the stock colors provided by Prettify.

We’d need a theme that could work in both light and dark modes, was informed by Stack Overflow’s branded colors, and introduced a bit more contrast throughout.

Thankfully, we weren’t starting from scratch. When we built our Stacks documentation, we’d spent some time making our Jekyll theme display code snippets that got pretty close to accomplishing those goals. However, this was before dark mode was a thing, and we’d only built a single theme that assumed a fixed dark background. We’d have to extend this theme to light mode and revisit contrast along the way.

Using the Stacks documentation as a playground, we’ve now got themes in both light and dark modes that look like Stack Overflow and add or maintain contrast levels. We did our best to accomplish a contrast level of AAA, with a few variables dipping into AA. You can see the exact measurements commented in our colors constants file.

Here are a few screencaps of the new theme taken from my local development environment (click on the images to expand them). You can preview more languages (with an easy dark/light mode toggle) over at the Stacks docs.

Before

After

FAQ

• Q: When is the rollout happening?

  A: We're planning to roll this out to meta.stackexchange and meta.stackoverflow on Thursday, September 10th. Rollout to the rest of the network is scheduled for September 24th, after the initial testing period. This is a soft rollout date, dependent on any bugs/feedback we get from the community during the testing period.

• Q: What if I find a bug?

  A: Report bugs in an answer (one per answer) to this question. We'll keep this open for a couple/few weeks (until Friday, October 2nd) to address any immediate issues and then we'll update this post and ask you to post bugs as new questions after that time.

Footnotes

¹ I checked, plural of syntax is syntaxes. Take that spell-checker!

² Client-side benchmarks being what they are, we measured anywhere from ~49%-60% increase in the rate of ops/second depending on the machine and browser. Outliers being Safari 13.1 which had a ~29% decrease (favoring prettify) and Edge “legacy” scoring a ~279% increase over prettify!

³ Size comparisons were done comparing the prettify-full.en.js file taken from production vs the new highlight.pack.js bundle. Both were minified and served via a webpack-dev-server instance with the compress flag set (enabling gzip support). They were then included onto a regular html page with script tags and measured using the built-in browser dev tools. At the time of measurement, prettify landed at 23.3kB over the wire (meaning that the file was minified + gzipped) vs highlight.js at 40.7kB. This is a 17.4kB increase or about a ~74% increase in file size.

Answer 1

Can Stack Exchange please update to newer versions of Highlight.js on a regular cycle, rather than only on request?

As I said in my prior post reporting Google's discontinuation of Prettify, the process of filing bugs and feature requests with syntax highlighting would be quite drawn out and take a needlessly long time. The process was like this:

1. File a bug with Prettify, which would take 6-8 months to be resolved, if at all. (I filed a bug in 2014 and it still hadn't been resolved by the time Google put the project to rest.)
2. Once the request was resolved in Prettify, file another feature request here on Meta to request that SE update to the newer version. This would take the typical 6-8 week response time, but would often take longer than most requests because as far as I can best tell, they were only actioned when a developer happened to stumble on them.

As far as I can tell, Highlight.js is very actively maintained and requests with it are resolved fairly quickly, so #1 isn't an issue anymore (at least not in the current term). However, #2 will still remain an issue if SE sticks with their pre-existing model of only updating to newer versions on request.

Can Stack Exchange please actively update to newer Highlight.js versions on a regular cycle (not necessarily immediately after they're released, as I understand that'd be too onerous), rather than only updating to newer highlighter versions upon request? This would eliminate the problem in #2 and make the overall process significantly faster as one need only file the bug or feature request with Highlight.js and it'd be fixed in SE fairly quickly.

Answer 2

I found it rather frustrating that I couldn't easily see how the before/after pictures differed, so I did a bit of cutting and pasting so I could look at the before/after side by side to compare them more easily. Then it occurred to me that others might like to do the same, so feel free to have a look. Should be the same basic info as in the question, but arranged for more meaningful viewing.

First dark mode:

And then light mode:

Sorry, my cutting wasn't quite perfect, so (especially in light mode) you can see some dark lines that really shouldn't be there. But even if there's a little extra junk, at least you can do a real comparison so the changes are reasonably apparent.

To me, the new color scheme appears to have at least a couple of different general types of problems.

One is technical accuracy (i.e., accuracy in the tokenization itself). For example, looking at the Python example, if is in one color, and None in a different color (which appears to be the same color for 0, 1, and 0b101 and for someFunc and SomeClass). if and None are both keywords, so it would appear reasonable that they both be the same color. It doesn't seem reasonable or useful for two keywords to be in clearly different colors, and one of them in the same color as some identifiers and literal values.

Another is the choices of colors themselves. Generally speaking, for comfortable viewing we want to balance between two extremes. If there's too little difference between colors, it's not always clear whether two things are the same or different colors. When colors can't be distinguished easily, we lose much of the benefit of using coloring to start with.

At the same time, we don't want too much contrast, especially when two things are immediately adjacent to each other. If we do this, viewing simply becomes uncomfortable¹.

In this case, we see what may be some of the first problem. As previously mentioned, in the Python example, None, someFunc, SomeClass, 1, 0 and 0b101 are all shown in what looks like the same color. It's possible that this isn't really a parsing problem--maybe it's assigning a unique color to each, and they just happen to be so similar that we can't distinguish them.

The old color scheme also differentiates between the class name and the function name, where the new one seems to use the same color for both. Given that they're both syntactically identifiers, it's open to argument that this doesn't affect accuracy (as such), but it seems pretty clear to me that the old scheme is providing more useful information.

In the dark mode pictures, we see at least a few clear-cut cases of excessive contrast. The most obvious are the parameters (param1 and param2) shown in bright white against a deep-black background. In this case, we've pretty clearly gone beyond the level of contrast that most people can look at comfortably. As an aside, there are a few cases where it's a bit more reasonable to break or at least bend this rule a bit. For example, if you're coloring something with a very small area (e.g., a period or comma) you can often get away with a bit higher contrast than if the area were larger.

At least in my opinion, the light mode version of the new coloring fares at least somewhat better in this respect. We still have None colored to match the identifiers and literals, and mismatching if. On the other hand, the background in this case is a light grey, and the parameter names are in a somewhat darker grey, so the contrast range is considerably more manageable.

Given a wide audience, we'd also like the color schemes to work well for people with color impaired vision. The most common color vision impairment is called Deuteranomaly. If we run the pictures through a filter, we can see a simulation of approximately how these would look. For example, here's the light-mode Python code with simulated deuteranomaly vision:

Here we see that in the new color scheme, the comment is only barely distinguishable from the preceding code, and even less so from the literals (e.g., 'gre\'ater') It might not not so close that I'd consider it a clear failure in this regard, but it's enough to make me at least a little uncomfortable (and at least with respect to serving people with color vision deficiencies, pretty close to an outright failure).

The old color scheme is clearly superior in this regard--although contrast is certainly reduced in some cases, everything that started out as a separate color remains quite easily distinct.

There are, of course, other forms of color vision deficiency, up to and including truly complete color blindness. Fortunately, that's pretty rare. Deuteranomaly is the most common, and dealing well with it will frequently also work out well for most of the other somewhat less common cases (e.g., Protanomaly, Tritanomaly, etc.)

Unfortunately, it's fairly difficult to do automated testing of when colors have enough contrast for the difference to be easily visible. There are computations for "delta E" to tell you how much difference there is between two colors, but eyes are easily deceived, so (for example) the surroundings can make two areas with identical colors look obviously different, or areas with different colors difficult to distinguish. About the best we can hope for in a case like this (retrofitting to a system, affecting far too many pages to review each individually) is to get rid of obvious problems.

---

1. Now rarely relevant, but back in the days of CRTs you could get away with more in this regard, because individual pixels tended to have some degree of gradient at the edges, so even the brightest white against the darkest black still had at least some degree of gradient from one to the other. That's much less true with LCDs though, so we have to be more careful as the technology no longer covers for our mistakes.

Answer 3

I'd like to say that I appreciate this post.

It is clear, very informative, very detailled, and to me shows that person's concern for the community.

Of course, there will always be different opinion on the result ("I prefer the former highlighting" "I prefer the new one!") but that is inevitable.

I find the reasons to change (and the choice) compelling enough, and the resulting highliting is pleasing to the eye.

(I see some concerns about having several things showing up with the same color: this is inevitable. The highlighting is there to have successive part in a different color, thus making transitions visible, and the overall structure appear, and not to have everything with its own specific color)

Thank you, @ben-kelly, for the information

Answer 4

Current maintainer of Highlight.js here, though I'd add a few quick comments.

highlight.js tends to not highlight punctuation, which makes it a bit less colorful than other highlighters. This is considered a feature. Not a deal breaker by any means, but something I should mention regardless.

This is something I'm open to improving if someone wants to work on PRs and figure out a good way to go about handling this (work with existing themes, not be invasive, etc). https://github.com/highlightjs/highlight.js/issues/2500

I assure you that Mathematica Stack Exchange will be supported at launch. Due to the large size of the mma language definitions, the language is actually split out from the rest.

Some languages MIGHT also be possible to Highlight with a wildcard vs a list of EVERY keyword under the sun... I'm not sure if Mathematica would be one such language or not. Some of our languages are quite heavy because the keyword approach was just simpler (and more accurate). That said just breaking out the files and loading them (as needed) is probably the best solution for some of the less popular languages. And would also help with auto-detect speed.

For example, looking at the Python example, if is in one color, and None in a different color (which appears to be the same color for 0, 1, and 0b101 and for someFunc and SomeClass). if and None are both keywords,

We've always highlighted literals and keywords differently. For Python False, None, and True are currently defined as literals.

the first 5 inline comments are not parsed as comments at all.

Definitely a bug (and should be an easy fix), a GitHub issue would be appreciated. :-)

Language auto-detection for assembly language seems to be broken.

Auto-detect is on a "best effort" basis... the smaller the snippet the worse the auto-detect, but some languages are also much harder to auto-detect than others. If you really think there is an OBVIOUS issue (a huge snippet that is constantly flagged incorrectly, etc) then a GitHub issue would be more than welcome...

Different flavours of assembly language use different comment characters, so this is a somewhat thorny problem.

Indeed, and why have multiple assembly grammars, not just a single one. I have no idea if it would be possible to have a single grammar for exactly this reason.

Answer 5

bug status-bydesign

<!-- language-all: lang-none --> hint doesn't seem to work anymore

This post has a <!-- language-all: lang-none --> hint at the top of the post to prevent all the code blocks in it from being highlighted. I tried changing lang-none to none and it still didn't work. (As you say in your post, Prettify identifiers will still continue to work even after the change.)

We were told at the time of the CommonMark migration that <!-- language-all: [language] --> hints would continue to be supported, unlike the <!-- language: [language] --> syntax which was being deprecated.

This issue seems to be specific to the lang-none and none hints as part of this style of HTML comment; other ones seem to be working fine. As an example, this post contains such a comment to indicate C as the highlighting language, and the below snippet is highlighted in C:

#include <stdio.h>

(To test, I also changed the comment to indicate Python and it highlighted the above as Python.)

It seems to work for individual code blocks, using the code fence notation (i.e. ```none and ```lang-none):

#include <stdio.h>

In summary: <!-- language-all: lang-none --> and <!-- language-all: none --> don't seem to work to disable syntax highlighting for a particular post.

Answer 6

C is not supported

There is no C syntax highlighter in highlight.js. highlight.js uses the C++ highlighter for C, and it is a nightmare. It actually makes code harder to read than not having any highlighting. I saw a post on Stack Overflow wherein the same two tokens struct List are coloured in 3 different ways:

Yes, I've checked that lang-c is in use.

I presume there is some logic that detects that the clause starting with struct List is a declaration and then colours the entire line brown:

struct List *newnode = (struct List *)malloc(size * sizeof(struct List));

But this is not helpful in any way, and if you actually used a typedef List, then it would be coloured differently:

List *newnode = (struct List *)malloc(size * sizeof(struct List));

Every other C language highlighter I have seen colours token classes, context-free. For example the token struct, a keyword, should always have the same colour.

(Though, since in struct X, X is a tag, it could be distinguished from X that is a typedef, or a variable or function name)

Answer 7

😄 Thanks for doing this! I'm happy about this outcome, as I was a major proponent of switching to highlight.js back in 2016.

Great! …but what changed?

To satisfy my own curiosity, I'm wondering if you have an explanation or theory for what changed between 2016 and now to make the switch feasible. Oded's performance analysis seemed to raise some major issues, and your post indicates they are no longer issues, but I don't see an explanation for why things changed. For example:

Size in 2016:

It is [too big] … an extra 5kb minimum for millions and millions of requests a day … This size concern only grows with adding more languages.

Size now:

… an extra ~17kB (over the wire) after including all the languages we support across the network. This extra weight gain was acceptable to us as a tradeoff for what we were getting in return.

Speed in 2016:

… (don't forget - we have a highly nested DOM, and many "benchmarks" are done on a very simple page - which is not indicative of performance on Stack Overflow). … In my tests, CPU time for highlight.js was anything between two and four times higher than for prettify … I have also tested by using console.time around our highlighting calls - highlight.js consistently performed worse than prettify.

Speed now:

In our internal performance benchmarks highlight.js scored better than Prettify consistently …

Is this size difference acceptable now because of changes in browsers/networks/CDNs, or just because different people were making the decision? Surely the number of requests per day has only increased since 2016?

Do you have information about what performance tests Oded ran in 2016 and why your results now are so different? Is the internal performance testing infrastructure new? Have there been underlying technical changes to the "highly nested DOM" to make highlighting more efficient? Or have there been significant performance improvements in highlight.js itself?

Again, I'm glad the change was made now — I'd just like to know if there was a legitimate reason to wait 4 years and what changed in that time. Was there something we could have done differently to encourage adoption sooner?

Answer 8

What to do if highlight.js supports a language but Stack Exchange doesn't?

There's an entire Mathematica StackExchange, and so Mathematica syntax highlight is clearly very important to us. But when I did some digging to find the highlight.js bundle that SE seems to be serving, Mathematica isn't in the registerLanguage("...", ...) blocks, even though it is in the set of aliases that StackExchange seems to be defining, i.e. this line is in the bundle

StackExchange.highlightjs=function(){var e={..."mma":"mathematica",...} ...}

It's vaguely annoying to be forgotten by the company to whose platform we've contributed so much, of course, but it'd be way more annoying for our nice syntax highlighting to disappear all of a sudden. We've been getting by with custom support for Google Prettify as written by one of our mods. The lack of Mathematica support is extra confusing when we consider that highlight.js already supports it.

So...what's the protocol for adding highlighting for a language that Stack Exchange, the company, need do nothing extra to support, since highlight.js already has it.

Sample Code

For reference, the following block is fenced with lang-mathematica as the spec. As of when I write this, it renders un-highlighted.

pot =
  Evaluate@With[
     {
      n = 4,
      l = 1,
      c = .25,
      s = .075,
      scale = 4,
      broad = 5
      },
     scale*(JacobiP[n, l, l, #/broad] + .2 JacobiP[2, l, l, #/broad])*
       PDF[
        MixtureDistribution[
         {1, 1},
         {
          NormalDistribution[-c, s],
          NormalDistribution[c, s]
          }
         ], 
        #/broad
        ] - PDF[NormalDistribution[0, .35], #](*+(#/broad)^2*)
     ] &
(* Out: *)
-1.1398350868612364/E^(4.081632653061225*#1^2) + 4*(2.659615202676218/E^(88.8888888888889*(-0.25 + #1/5)^2) + 
    2.659615202676218/E^(88.8888888888889*(0.25 + #1/5)^2))*(5 + 0.2*(3 + (15*(-1 + #1/5))/2 + (15*(-1 + #1/5)^2)/4) + 
    35*(-1 + #1/5) + 70*(-1 + #1/5)^2 + (105*(-1 + #1/5)^3)/2 + (105*(-1 + #1/5)^4)/8) &

Answer 9

status-bydesign

Will the default code markdown be changed to code fences?

Currently if you click the code ({}) icon in the editor, then the selected text is still indented (or unindented) by 4 white spaces.

As the indent method's way of defining a language for a specific code block (<!-- language: python --> for example) is deprecated¹, then shouldn't the default functionality of the button be to wrap the code in a code fence (```) instead?

1

^{The former method of specifying a highlighting language can still be used for HTML code blocks: place an HTML comment <!-- language: lang-or-tag-here --> before the <pre><code> tags and it will work.}

^{Also, this former method hasn't been completely removed for four-space indented code blocks, but merely deprecated. While it will still work for the time being on four-space indented code blocks, it may/will be removed in the future.}

Answer 10

Apparently asm / assembly has never(?) been supported as a syntax-highlighting language, and the somewhat decent highlighting we got in the past was from auto-detection (presumably as some other language with # or ; comment characters.)

highlight.js auto-detection happens to produce way worse results for assembly than whatever prettify.js did, so in practice there is a real regression here.

Assembly language really doesn't need much highlighting to be readable; it's already syntactically simple and has a regular line format. But it does benefit significantly from fading comments into a colour that stands out less than the rest of the code.

The rest of this answer was written without realizing asm wasn't (ever?) supported; the highlight.js languages including x86asm are not enabled on Stack Overflow / SE, so of course using them doesn't help.

---

Language auto-detection for assembly language seems to be broken. For example, note the lack of highlighting in the question on Printing an integer as a string with AT&T syntax, with Linux system calls instead of printf. After editing my answer to use ```lang-assembly on the main code block, that block has highlighting but the others don't. (And does actually look decent.)

Separately, syntax highlighting for NASM (a different asm syntax that uses ; as the comment character) is (was?) broken. In Unexpected result of subtracting a NASM macro in an expression, ```lang-nasm or lang-assembly leads to a mess that's arguably worse than nothing, because of single-quote used as English punctuation in a comment. Same result with <!-- language: lang-assembly -->.

(Update: no longer as bad as a couple weeks ago. An apostrophe in comments seems to only affect the end of the contracted word, not the entire rest of the block. But this NASM syntax is block is still not very usefully highlighted, e.g. comments aren't grayed, and only the 0 in 0x.. is in red. At least it's not clearly or much worse than nothing. x86asm is listed in the supported languages and highlight.js's x86 asm highlighter is supposed to be for NASM syntax. x86asm results in no highlighting; I had to use lang-x86asm to get the current highlighting.)

section .rodata           ; groups read-only together inside the .text section
    msg: db "Thank you"
    var: db 0x34, 0x33, 0x32, 0x31   ; dd 0x31323334  ; Since you're passing these bytes to write(2), writing them separately is probably less confusing.  (x86 is little-endian, so they will come out "backwards")

    ;; apparently you want to include var as part of the message, for some reason
    msglen equ $ - msg    ; $ is the current position

Previously, this meta answer wasn't getting any syntax highlighting; that's changed now.

---

SO asm answers tend to be more heavily commented than you'd do in real life, because the target audience is people that don't understand the basics of asm. And SO code blocks are more cramped horizontally than a normal text editor so it encourages leaving comments closer to the end of the code, making it worse if they're visually harder to ignore. (Especially in some poorly formatted beginner questions and answers where comments are ragged and literally no space is left after instructions.)

---

Different flavours of assembly language use different comment characters, so this is a somewhat thorny problem. Some use # to decorate numeric literals (e.g. ARM), so treating ;, #, and @ as comment characters won't always work.

As discussed in comments, highlight.js has highlighters for a few different asm syntaxes, no generic asm.

By looking at tags like [arm] as well as [assembly], Stack Overflow should (in theory) be able to pick the right asm syntax highlighting.

For cases like explicit lang-asm overrides in the markdown (which isn't explicit enough: doesn't say which flavour), Stack Overflow could (in theory) still auto-detect which syntax to highlight for based on the ISA tag. e.g. for a post with [c] [x86] tags, a lang-asm block could still pick x86 highlighting.

Except that doesn't disambiguate MASM vs. NASM vs. [gnu-assembler] syntax, with GAS using a different comment character (#) than most other x86 assemblers (;). Many questions aren't tagged with a specific assembler syntax name, just x86. (Most non-x86 ISAs only have one syntax in wide usage; this is mostly an x86 problem.)

To make matters more complicated GAS .intel_syntax noprefix still uses GAS directive and comment character, same as when GAS is in AT&T syntax mode. So [att] syntax questions aren't the only ones where # is the right comment character, even if we could rely on all questions that happen to use AT&T syntax being tagged [att].

But unless / until that happens, I guess we should be tagging asm blocks with one of:

• lang-x86asm
• lang-armasm (I guess this is Keil's ARMASM for directive syntax, not GAS? Instruction syntax is the same between both, though.)
• lang-avrasm

I haven't dug into how Stack Overflow dispatches anything to those internally supported highlight.js things.

Answer 11

I've just tried the following piece of JavaScript code (from this answer of mine in Code Golf) because Google Prettify was not parsing the regular expression followed by an inline comment correctly. That's why I used alternate slash characters in the original post (I've turned them back into regular slash characters below).

But this is much worse now, as the first 5 inline comments are not parsed as comments at all.

f = (                // f is a recursive function taking:
  [c,                //   c   = next digit character
      ...a],         //   a[] = array of remaining digits
  o = '',            //   o   = output string
  S = new Set        //   S   = set of solutions
) =>                 //
  c ?                // if c is defined:
    f(               //   do a recursive call:
      a,             //     pass a[]
      o + c,         //     append c to o
      o ?            //     if o is non-empty:
        f(           //       do another recursive call
          a,         //         pass a[]
          o + [, c], //         append a comma followed by c to o
          S          //         pass S
        )            //       end of recursive call (returns S)
      :              //     else:
        S            //       just pass S as the 3rd argument
    )                //   end of recursive call (returns S)
  :                  // else:
    S.add(           //   add to the set S:
      o.replace(     //     the string o with ...
        /\d+/g,      //       ... all numeric strings
        n => +n      //       coerced to integers to remove leading zeros
                     //       (and coerced back to strings)
      )              //     end of replace()
    )                //   end of add() (returns S)

I'm sure this is going to be fixed quickly, so here's a picture of the current rendering for later reference. :-)

Answer 12

In LaTeX code, @ should be treated as a letter. There are any number of examples on tex.stackexchange, but

Undefined control sequence on \beamer@leftmargin indentation

\begin{frame}[fragile]{E}
\makeatletter
\hskip-\beamer@leftmargin
\makeatother
\lipsum[2]
\end{frame}

\beamer@leftmargin is a single token, but beamer is coloured and @leftmargin is left as unstyled text which makes the code very hard to read.

Technically, @ is not always a letter, but it is almost always a letter when appearing in code sections and is a far better default in a syntax highlighter.

Answer 13

There has been times that I've turned off code highlighting with <!-- language: lang-none --> because Prettify was getting it wrong, and no highlighting is better than wrong highlighting. (The example that comes to mind was a Bash snippet where # wasn't a comment indicator, but Prettify thought it was.) After this change goes through, should I go back over those posts and turn code highlighting on again? Is it better?

I suppose I can test it.

Answer 14

We have been waiting for Verilog and SystemVerilog (SV) highlighting for a long time. Apparently we will have Verilog support with highlight.js, but SV will continue to be unsupported. Still much better than before. I'm happy with the change and appreciate your effort.

Let me put some Verilog code (from highlight.js demo) here to see the result after the roll-out. I assume the language code will be lang-verilog.

EDIT: We haven't got Verilog support as Ben Kelly mentioned in the comments. The following snippet has no language code, thus we see the result of auto detection.

`timescale 1ns / 1ps

/**
 * counter: a generic clearable up-counter
 */

module counter
    #(parameter WIDTH=64, NAME="world")
    (
        input clk,
        input ce,
        input arst_n,
        output reg [WIDTH-1:0] q
    );
    
    string name = "counter";
    localparam val0 = 12'ha1f;
    localparam val1 = 12'h1fa;
    localparam val2 = 12'hfa1;

    // some child
    clock_buffer #(WIDTH) buffer_inst (
      .clk(clk),
      .ce(ce),
      .reset(arst_n)
    );

    // Simple gated up-counter with async clear

    always @(posedge clk or negedge arst_n) begin
        if (arst_n == 1'b0) begin
            q <= {WIDTH {1'b0}};
            end
        else begin
            q <= q;
            if (ce == 1'b1) begin
                q <= q + 1;
            end
        end
    end

    function int add_one(int x);
      return x + 1;
    endfunction : add_one

`ifdef SIMULATION
initial $display("Hello %s", NAME);
`endif
endmodule : counter

class my_data extends uvm_data;
  int x, y;

  function add_one();
    x++;
    y++;
  endfunction : add_one
endclass : my_data

Answer 15

Will lazy loading of syntaxes be supported?

This would allow for syntax highlighting of less common languages that don't need to be eager-loaded on every page load.

Here's a proof-of-concept that doubles as a Tampermonkey user-script:

Highlight.js lazy-loading proof of concept

Naturally it's a little hacky, but it works on all of the following examples:

```lang-diff

- print('failure')
+ print('success')

```lang-elixir

spawn_link(fn ->
  send(current_process, {:msg, "hello world"})
end)
  
receive do
  {:msg, contents} -> IO.puts(contents)
end

```lang-bf

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

Answer 16

status-completed

Visual Basic code is no longer highlighted

The error in the console is:

Could not find the language 'vb', did you forget to load/include a language module?
Falling back to no-highlight mode for this block. <pre class="lang-vb s-code-block">

Answer 17

Will the <!-- language: [language] --> hint be disappearing?

Back when SE was switching to CommonMark, we were told that the old <!-- language: [language] --> syntax had been deprecated and was subject to removal in the future. (Prior to the implementation of code fences, this was the proper syntax to force a single block of code to be highlighted differently from the rest of the post.) With this change, will that comment style be removed once this is rolled out to all the sites?

It seems to work fine at the moment. The following is specified as a C code block:

#include <stdio.h>

...and here's the same text, but as a Python code block:

#include <stdio.h>

Are there plans to remove it, or will it remain for the foreseeable future? If it is going to be removed, will it still be that posts rendered prior to its removal will still honor it until they're edited, as we were told at the time?

Answer 18

status-bydesign

PowerShell and batch syntax highlighting is off all around and neither works correctly (left: Stack Exchange; right: Microsoft's Visual Studio Code)

• It appears batch and PowerShell syntax have been linked to each other, which simply doesn't work for either due to the different ways variables and other characters are used between the two:
  • PowerShell comments use #, whereas batch uses ::
  • PowerShell variables use $, whereas batch uses %<variable>%
    
  • PowerShell doesn't support linking commands via &||&&, using ; instead, which batch doesn't support
• PowerShell syntax only syntax highlights if first letter of command or parameter is capitalized, leading to a ridiculous amount of posts not syntax highlighted unless edited, as it doesn't syntax highlight if the entire command/parameter is in all lowercase or all uppercase (latter also affecting batch), which it should since PowerShell isn't case aware
  • PowerShell and batch syntax highlights don't apply as they should when code fences are used (also an issue with other languages), regardless whether the syntax is specified after the code fence or not - the only way to reliably have it syntax highlighted is to use HTML syntax comment <!-- language-all: lang-powershell --> or lang-bat (it was also an issue with Prettify)

Answer 19

SQL Formatting issues

As I almost exclusively stick to SQL Server related tags, I've picked up on a few issues/features with the sql language formatting.

Hash character incorrectly interpreted as comment character in SQL

In the below example, on the first line, everything after the # in VIN# is a coloured as a comment. On the third line, everyting after the # in #TempTable is. This doesn't, however, occur within the literal string, does within brackets ([]) (used by T-SQL as a delimit identifier), and doesn't within double quotes (") (the ANSI SQL delimit identifier).

SELECT VIN#, NTT.fID, GETDATE(),
       SYSDATETIME()
FROM #TempTable TT
     JOIN dbo.NonTempTable NTT ON TT.ID = NTT.fID
WHERE Description = 'Hello#there' AND NTT.Val = 3
  AND [VIN#] > 7
   OR "VIN#" < -12;
--This is an actual single line comment
/* 
This is a
Multiline
Comment
*/

# isn't even a comment character in SQL. Single line comments are defined with -- and multiple with /* ... */.

This is actually quite a problem, especially when temporary objects start with a #, and are used frequently with DDL and DML examples.

---

Further edit

Brackets ([]) not treated as delimit identifier

In T-SQL (as stated above) Brackets ([]) are the default delimit identifier, rather that double quotes ("), which are the ANSI delimit Identifier.

If a key work is within brackets, it is highlighted incorrectly. For example:

SELECT [name]
FROM dbo.[Table] T
     JOIN dbo."VIEW" V ON T.ID = V.IDl

I did decide to check, and there isn't a T-SQL variant option:

SELECT [name]
FROM dbo.[Table] T
     JOIN dbo."VIEW" V ON T.ID = V.IDl

---

Another edit:

The @ character isn't recognised as a variable identifier

Variable names aren't highlighted, or "immune" to other highlighting. Variable names are prefixed with an @ in SQL. For example:

DECLARE @variable varchar(10),
        @Table table (ID int),
        @Date datetime2(0),
        @1 int,
        @NonReservedWord sysname;

Notice that all the variable names, apart from NonReservedWord, receive incorrect syntax highlighting.

Answer 20

status-bydesign ^{(Manually changing this from bug to status-bydesign given my discoveries documented below.)}

I searched around, but I couldn't find any previous posts referencing regular expressions.
Regular expressions are stated to be currently supported, but it is not in the list of languages supported by highlight.js (it was supported by Prettify).

There are some weird effects when highlighting complex expressions, e.g., from this answer:

(?:[a-z0-9!#$%&'*+\/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+\/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

It sometimes italicizes the characters between asterisks *, and other times fails to highlight character lists inside square brackets, [].

If it's not supported by highlight.js, where is this highlighting scheme even coming from?^{See update} Are regular expressions included in the FAQ list by mistake¹? I notice that the default highlighter for the regex tag on SO is lang-default rather than lang-regex.

---

Update

So I've done a little digging, and it appears what's really going on here is that the regular expression in this post is getting auto-recognized as Markdown, even when specified as regex.

Setting the identifier of the same snippet as lang-markdown has an identical effect as regex:

(?:[a-z0-9!#$%&'*+\/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+\/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

This leads into the discovery I made, which largely revolves around the last sentence of my original post:

I notice that the default highlighter for the regex tag on SO is lang-default rather than lang-regex.

As described in this post by @T.J.Crowder, and backed up by the help center, there is a difference between identifying a code block as lang-X vs. just X.

As per the help center (emphasis mine):

You can use either one of the supported language codes, like lang-cpp or lang-sql, or you can specify a tag, and the syntax highlighting language associated with this tag will be used.

This was news to me! I had been under the impression, which I'm sure many others are as well, that ID X was simply a shortcut to lang-X. This is incorrect.

Therefore, ID'ing a snippet as regex is really saying "identify this snippet as the defined identifier for regex". This happens to be lang-default, which is really a shortcut to tell the highlighter to "guess" what the correct highlight should be, which in this specific case, becomes Markdown.

So it's going regex ==> lang-default ==> lang-markdown.

Popping open the console to take a look at the first snippet here will still show class="lang-regex s-code-block hljs", even though it's getting highlighted as Markdown. I believe this is due to how highlight.js works. It appears it never actually changes the identifier class name itself, but rather injects the child syntax classes underneath it in regardless.

---

¹ - It looks like it was added back into the list in the FAQ post on Sept. 28 (Rev. 100), and given my discoveries below, the answer is yes, it is a mistake.

Answer 21

Does highlight.js support emphasis in blocks formatted as "code" (ie indented 4 spaces)?

Paraphrasing an MSE question:

In-code highlighting (anything will do), would be a great way to emphasise the important parts.

Currently, the best people can do is ASCII art arrows, eg:

printf("%5s", "foo")
         ^--- add a width value

which happens often enough and is probably not done more because it's a pain and ugly.

It would be great to be able to highlight (in this case) the 5 by making it red, bold, or whatever by surrounding it with some special chars, maybe like !5! or whatever works.

Awesome would be highlighting with a comment that isn't selected when copy-pasting the code block.

---

I'm putting my hand up to donate my time and coonsiderable software engineering skills to make this happen. Let me know when you have a github repo up, you've added me as a contributor, and you have a task management system up (eg Trello, please not jira!)

Answer 22

Syntax highlighting isn't always present in the entire code block

This is an odd one. I've noticed this in a few languages, not just SQL, but sometimes the highlighting just doesn't work on the entire code block. This appears to happen more when the code snippet isn't complete on its own (and so isn't valid syntax on its own).

Take the below SQL snippet for example:

SUM(CASE WHEN SIPCOD in ('001','500') or (SIPCOD = '013' and SISHCD = 'OTA')
         THEN 1
         ELSE 0
    END) -
SUM(CASE WHEN SIPCOD in ('501','502') and SIHRS >= 3.0
         THEN 0.5
         ELSE 0
    END) as [Days Worked]

Even with the language defined (both with sql or lang-sql), the first line to receive syntax highlighting is the fourth line (END) -); the prior lines have no highlighting. The image below is from SO Dark Theme:

I'll try and repro this with some other languages and edit it in, or if I see other examples (I'm sure I've seen at least one C# and PowerShell example over the weekend on my mobile).

This is SQL again. However, this one doesn't highlight the last line, for some reason:

IF EXISTS (SELECT 1 FROM [135.282.123.12].tempdb.sys.tables WHERE [name] = N'##Tmp1')
    PRINT N'YES';
ELSE
    PRINT N'No';

---

Apologies, this is SQL again, but the highlighting is all kinds of wrong in this code block. It starts, then suddenly stops, and then picks up again it the oddest place:

CREATE TABLE dbo.RealTable (ID int IDENTITY);
GO

DECLARE @SQL nvarchar(MAX);
--Good attempt
EXEC dbo.CreateNewColumn @TableName = N'RealTable',
                         @ColumnName = N'SomeString',
                         @sql_dtype = N'nvarchar',
                         @length = '255',
                         @SQL = @SQL OUTPUT;

PRINT @SQL;
--Another good attempt
EXEC dbo.CreateNewColumn @TableName = N'RealTable',
                         @ColumnName = N'SomeInt',
                         @sql_dtype = N'int',
                         @SQL = @SQL OUTPUT;

PRINT @SQL;
GO
DECLARE @SQL nvarchar(MAX);
--A bad attempt
EXEC dbo.CreateNewColumn @TableName = N'RealTable',
                         @ColumnName = N'AChar',
                         @sql_dtype = N'char',
                         @length = N'CREATE USER test WITHOUT LOGIN',
                         @SQL = @SQL OUTPUT;

PRINT @SQL;
GO
DECLARE @SQL nvarchar(MAX);
--Bad parameters
EXEC dbo.CreateNewColumn @TableName = N'RealTable',
                         @ColumnName = N'SomeNumeric',
                         @sql_dtype = N'decimal',
                         @length = 7, --This should be precision and scale
                         @SQL = @SQL OUTPUT;
GO
DECLARE @SQL nvarchar(MAX);
--Good parameters
EXEC dbo.CreateNewColumn @TableName = N'RealTable',
                         @ColumnName = N'SomeNumeric',
                         @sql_dtype = N'numeric',
                         @Precision = 7, --This should be precision and scale
                         @Scale = 2,
                         @SQL = @SQL OUTPUT;

SELECT *
FROM dbo.RealTable;
GO
DROP PROC dbo.CreateNewColumn
DROP TABLE dbo.RealTable

Answer 23

bug

Questions which do not have any tags associated with any languages do not get their code blocks automatically highlighted at all. Examples:

Preloader is not working on Angular universal SSR app

How to Get a List of Members in a Guild Discord.js

Note that questions will get highlighted if they have at least one tag with a "Highlight Language" in their wiki, even if that language is default - like with regex. Questions with at least one such tag will get their code blocks automatically highlighted. In contrast, a question only with tags like discord.js which has no highlight language (not even default) will not have any code blocks highlighted.

I think when no tags have languages associated with them, the question's code blocks should be highlighted automatically. Maybe remove the difference between the association with default highlighting and a non-existing language association while you're at it, unless it's needed for something. (Or give all tags a default language association.)

All questions should at least have something like

<div style="display:none" id="js-codeblock-lang">default</div>

but it should never be empty, or auto-highlighting won't work:

<div style="display:none" id="js-codeblock-lang"></div>

This issue is pretty similar to a related standalone question: Improving syntax highlighting language auto-detection.

Answer 24

Bash highlighting seems to be broken.

echo "$(true)"
echo $(true)

As you can see, the command in the first subshell is not highlighted, presumably due to the quotes, but the second is (no quotes). Both should be highlighted.

Adding a PNG image in case this gets fixed.

Answer 25

There is something a bit odd with PL/SQL (or SQL - I'm not sure whether PL/SQL is actually supported. It seems to be sadly unpopular with syntax highlighting plugins.)

A quoted SQL statement seems to defeat the quoting, but only when an earlier line ends with a semicolon.

select blah into blahblah from blahblahblah;  -- Semicolon here seems to do it

xxx := 'select select';

Quoting is now reversed.

Looking at other SQL-related issues, I see Syntax highlighting isn't always present in the entire code block also has an example where quoting is broken by a quote that includes a SQL keyword.

Screenshot for posterity:

The actual post where this came up is here: https://stackoverflow.com/a/64183788/230471

Edit: Marking as Lua seems to work better with quoting:

select blah into blahblah from blahblahblah;  -- This is a comment

xxx := 'select select';

Quoting is not reversed.

Answer 26

support

What are we supposed to do if syntax highlighting doesn't turn on at all?

In a question with a very simple code block, with only the c# tag, this is what I see:

For the block of code:

MapperConfiguration config = new MapperConfiguration( cfg => cfg.CreateMap<Source, Dest>()
    .ForMember( k => k.Sector, opt => opt.MapFrom<MyResolver>() ) );

Mapper.Initialize( config );

The only highlight is on new for some reason. The rest of the c# questions have the proper highlighting for me.
If it makes any difference, using latest Firefox on Windows and no console errors.

Answer 27

bug

I noticed that on this question the syntax highlighting for some C++ code stops partway thru.

In particular, it gets tripped up by this bit of code:

template <class T>
ostream& operator<< (ostream& os, const skg::Triplet<T>& p_t) ;
void other_stuff_that_isnt_colored();

If the operator is changed from << to something else, the coloring continues

template <class T>
ostream& operator+(ostream& os, const skg::Triplet<T>& p_t) ;
void other_stuff_that_is_colored();

but the color for the operator keyword is the identifier color, and not the keyword color.

If the template <class T> part is removed the coloring is correct.

ostream& operator<< (ostream& os, const skg::Triplet<T>& p_t) ;
void other_stuff_that_is_colored();

Answer 28

LaTeX highlight in TikZ environments is wrong.

Look at https://tex.stackexchange.com/a/564540/38080:

It seems that a newline in a macro arguments desynchronize the parser...

Thanks!

PS: could be this:https://github.com/highlightjs/highlight.js/issues/2709 ...

Answer 29

status-completed

No Objective-C highlighting

I commented to say how disappointing highlighting of Objective C is, but I was told to open a bug as it is not an issue of Highlight.js, but of Stack Overflow not applying it (it applies C highlighting instead, and with that what I see highlighted makes sense).

bug

Answer 30

status-completed

There are two problems with Groovy syntax highlighting:

1. The old syntax was not automatically converted to the new syntax, i.e. thousands of Groovy-related questions and answers lost syntax highlighting.

2. Groovy syntax highlighting via ```groovy does not work in many cases (e.g. here), only in some.

For details please read this question and its comments.