238

Update

This is now live on Meta Stack Exchange and Meta Stack Overflow. Any bugs and feedback can be posted here as an answer.


I’m Ben and I’m a dev on the Teams team here at Stack Overflow - we're the team focused on building the private Teams experience on SO. I’ve recently been working on our post editing experience and I’d like to show off some preliminary work that’s coming to the network soon.

TL;DR

We’re switching our code block highlighting library from Google Prettify to highlight.js. All your favorite languages are still supported and you won’t need to change how you write posts at all. The only major change is how we render highlighted code blocks. In addition, we’re taking this opportunity to introduce our new highlighting theme as well. We’re rolling this out in stages, starting with MSE/MSO with other sites to follow. (See the FAQ at the bottom of this post for dates)

Some history on Prettify / code block highlighting

I tried to do some digging on when we first adopted Prettify, but it seems that its history goes allll the way back to site’s earliest days. The earliest reference I could find was from back in ‘08. I asked around internally too and the best answers I could get were along the lines of:

¯\_(ツ)_/¯ - Everyone

Ask Atwood - Dean

and

If I had to guess, it was something along the lines of "there's not a lot of options, and this is used by Google so it's probably fine" - Kevin

Eventually the wonderful Tim Post pointed me to Stack Overflow Podcast #11, aired June 2008, where Jeff and Joel talk about how incredible it was for the time and how Google uses it themselves for syntax highlighting in Google Code (RIP). They also put out a call for alternatives, which I’d have to assume came up short.

Why the change?

Google Prettify hasn't been under active development for a while, and was officially discontinued by Google in April, as you all have let us know repeatedly. This means that no new language syntaxes1 are being supported and that existing language syntaxes aren’t getting updated to support all their new features. It’s time to move on to something that supports modern front-end workflows (such as providing an npm package, for starters) and continues to evolve to meet the needs of developers.

What’s changing about how I write posts?

Absolutely nothing :). There is absolutely no change to how posts are written. We still support all the Prettify language aliases you know and love, along with the new aliases from highlight.js. However, we are not adding support for any new languages at this time, instead choosing to keep the initial changeset simple and aiming for current feature parity instead. All the current markdown syntax is still supported, along with determining code highlighting from tags and site defaults.

So what is changing?

The “only” changes are visual. We are updating the client-side code block renderer that styles your code in posts (Questions, Answers, etc) and in the editor preview. Syntax autodetection when a language is not specified should be much better overall, along with syntax highlighting coverage in general. The biggest outward facing change for the typical user is going to be our new theme (see below for details).

Why highlight.js? Why not…

Why did we pick highlight.js over Prettify? Well, first off, you asked for it specifically. More convincingly, it’s open source, actively maintained, and overall just a solid product.

We’re extremely concerned about perf here at SO (both on the client and on the server), so we needed to ensure that this major change on our hottest page on the site didn’t negatively impact our users. There was some prior investigation into highlight.js's perf back in 2016, but I figured we should give it another shot.

In our internal performance benchmarks highlight.js scored better than Prettify consistently across all browsers (except macOS Safari 13.1, where it was actually a bit slower)2. It is a tad heavier than Prettify3, weighing in at an extra ~17kB (over the wire) after including all the languages we support across the network. This extra weight gain was acceptable to us as a tradeoff for what we were getting in return.

Why did we pick highlight.js over other contenders? Simply put, it was the best option that served our needs. We needed a library that we could easily control for use in the browser (deferred loading, theming specific elements), while also being simple to consume via a npm package, not needing specific build steps or a special babel plugin to pull in only the parts we need. Additionally, we could run it on the server (via Node.js) to unify our syntax highlighting in our Stacks documentation, giving us a single syntax highlighter across our products. Also a major plus was the ability to tokenize the highlighting result for use in our new editor (stay tuned!).

What are some potential drawbacks?

The most obvious not-quite-a-drawback is that language autodetection is different from Prettify. In general, it will be much more accurate, but will possibly end up with a different result that what Prettify would give us. This isn’t so much a bad thing, as it is just a thing that might take some getting used to if you’re a Prettify power user.

As mentioned earlier, the overall code bundle size is a bit bigger too. The vast majority of users wouldn’t even notice the change, which would only affect the first fetch since the browser will cache the file locally for subsequent hits anyways.

The last item is a bit of a personal preference. highlight.js tends to not highlight punctuation, which makes it a bit less colorful than other highlighters. This is considered a feature. Not a deal breaker by any means, but something I should mention regardless.

Designing the new theme

To offer some insight into how the new theme was designed, I reached out to the author, principal design systems designer Aaron Shekey.

Since we’re upgrading, we wanted to take this opportunity to design a Stack Overflow-flavored theme that takes advantage of newer tech like CSS variables that are aware of both light and dark modes. While we’ve improved it over the years, it’s highly likely that the current production theme simply used the stock colors provided by Prettify.

We’d need a theme that could work in both light and dark modes, was informed by Stack Overflow’s branded colors, and introduced a bit more contrast throughout.

Thankfully, we weren’t starting from scratch. When we built our Stacks documentation, we’d spent some time making our Jekyll theme display code snippets that got pretty close to accomplishing those goals. However, this was before dark mode was a thing, and we’d only built a single theme that assumed a fixed dark background. We’d have to extend this theme to light mode and revisit contrast along the way.

Using the Stacks documentation as a playground, we’ve now got themes in both light and dark modes that look like Stack Overflow and add or maintain contrast levels. We did our best to accomplish a contrast level of AAA, with a few variables dipping into AA. You can see the exact measurements commented in our colors constants file.

Here are a few screencaps of the new theme taken from my local development environment (click on the images to expand them). You can preview more languages (with an easy dark/light mode toggle) over at the Stacks docs.

Before

hljs before

After

hljs after

FAQ

  • Q: When is the rollout happening?

    A: We're planning to roll this out to meta.stackexchange and meta.stackoverflow on Thursday, September 10th. Rollout to the rest of the network is scheduled for September 24th, after the initial testing period. This is a soft rollout date, dependent on any bugs/feedback we get from the community during the testing period.

  • Q: What if I find a bug?

    A: Report bugs in an answer (one per answer) to this question. We'll keep this open for a couple/few weeks (until Friday, October 2nd) to address any immediate issues and then we'll update this post and ask you to post bugs as new questions after that time.

Footnotes

1 I checked, plural of syntax is syntaxes. Take that spell-checker!

2 Client-side benchmarks being what they are, we measured anywhere from ~49%-60% increase in the rate of ops/second depending on the machine and browser. Outliers being Safari 13.1 which had a ~29% decrease (favoring prettify) and Edge “legacy” scoring a ~279% increase over prettify!

3 Size comparisons were done comparing the prettify-full.en.js file taken from production vs the new highlight.pack.js bundle. Both were minified and served via a webpack-dev-server instance with the compress flag set (enabling gzip support). They were then included onto a regular html page with script tags and measured using the built-in browser dev tools. At the time of measurement, prettify landed at 23.3kB over the wire (meaning that the file was minified + gzipped) vs highlight.js at 40.7kB. This is a 17.4kB increase or about a ~74% increase in file size.

  • 44
    Good to see that the Teams devs are also bringing features to the public network! – Luuklag Sep 8 at 18:40
  • 13
    I have some updating to do then, for example on this one: What is the default language for the syntax highlighter? ... – rene Sep 8 at 18:56
  • 4
    @rene I have a (incomplete, I'm sure) list of posts that need updating after this goes live. I'll add this one to the list. Thanks for bringing it to my attention – Ben Kelly Sep 8 at 19:04
  • 7
    I hope the syntax highlighting FAQs (both here and on MSO) are part of that list as well. – Sonic the Masked Werehog Sep 8 at 19:18
  • 2
    @SonictheMaskedWerehog Yup, they both are! Thanks for checking in though. Better safe than sorry with those posts since they're the "source of truth" for this feature. – Ben Kelly Sep 8 at 19:29
  • 12
    How badly does it choke when asked to highlight the omni-glot? (Yes, that's a single piece of code that's a different valid program in each of 294 languages.) – Mark Sep 8 at 20:08
  • 9
    @Mark Honestly, not too bad. Speed was (visually, didn't actually benchmark it) comparable to my test page that had ~16 different languages on it. For the curious, the autohighlight detection marked that code snippet as bash. – Ben Kelly Sep 8 at 20:25
  • 15
    @SonictheMaskedWerehog why invest resources into something that isn't the core product? – Braiam Sep 8 at 22:41
  • 2
    @Braiam I thought SO was their core product. Syntax highlighting is a core part of that. – Sonic the Masked Werehog Sep 8 at 23:03
  • 28
    Q&A is the core product, @Sonic, not syntax highlighters. Syntax highlighting is just a small portion of that, and not in any way critical to the platform's success. It isn't something Stack Overflow needs ownership or control of. Just as you shouldn't develop your own JavaScript framework, you shouldn't develop all your own tools when there's something already out there that does the job well. – Cody Gray Sep 9 at 0:07
  • 3
    Given that highlighting occurs client side, how about a feature that allows the user to specify the highlight.js template(s)? It would be very cool if code on SO looked the same as my IDE. Familiar (color) styling makes code easier to read. – Mike Sep 9 at 1:02
  • 5
    Please, ask your questions as answers, as that gives Ben more room to respond individually! – Catija Sep 9 at 6:19
  • 13
    Whatever you do, please don't change the line-height for code blocks – hkotsubo Sep 9 at 12:53
  • 2
    Don't forget to update this page to mention highlight.js instead of Google Prettify. – Clonkex Sep 17 at 1:43
  • 2
    @Clonkex Already good to go! You can see the update on the meta version of that page (happens automatically when turned on). Good looking out though – Ben Kelly Sep 17 at 19:21

14 Answers 14

132

Can Stack Exchange please update to newer versions of Highlight.js on a regular cycle, rather than only on request?

As I said in my prior post reporting Google's discontinuation of Prettify, the process of filing bugs and feature requests with syntax highlighting would be quite drawn out and take a needlessly long time. The process was like this:

  1. File a bug with Prettify, which would take 6-8 months to be resolved, if at all. (I filed a bug in 2014 and it still hadn't been resolved by the time Google put the project to rest.)
  2. Once the request was resolved in Prettify, file another feature request here on Meta to request that SE update to the newer version. This would take the typical 6-8 week response time, but would often take longer than most requests because as far as I can best tell, they were only actioned when a developer happened to stumble on them.

As far as I can tell, Highlight.js is very actively maintained and requests with it are resolved fairly quickly, so #1 isn't an issue anymore (at least not in the current term). However, #2 will still remain an issue if SE sticks with their pre-existing model of only updating to newer versions on request.

Can Stack Exchange please actively update to newer Highlight.js versions on a regular cycle (not necessarily immediately after they're released, as I understand that'd be too onerous), rather than only updating to newer highlighter versions upon request? This would eliminate the problem in #2 and make the overall process significantly faster as one need only file the bug or feature request with Highlight.js and it'd be fixed in SE fairly quickly.

| improve this answer | |
  • 9
    How about quarterly. I personally wouldn't consider that too often, in terms of sending the update package over the wire, and it would be regular enough to not having to live with a bug for too long. Question remains then if you want to immediately update to the latest version, or want to lag a version or 2 behind, to see if there are any obvious bugs that you might be introducing. – Luuklag Sep 8 at 19:59
  • 1
    @Luuklag Yep, that's definitely a regular cycle. – Sonic the Masked Werehog Sep 8 at 20:00
  • 94
    This is a good question. I'll bring it up internally to formally decide as to what team will be responsible for keeping this up to date as well as how we're going to ensure that we're alerted when a new version comes out. As it currently stands, I have a few OSS side projects relying on highlightjs that get weekly notifications when dependency updates hit npm, so at the very least, I'll be aware of them. Thanks for bringing this to my top of mind. This will be very important moving forward, because, as you mentioned, highlightjs is being actively maintained. – Ben Kelly Sep 8 at 20:05
  • 1
    Would an update address the problem exhibited in unix.stackexchange.com/a/608110/5132 ? – JdeBP Sep 9 at 5:48
  • This is all the more relevant as the current roster of lexers supported by highlightjs is quite basic. It supports a lot of languages, but many only at a rudimentary level (then again, it’s hard to do worse than the current Stack Overflow syntax highlighting …). – Konrad Rudolph Sep 10 at 13:35
  • 8
    As an aside, it pains me to say that I'll be technically shipping this today with an out-of-date version. highlight.js released a 10.2.0 release the day before posting this announcement. We're shipping 10.1.2, so this will be a good test of how long it takes to get an update out the door ;). – Ben Kelly Sep 10 at 15:09
  • 1
    @BenKelly 10.2.1 has a pretty important fix you probably want before you go rolling out across the whole network. The match "skip" (in some cases, like end must match begin, etc.) functionality was broken in some cases (depends on input code). While our entire test suite actually still passed (which is pretty incredible) some code snippets could end up very badly broken (highlighting just stops abruptly in the middle, etc). – Josh Goebel 15 hours ago
  • @JoshGoebel Thanks for the update and the support! 10.2.0 is the current version on Meta. I will get a 10.2.1 merged before going live on Thursday. Good looking out and thanks for all your hard work on the library – Ben Kelly 13 hours ago
74

I found it rather frustrating that I couldn't easily see how the before/after pictures differed, so I did a bit of cutting and pasting so I could look at the before/after side by side to compare them more easily. Then it occurred to me that others might like to do the same, so feel free to have a look. Should be the same basic info as in the question, but arranged for more meaningful viewing.

First dark mode:

enter image description here

And then light mode:

enter image description here

Sorry, my cutting wasn't quite perfect, so (especially in light mode) you can see some dark lines that really shouldn't be there. But even if there's a little extra junk, at least you can do a real comparison so the changes are reasonably apparent.

To me, the new color scheme appears to have at least a couple of different general types of problems.

One is technical accuracy (i.e., accuracy in the tokenization itself). For example, looking at the Python example, if is in one color, and None in a different color (which appears to be the same color for 0, 1, and 0b101 and for someFunc and SomeClass). if and None are both keywords, so it would appear reasonable that they both be the same color. It doesn't seem reasonable or useful for two keywords to be in clearly different colors, and one of them in the same color as some identifiers and literal values.

Another is the choices of colors themselves. Generally speaking, for comfortable viewing we want to balance between two extremes. If there's too little difference between colors, it's not always clear whether two things are the same or different colors. When colors can't be distinguished easily, we lose much of the benefit of using coloring to start with.

At the same time, we don't want too much contrast, especially when two things are immediately adjacent to each other. If we do this, viewing simply becomes uncomfortable1.

In this case, we see what may be some of the first problem. As previously mentioned, in the Python example, None, someFunc, SomeClass, 1, 0 and 0b101 are all shown in what looks like the same color. It's possible that this isn't really a parsing problem--maybe it's assigning a unique color to each, and they just happen to be so similar that we can't distinguish them.

The old color scheme also differentiates between the class name and the function name, where the new one seems to use the same color for both. Given that they're both syntactically identifiers, it's open to argument that this doesn't affect accuracy (as such), but it seems pretty clear to me that the old scheme is providing more useful information.

In the dark mode pictures, we see at least a few clear-cut cases of excessive contrast. The most obvious are the parameters (param1 and param2) shown in bright white against a deep-black background. In this case, we've pretty clearly gone beyond the level of contrast that most people can look at comfortably. As an aside, there are a few cases where it's a bit more reasonable to break or at least bend this rule a bit. For example, if you're coloring something with a very small area (e.g., a period or comma) you can often get away with a bit higher contrast than if the area were larger.

At least in my opinion, the light mode version of the new coloring fares at least somewhat better in this respect. We still have None colored to match the identifiers and literals, and mismatching if. On the other hand, the background in this case is a light grey, and the parameter names are in a somewhat darker grey, so the contrast range is considerably more manageable.

Given a wide audience, we'd also like the color schemes to work well for people with color impaired vision. The most common color vision impairment is called Deuteranomaly. If we run the pictures through a filter, we can see a simulation of approximately how these would look. For example, here's the light-mode Python code with simulated deuteranomaly vision:

enter image description here

Here we see that in the new color scheme, the comment is only barely distinguishable from the preceding code, and even less so from the literals (e.g., 'gre\'ater') It might not not so close that I'd consider it a clear failure in this regard, but it's enough to make me at least a little uncomfortable (and at least with respect to serving people with color vision deficiencies, pretty close to an outright failure).

The old color scheme is clearly superior in this regard--although contrast is certainly reduced in some cases, everything that started out as a separate color remains quite easily distinct.

There are, of course, other forms of color vision deficiency, up to and including truly complete color blindness. Fortunately, that's pretty rare. Deuteranomaly is the most common, and dealing well with it will frequently also work out well for most of the other somewhat less common cases (e.g., Protanomaly, Tritanomaly, etc.)

Unfortunately, it's fairly difficult to do automated testing of when colors have enough contrast for the difference to be easily visible. There are computations for "delta E" to tell you how much difference there is between two colors, but eyes are easily deceived, so (for example) the surroundings can make two areas with identical colors look obviously different, or areas with different colors difficult to distinguish. About the best we can hope for in a case like this (retrofitting to a system, affecting far too many pages to review each individually) is to get rid of obvious problems.


  1. Now rarely relevant, but back in the days of CRTs you could get away with more in this regard, because individual pixels tended to have some degree of gradient at the edges, so even the brightest white against the darkest black still had at least some degree of gradient from one to the other. That's much less true with LCDs though, so we have to be more careful as the technology no longer covers for our mistakes.
| improve this answer | |
  • 5
    Looking beyond the individual colors in isolation, however, we see far too high of contrast. The bright white of the variable names on the dark background is literally painful. The Haloween orange used for the function name also contrasts much too strongly with the aforementioned putrid green. – Jerry Coffin Sep 9 at 6:48
  • 8
    The new color scheme is also problematic from a functional viewpoint. For example, it appears to use the same color for the name of the class, the function, the numeric literals, and the None (in the Python example). Some of it doesn't even may sense syntactically--for example, in Python both if and None are keywords, but the new colorizer has given them different colors. – Jerry Coffin Sep 9 at 6:54
  • Oh, and just in case the subject should arise: yes, I calibrate my monitors on a regular basis, and I keep the background lighting dim enough to assure that my eyes are adjusted almost exclusively to the white point of the monitors, not the room (though it's at a fairly reasonable approximation of the standard D55 illuminant in any case). – Jerry Coffin Sep 9 at 7:07
  • 5
    I prefer the old colors, because they look like Visual Studio. The new colors feel really weird (especially the green strings, which I always associate to comments). – Métoule Sep 9 at 8:01
  • 6
    I disagree that the new scheme is at all garish, though obviously that is a subjective consideration, and they'll never keep everyone happy with any change (or no change!) in that area. As mentioned by @Métoule though, green for string literals could be an issue given several very common tools use green for comments. I like that you are keeping comments grey though. In any case I find what is highlighted and the contrast levels far more important than the specific colours, I quickly get used to new colour schemes as long as I can see the key things I find helpful to see identified. – David Spillett Sep 9 at 9:49
  • 9
    Having sucessfully ranted against a design change - the specific issues probably should be in the questions, especially things like "Some of it doesn't even may sense syntactically--for example, in Python both if and None are keywords, but the new colorizer has given them different colors." One finds things like insipid and garish are often a matter of personal taste though – Journeyman Geek Sep 9 at 10:37
  • 1
    @Métoule the IDEs I use have green for strings, grey for comments. – OrangeDog Sep 9 at 13:48
  • 8
    Thank you for creating the side-by-side before/after shots. I didn't consciously create them in any certain manner other than "I took the old shots at the same time and the new shots at the same time". I'll pass your feedback on the color scheme to the designer. I do see your point about some colors being potentially mismatched or not matching other IDEs. I'll do a bit of cross referencing and see if we can't make some tweaks there for a quick win. – Ben Kelly Sep 9 at 14:28
  • 8
    Side note: I found the info in your comments to be very helpful. I'd recommend adding it to your post to help flesh out the statement/request a bit more and to help deter drive-by downvotes for "personal taste" vs "specific issues" – Ben Kelly Sep 9 at 14:30
  • 18
    Re Python specifically, None and if are both keywords, but they have different semantic roles: if is control structure, None is a constant. Highlighting them in different colors is common in IDEs and I actually think it's better that way. – zwol Sep 10 at 12:59
  • @zwol: I can see at least some merit to that line of thinking. My personal preference would be to start from a high level and make things more or less hierarchical, so (for example) key words are in shades of green and values in shades of blue.In this case, I can see where it would make sense for None to be more or less a teal color to accurately reflect that it's a keyword that's typically used in a value context, so to speak. – Jerry Coffin Sep 10 at 18:10
  • 2
    Hey Jerry, I want to thank you for your updated answer - it's really great. :D – Catija Sep 10 at 18:49
  • In IDLE, repl.it (online IDE), and PyCharm, if and None are the same color by default. – Rainbolt Sep 14 at 14:12
  • If the old colour scheme is better, as it seems to me that it is, is it possible to create a "theme" for the new highlighter to match the old highlighter's colour scheme? – Sam Watkins Sep 14 at 22:19
47

I'd like to say that I appreciate this post.

It is clear, very informative, very detailled, and to me shows that person's concern for the community.

Of course, there will always be different opinion on the result ("I prefer the former highlighting" "I prefer the new one!") but that is inevitable.

I find the reasons to change (and the choice) compelling enough, and the resulting highliting is pleasing to the eye.

(I see some concerns about having several things showing up with the same color: this is inevitable. The highlighting is there to have successive part in a different color, thus making transitions visible, and the overall structure appear, and not to have everything with its own specific color)

Thank you, @ben-kelly, for the information

| improve this answer | |
  • 4
    Thank you for the kind words! I'm glad the post resonated with you. If have any specific concerns about certain items looking funny (either overall or with a specific language), I'm happy to field those as well – Ben Kelly Sep 9 at 14:56
  • @BenKelly: thank you! And about corrections: I'll let others much better than me answer that (and I personnally find it fine so far. And it is probably good to keep things as 'default/vanilla' as possible for now until it is validated and fully stress-tested, unless some glaring problem is pointed out) – Olivier Dulac Sep 9 at 15:07
  • 2
    @BenKelly: I'd just advise (it's probably done): test with several different persons having several type of 'color vision deficiencies', as it is a common problem and this affects a lot of people (some webpages or charts rely only on color to convey some informations, and they are not useable by all). Whenever possible other means (graphics, signs, italics, etc) help as well (for syntax coloring, there is not much option, but contrast or thickness may help to help differentiate some colors. I'm quite sure the devs of the .js took this into account: it may even explain their color choices?) – Olivier Dulac Sep 9 at 15:16
  • 2
    Believe it or not, we did actually do some testing with Chrome's new-ish Emulate vision deficiencies tool. You can see some of my comments (w/ pics) on the public Stacks PR. Obviously, emulation is not perfect, but definitely better than not checking at all. – Ben Kelly Sep 9 at 15:29
  • 1
    @BenKelly: Good to know! – V2Blast Sep 11 at 0:46
24

<!-- language-all: lang-none --> hint doesn't seem to work anymore

This post has a <!-- language-all: lang-none --> hint at the top of the post to prevent all the code blocks in it from being highlighted. I tried changing lang-none to none and it still didn't work. (As you say in your post, Prettify identifiers will still continue to work even after the change.)

We were told at the time of the CommonMark migration that <!-- language-all: [language] --> hints would continue to be supported, unlike the <!-- language: [language] --> syntax which was being deprecated.

This issue seems to be specific to the lang-none and none hints as part of this style of HTML comment; other ones seem to be working fine. As an example, this post contains such a comment to indicate C as the highlighting language, and the below snippet is highlighted in C:

#include <stdio.h>

(To test, I also changed the comment to indicate Python and it highlighted the above as Python.)

It seems to work for individual code blocks, using the code fence notation (i.e. ```none and ```lang-none):

#include <stdio.h>

In summary: <!-- language-all: lang-none --> and <!-- language-all: none --> don't seem to work to disable syntax highlighting for a particular post.

| improve this answer | |
  • 5
    Thanks for the report. Will look into this today. – Ben Kelly Sep 11 at 14:04
  • 4
    @BenKelly One week later, what came of the review? – Sonic the Masked Werehog Sep 19 at 5:08
  • Sorry, forgot to respond to this one. This is currently working identically to other sites where highlight.js is not turned on yet. For instance, try on Stack Overflow (just the preview, don't need to post!). language-all only works on blocks that do not have an override language set, such as via the code fence string or by a regular language comment. – Ben Kelly 12 hours ago
18

😄 Thanks for doing this! I'm happy about this outcome, as I was a major proponent of switching to highlight.js back in 2016.

Great! …but what changed?

To satisfy my own curiosity, I'm wondering if you have an explanation or theory for what changed between 2016 and now to make the switch feasible. Oded's performance analysis seemed to raise some major issues, and your post indicates they are no longer issues, but I don't see an explanation for why things changed. For example:

Size in 2016:

It is [too big] … an extra 5kb minimum for millions and millions of requests a day … This size concern only grows with adding more languages.

Size now:

… an extra ~17kB (over the wire) after including all the languages we support across the network. This extra weight gain was acceptable to us as a tradeoff for what we were getting in return.

Speed in 2016:

… (don't forget - we have a highly nested DOM, and many "benchmarks" are done on a very simple page - which is not indicative of performance on Stack Overflow). … In my tests, CPU time for highlight.js was anything between two and four times higher than for prettify … I have also tested by using console.time around our highlighting calls - highlight.js consistently performed worse than prettify.

Speed now:

In our internal performance benchmarks highlight.js scored better than Prettify consistently

Is this size difference acceptable now because of changes in browsers/networks/CDNs, or just because different people were making the decision? Surely the number of requests per day has only increased since 2016?

Do you have information about what performance tests Oded ran in 2016 and why your results now are so different? Is the internal performance testing infrastructure new? Have there been underlying technical changes to the "highly nested DOM" to make highlighting more efficient? Or have there been significant performance improvements in highlight.js itself?

Again, I'm glad the change was made now — I'd just like to know if there was a legitimate reason to wait 4 years and what changed in that time. Was there something we could have done differently to encourage adoption sooner?

| improve this answer | |
  • 3
    Well, for one thing, version 10 was released, which users EC2015, which is not supported by IE. However, as SE has long retired IE support (and removed all IE-specific code last year), that's not an issue. – Sonic the Masked Werehog Sep 9 at 18:00
  • 6
    What also happened is Yaakov. I've managed to get some low-hanging fruit Prettify FR past him in January and that might have triggered him enough to realize they were on a dead-end. We're blessed with him as Community Advocate. – rene Sep 9 at 19:22
  • 4
    I assume a very large factor was the matter of prettify not being available as a "modern" package and no longer being maintained. That has a major impact on the maintainability of the application that's not directly obvious. The caching infra also changed between 2016 and now and four years are quite enough time for an OSS team to improve performance... – Vogel612's Shadow Sep 10 at 10:51
  • 6
    I've seen this post and have not forgotten about it. I'm touching base with our Architecture team to draft up a satisfying answer since they are the ones that made the final decision on the changes, given the benchmarks and size delta. – Ben Kelly Sep 10 at 14:13
  • @rene Don't entirely think that was responsible. My later post in June about Prettify being discontinued by Google wasn't responsible either, in my opinion, as it originally alluded to creating a new home-grown SE syntax highlighter, which probably resulted in that post being internally demoted. I think the real trigger was Ham Vocke's post on CommonMark, on which someone had asked if syntax highlighting would be changed, to which Ham's response was "we'll think about it once CommonMark is fully rolled out", and I think it was placed on their internal radar at that time. – Sonic the Masked Werehog Sep 11 at 19:21
10

What to do if highlight.js supports a language but Stack Exchange doesn't?

There's an entire Mathematica StackExchange, and so Mathematica syntax highlight is clearly very important to us. But when I did some digging to find the highlight.js bundle that SE seems to be serving, Mathematica isn't in the registerLanguage("...", ...) blocks, even though it is in the set of aliases that StackExchange seems to be defining, i.e. this line is in the bundle

StackExchange.highlightjs=function(){var e={..."mma":"mathematica",...} ...}

It's vaguely annoying to be forgotten by the company to whose platform we've contributed so much, of course, but it'd be way more annoying for our nice syntax highlighting to disappear all of a sudden. We've been getting by with custom support for Google Prettify as written by one of our mods. The lack of Mathematica support is extra confusing when we consider that highlight.js already supports it.

So...what's the protocol for adding highlighting for a language that Stack Exchange, the company, need do nothing extra to support, since highlight.js already has it.

Sample Code

For reference, the following block is fenced with lang-mathematica as the spec. As of when I write this, it renders un-highlighted.

pot =
  Evaluate@With[
     {
      n = 4,
      l = 1,
      c = .25,
      s = .075,
      scale = 4,
      broad = 5
      },
     scale*(JacobiP[n, l, l, #/broad] + .2 JacobiP[2, l, l, #/broad])*
       PDF[
        MixtureDistribution[
         {1, 1},
         {
          NormalDistribution[-c, s],
          NormalDistribution[c, s]
          }
         ], 
        #/broad
        ] - PDF[NormalDistribution[0, .35], #](*+(#/broad)^2*)
     ] &
(* Out: *)
-1.1398350868612364/E^(4.081632653061225*#1^2) + 4*(2.659615202676218/E^(88.8888888888889*(-0.25 + #1/5)^2) + 
    2.659615202676218/E^(88.8888888888889*(0.25 + #1/5)^2))*(5 + 0.2*(3 + (15*(-1 + #1/5))/2 + (15*(-1 + #1/5)^2)/4) + 
    35*(-1 + #1/5) + 70*(-1 + #1/5)^2 + (105*(-1 + #1/5)^3)/2 + (105*(-1 + #1/5)^4)/8) &
| improve this answer | |
  • 4
    I would say it is a bug that it's not included. I don't understand highlight.js enough myself, but for some reason the MMA language JS file is in a separate additional-langs folder (by itself) that doesn't seem to get included when the file is built out. It seems like it is supposed to be included, especially given it is listed in the keywords. – animuson Sep 18 at 4:17
  • 2
    Excellent question. I assure you that Mathematica Stack Exchange will be supported at launch. Due to the large size of the mma language definitions, the language is actually split out from the rest. We do this currently with Prettify as well (check the network logs for lang-mma.min.js). I've already ensured this bundle has been created and works on my local machine. Thanks for checking in on this, I appreciate the concern and understand any potential anxiety around the matter. – Ben Kelly Sep 18 at 17:20
  • 3
    Some extra comments here for "fun": I did actually attempt to add mma support to the main bundle, but it added 31.1k after gzipping! That means a 75% bundle size increase for a single language. Unfortunately, that wasn't going to fly, so we're continuing to serve it to Mathematica.SE as a separate js file. – Ben Kelly Sep 18 at 17:26
  • 2
    I personally definitely think that's fair enough and I'm glad to hear that it'll work on our site! Do you have a rough idea of what you would be happy with to add it to the main bundle? Our old highlighting makes use of a trie to save as many bytes as possible and I wonder if it would be worth spending the time lowering the impact of the highlight.js implementation. (Also, just a note that MMA adds symbols regularly that need highlighting, so having a regular update schedule would be really great) – Carl Lange Sep 18 at 22:28
9

There has been times that I've turned off code highlighting with <!-- language: lang-none --> because Prettify was getting it wrong, and no highlighting is better than wrong highlighting. (The example that comes to mind was a Bash snippet where # wasn't a comment indicator, but Prettify thought it was.) After this change goes through, should I go back over those posts and turn code highlighting on again? Is it better?

I suppose I can test it.

| improve this answer | |
  • 4
    Honestly, I'd recommend simply adding the language any time you consciously think about it. If you use the code fence syntax (my personal preference) you can easily set the language. That being said, give it a shot! I have a sibling post on Meta Stack Overflow that explains how autohighlighting is changed (at 5000 ft anyways). – Ben Kelly Sep 9 at 14:33
9

We have been waiting for Verilog and SystemVerilog (SV) highlighting for a long time. Apparently we will have Verilog support with highlight.js, but SV will continue to be unsupported. Still much better than before. I'm happy with the change and appreciate your effort.

Let me put some Verilog code (from highlight.js demo) here to see the result after the roll-out. I assume the language code will be lang-verilog.

EDIT: We haven't got Verilog support as Ben Kelly mentioned in the comments. The following snippet has no language code, thus we see the result of auto detection.

`timescale 1ns / 1ps

/**
 * counter: a generic clearable up-counter
 */

module counter
    #(parameter WIDTH=64, NAME="world")
    (
        input clk,
        input ce,
        input arst_n,
        output reg [WIDTH-1:0] q
    );
    
    string name = "counter";
    localparam val0 = 12'ha1f;
    localparam val1 = 12'h1fa;
    localparam val2 = 12'hfa1;

    // some child
    clock_buffer #(WIDTH) buffer_inst (
      .clk(clk),
      .ce(ce),
      .reset(arst_n)
    );

    // Simple gated up-counter with async clear

    always @(posedge clk or negedge arst_n) begin
        if (arst_n == 1'b0) begin
            q <= {WIDTH {1'b0}};
            end
        else begin
            q <= q;
            if (ce == 1'b1) begin
                q <= q + 1;
            end
        end
    end

    function int add_one(int x);
      return x + 1;
    endfunction : add_one

`ifdef SIMULATION
initial $display("Hello %s", NAME);
`endif
endmodule : counter

class my_data extends uvm_data;
  int x, y;

  function add_one();
    x++;
    y++;
  endfunction : add_one
endclass : my_data
| improve this answer | |
  • 2
    Hurrah. About ....ing time. – Tom Carpenter Sep 10 at 11:17
  • 1
    If you want SV support, I suggest consulting the highlightjs dev documentation and creating a pull request. I’m in the process of doing this for multiple languages. – Konrad Rudolph Sep 10 at 13:37
  • 7
    I hate to be the bearer of bad news, but this update has not added any new language support (officially. Technically sublanguages like Less/scss got snuck in with the css support). I tested this code snippet on my dev environment and it autodetects as scala. That being said, adding more languages to our bundle is technically trivial, but we need to be mindful of bloating the deliverable size. I'll keep this suggestion in mind once we revisit and decide whether to add additional language support. – Ben Kelly Sep 10 at 14:10
  • @BenKelly Thanks for information. I removed the language code, but there is no highlighting for scala either. – ahmedus Sep 11 at 14:37
  • @BenKelly To avoid bloated bundles, could new synxaxes not be loaded dynamically depending on what's present on the page? (I'm holding out hope for ```diff and ```elixir support...) – Lionel Rowe Sep 11 at 15:18
  • 2
    @LionelRowe In theory, yes. That being said, I've not looking into the possibilities there or how it would impact us (from a bandwidth/hosting perspective) or our users (from a UX/performance perspective). This is definitely something we'll keep in mind if we decide to expand our supported languages in the future. – Ben Kelly Sep 11 at 17:35
  • 3
    I suppose we've been collectively waiting 7 years, whats a few more. ¯\_(ツ)_/¯ – Tom Carpenter Sep 11 at 19:56
  • @TomCarpenter You must be referring to that question. We have 15 days to the 7th anniversary. :) – ahmedus Sep 11 at 21:34
  • @BenKelly If performance is a concern, perhaps maintain two separate files: highlightjs-compact (contains only core languages, e.g. C++, Java, Python ...) and highlightjs-all (contains a larger selection of languages, including Kotlin, Fortran, SystemVerilog, ...). – Mateen Ulhaq Sep 19 at 2:54
  • 1
    @BenKelly There is an issue open for "dynamic loading" of languages on the Highlight.js GitHub, but it's not a huge priority right now. Someone has a plugin also, but I wouldn't call it "solid" yet... (not sure it's the proper way to go about this) The best way to do this is likely to "wrap" HLJS (or at least highlightBlock)... if the language is valid (but not bundled and not loaded yet), issue a fetch... and then on competition just call highlightBlock again in the promise success callback. While loading you'd apply the hljs class (so the blocks appear as unhighlighted code). – Josh Goebel 16 hours ago
  • I had a demo/prototype of something like this working for our codebase, but the problem isn't just the size of the bundle, but it's about number of network requests too. Adding even a single extra request to our most hit page on our most hit site (Question view on SO) would be murder on our CDN bill. Not insurmountable by any means, but it just needs extra thought/engineering at our scale. We wanted to get the base out first and make sure it works well before we really started messing with the process. – Ben Kelly 12 hours ago
9

I've just tried the following piece of JavaScript code (from this answer of mine in Code Golf) because Google Prettify was not parsing the regular expression followed by an inline comment correctly. That's why I used alternate slash characters in the original post (I've turned them back into regular slash characters below).

But this is much worse now, as the first 5 inline comments are not parsed as comments at all.

f = (                // f is a recursive function taking:
  [c,                //   c   = next digit character
      ...a],         //   a[] = array of remaining digits
  o = '',            //   o   = output string
  S = new Set        //   S   = set of solutions
) =>                 //
  c ?                // if c is defined:
    f(               //   do a recursive call:
      a,             //     pass a[]
      o + c,         //     append c to o
      o ?            //     if o is non-empty:
        f(           //       do another recursive call
          a,         //         pass a[]
          o + [, c], //         append a comma followed by c to o
          S          //         pass S
        )            //       end of recursive call (returns S)
      :              //     else:
        S            //       just pass S as the 3rd argument
    )                //   end of recursive call (returns S)
  :                  // else:
    S.add(           //   add to the set S:
      o.replace(     //     the string o with ...
        /\d+/g,      //       ... all numeric strings
        n => +n      //       coerced to integers to remove leading zeros
                     //       (and coerced back to strings)
      )              //     end of replace()
    )                //   end of add() (returns S)

I'm sure this is going to be fixed quickly, so here's a picture of the current rendering for later reference. :-)

rendering

| improve this answer | |
  • 6
    This looks like a bug upstream in the language processor itself. The first few comments are getting marked as being included in the params of the function. I'd recommend filing a bug or submitting a PR against highlight.js itself – Ben Kelly Sep 14 at 14:25
  • 5
    @BenKelly At any rate, there are probably a couple of good test cases on Code Golf as you'll find there plenty of unusual (yet valid) ways of writing and commenting code, at least for the few standard languages that are actually recognized by highlight.js. (We obviously don't use syntax highlighting at all for all the esolangs we're using.) – Arnauld Sep 14 at 14:46
8

Will the <!-- language: [language] --> hint be disappearing?

Back when SE was switching to CommonMark, we were told that the old <!-- language: [language] --> syntax had been deprecated and was subject to removal in the future. (Prior to the implementation of code fences, this was the proper syntax to force a single block of code to be highlighted differently from the rest of the post.) With this change, will that comment style be removed once this is rolled out to all the sites?

It seems to work fine at the moment. The following is specified as a C code block:

#include <stdio.h>

...and here's the same text, but as a Python code block:

#include <stdio.h>

Are there plans to remove it, or will it remain for the foreseeable future? If it is going to be removed, will it still be that posts rendered prior to its removal will still honor it until they're edited, as we were told at the time?

| improve this answer | |
  • 2
    At this moment, there is no plan to completely remove support for the language comment syntax. If this were to change, we'd make an announcement on meta first. As much as I am not a fan of the syntax, we don't have a good replacement for language-all in commonmark or our supported extended syntax. I've not changed anything in the implementation for the highlightjs switch, so any issues are to be considered bugs. (I see the one you reported above, thanks for reporting!) – Ben Kelly Sep 11 at 14:10
  • 3
    @BenKelly To be clear, language-all would continue to be supported; only language had been deprecated. As you say, there isn't a good replacement for language-all, but code fence hints exist as the preferred alternative to language today. – Sonic the Masked Werehog Sep 11 at 17:17
  • 2
    I should have specified: There is at this time no plan to completely remove either language or language-all. One or both could possibly be removed in the future (with language potentially going first since there's a real, widely accepted alternative). If we do decide to do this in the future, then we'll broadcast the change ahead of time in order to gather feedback from the community. – Ben Kelly Sep 11 at 17:33
  • @BenKelly I just read further into that post, and it seems like the plan wasn't to remove language entirely, but to remove its support only for four-space indent code blocks and retain it for HTML <pre><code> blocks. I think that may be too technically difficult to implement, so I think the best solution would be to retain it as is. (It seems to me that Ham intended to remove it entirely at that time, but ran into issues with it, so he simply marked it as "deprecated, subject to removal" instead.) – Sonic the Masked Werehog Sep 13 at 21:25
8

Current maintainer of Highlight.js here, though I'd add a few quick comments.

highlight.js tends to not highlight punctuation, which makes it a bit less colorful than other highlighters. This is considered a feature. Not a deal breaker by any means, but something I should mention regardless.

This is something I'm open to improving if someone wants to work on PRs and figure out a good way to go about handling this (work with existing themes, not be invasive, etc). https://github.com/highlightjs/highlight.js/issues/2500

I assure you that Mathematica Stack Exchange will be supported at launch. Due to the large size of the mma language definitions, the language is actually split out from the rest.

Some languages MIGHT also be possible to Highlight with a wildcard vs a list of EVERY keyword under the sun... I'm not sure if Mathematica would be one such language or not. Some of our languages are quite heavy because the keyword approach was just simpler (and more accurate). That said just breaking out the files and loading them (as needed) is probably the best solution for some of the less popular languages. And would also help with auto-detect speed.

For example, looking at the Python example, if is in one color, and None in a different color (which appears to be the same color for 0, 1, and 0b101 and for someFunc and SomeClass). if and None are both keywords,

We've always highlighted literals and keywords differently. For Python False, None, and True are currently defined as literals.

the first 5 inline comments are not parsed as comments at all.

Definitely a bug (and should be an easy fix), a GitHub issue would be appreciated. :-)

Language auto-detection for assembly language seems to be broken.

Auto-detect is on a "best effort" basis... the smaller the snippet the worse the auto-detect, but some languages are also much harder to auto-detect than others. If you really think there is an OBVIOUS issue (a huge snippet that is constantly flagged incorrectly, etc) then a GitHub issue would be more than welcome...

Different flavours of assembly language use different comment characters, so this is a somewhat thorny problem.

Indeed, and why have multiple assembly grammars, not just a single one. I have no idea if it would be possible to have a single grammar for exactly this reason.

| improve this answer | |
New contributor
Josh Goebel is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 2
    Off topic comment: the links to your GH profile on this page are broken, they point to github.com/yyyc514 which does not exist. Can you please fix it? – Shadow Wizard Wearing Mask 17 hours ago
  • 1
    I'm not sure I can update the website itself (I'll ask - it's still managed by the original author), but I've updated the CHANGES.md file on GitHub and it'll be correct in future releases notes. Also added a placeholder/pointer profile so no one should get too lost in the future. Thanks for pointing it out! – Josh Goebel 17 hours ago
  • 1
    Nice trick, @Josh. Thanks! – Shadow Wizard Wearing Mask 16 hours ago
  • @JoshGoebel Thanks so much for checking in! Really excited to be using highlight.js. I've written a couple of projects against it and I can say that I'm really digging it for the most part. Looking forward to giving back to the project as well soon. Once I can find the time, I have a few suggestions / PRs to file against it ;) – Ben Kelly 12 hours ago
5

Does highlight.js support emphasis in blocks formatted as "code" (ie indented 4 spaces)?

Paraphrasing an MSE question:

In-code highlighting (anything will do), would be a great way to emphasise the important parts.

Currently, the best people can do is ASCII art arrows, eg:

printf("%5s", "foo")
         ^--- add a width value

which happens often enough and is probably not done more because it's a pain and ugly.

It would be great to be able to highlight (in this case) the 5 by making it red, bold, or whatever by surrounding it with some special chars, maybe like !5! or whatever works.

Awesome would be highlighting with a comment that isn't selected when copy-pasting the code block.

| improve this answer | |
  • There's nothing like this in highlight.js proper that I'm aware of. Technically, we could support diff highlighting, but that brings its own set of drawbacks. highlight.js does support plugins, so this could be something that we could potentially write a plugin for. This is not currently on our radar as far as I know, but it is definitely something to think about for the future. – Ben Kelly Sep 18 at 17:31
  • It wouldn't help in the code block situation since the HTML tag would be displayed rather than treated as a tag, but it would be nice if <mark> were a supported HTML tag on StackExchange so that relevant potions of text could be highlighted. – M. Justin Sep 18 at 18:42
  • @ben something that both amazes and saddens me is SO not tapping into the 1000’s of top class developers that would willingly donate their time (they already do) to writing code to deliver features. For some a reason I can’t fathom, SO is dev is a closed shop. In this particular case, surely there are 100’s of devs that are members of this community who could write such a plugin. Why not just ask them to do it, and it would get done pronto. – Bohemian Sep 18 at 18:47
  • @M.Justin yes, there must be a way. If we write a plugin, we can pick what we like. Eg <mark comment="some comment that is visible, but not selected when copying code">some.code()</mark> – Bohemian Sep 18 at 18:53
  • <pre><code> blocks can include arbitrary HTML (IF the editor permits AND they're using the traditional highlightBlock)... so you could just wrap it in <span class='highlight'>my code</span> and assuming there was a highlight class the HTML is preserved and will be "passed thru" while the code is highlighted... so when finished that span will still be in place. Requires a CSS class though. Long-term a solution like <mark> via some sort of plugin would probably be better though. – Josh Goebel 17 hours ago
4

Language auto-detection for assembly language seems to be broken. For example, note the lack of highlighting in the question on Printing an integer as a string with AT&T syntax, with Linux system calls instead of printf. After editing my answer to use ```lang-assembly on the main code block, that block has highlighting but the others don't. (And does actually look decent.)

Separately, syntax highlighting for NASM (a different asm syntax that uses ; as the comment character) is broken. In Unexpected result of subtracting a NASM macro in an expression, ```lang-nasm or lang-assembly leads to a mess that's arguably worse than nothing, because of single-quote used as English punctuation in a comment. Same result with <!-- language: lang-assembly -->.

I tried to reproduce an example in this meta answer, but it's not doing highlighting at all, with either or both of <!-- language: lang-assembly --> or ```lang-assembly. (I'm using dark mode on SO, light mode on meta, in case that matters.)

section .rodata           ; groups read-only together inside the .text section
    msg: db "Thank you"
    var: db 0x34, 0x33, 0x32, 0x31   ; dd 0x31323334  ; Since you're passing these bytes to write(2), writing them separately is probably less confusing.  (x86 is little-endian, so they will come out "backwards")

    ;; apparently you want to include var as part of the message, for some reason
    msglen equ $ - msg    ; $ is the current position

SO asm answers tend to be more heavily commented than you'd do in real life, because the target audience is people that don't understand the basics of asm. And SO code blocks are more cramped horizontally than a normal text editor so it encourages leaving comments closer to the end of the code, making it worse if they're visually harder to ignore. (Especially in some poorly formatted beginner questions and answers where comments are ragged and literally no space is left after instructions.)


Different flavours of assembly language use different comment characters, so this is a somewhat thorny problem. Some use # to decorate numeric literals (e.g. ARM), so treating ;, #, and @ as comment characters won't always work.

I assume that's an upstream problem, not really stack overflow, except for the tag-detection issue.

| improve this answer | |
1

Here's a proof-of-concept for lazy loading of syntaxes that doubles as a Tampermonkey user-script:

Highlight.js lazy-loading proof of concept

Naturally it's a little hacky, but it works on all of the following examples:

lang-diff

- print('failure')
+ print('success')

lang-elixir

spawn_link(fn ->
  send(current_process, {:msg, "hello world"})
end)

receive do
  {:msg, contents} -> IO.puts(contents)
end

lang-bf

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.
| improve this answer | |
New contributor
Lionel Rowe is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • Does it work on CSS? The Google Prettify highlighter has always had trouble recognizing CSS. – user289905 8 hours ago
  • @user289905 It works only for languages specified with lang-XXX. Shouldn't be required for CSS, though, because that's already eager-loaded by default. – Lionel Rowe 23 mins ago

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .