Stop guessing/auto-detecting a language when you KNOW it will be incorrect

Question

The problem: SE asks Highlight.js to autodetect the language when it knows there isn’t any optimal/correct choice for us to make - resulting in very poor outcomes.

Disclaimer: I say this as the current Highlight.js maintainer

---

Example: SE currently does not load our groovy grammar. When one adds a Groovy block of code and hints it as ```groovy or <!-- language: groovy -->, SE will still ask Highlight.js to auto-detect the language - even knowing the language is groovy and that they've purposely chosen not to enable our Groovy grammar.

This results in poor and inconsistent highlighting for many snippets and encourages bad user behavior that will only make the situation worse long-term. Auto-detect is not intended to be used to find "next best" matches for built-in grammars purposely excluded from a build. This will frequently result in highlighting that appears entirely random (based on variable names that match keywords, etc.).

List of reasons the existing behaviour is bad:

• It makes users think a language is supported when it is not (this confusion is obvious in many threads following the switch to Highlight.js)
• It results in incorrect/poor highlighting here and now (since the correct grammar is not available).
• It results in seemingly random highlighting (different snippets of a single language end up highlighted with many different languages based on the exact content of the snippet).
  • Worse, this can encourage people to mis-hint or mistag posts consistently (i.e., always using java instead of groovy) just to get more consistent highlighting. This has already been mentioned/suggested in other threads (see Groovy discussion).
  • This mis-hinting/mistagging is not future-proof... if one day SO decides to add proper Groovy support, but older posts are tagged/hinted java (as a workaround)... those posts will not receive the new highlighting that would be possible if they had been hinted properly.
• It can encourage hinting snippets with none (to avoid terrible auto-formatting) or even picking a random language just to find something that looks "better".
  • This is also not future proof in that if the missing language is ever added in the future the incorrect suboptimal hint will continue to be used indefinitely.
• It can encourage users to endlessly fiddle with their snippet just to see if they can "push" the highlighter towards a better choice.

---

What should happen instead:

If it's known that the language requested is not supported then one of several things should happen:

• No highlighting should be used, i.e. alias to none or plaintext. Unfortunate, but consistent.
• The next closest match should be hard-coded as an alias. You're already doing this for some languages, like your VBScript → VB.NET mapping.
  • This results in consistent behaviour (keywords will always be highlighted the same from snippet to snippet).
  • Users can learn the pros and cons of this behaviour (i.e., its quirks, etc...)
  • If/when additional language support added in the future, the alias is removed and all existing posts that are correctly hinted are immediately "upgraded" will full and correct highlighting.
• Lazy-load individual grammars (if it's not part of the default bundle) via a CDN and then perform highlighting as normal.

In summary:

No highlighting should be preferred over random highlighting for hinted snippets where SE has purposely chosen not to load a grammar module. Lazy-loading of grammars or manual hinting of alternatives (i.e., "java is a reasonable approx. of groovy") are some other options.

Also: no formatting may be a better choice for all snippets that have an explicit hint than cannot be resolved to any known language - though that's likely a larger discussion.

This was prompted by the Groovy discussion among others: What happened to Groovy syntax highlighting?

---

A small auto-detect primer and why this is a "worst-case" scenario for auto-detect.

Highlight.js auto-detection is based on analyzing a code snippet with all available language grammars and scoring its relevancy with each. The highest score "wins". While the keyword class or a variable named $blah is somewhat relevant in indicating a given piece of code might be PHP - the tag <?php is highly relevant, as it only ever appears in PHP templates. We're looking for which language seems to be the most "relevant" for a particular code snippet.

Let’s say we're asked to auto-detect the language and we find (in a perfect world) relevance scores something like:

C++:    9
SQL:    10
Java:   11
Groovy: 102

The code in question registers as 10x more "relevant" for Groovy, so it is highly likely this is a Groovy snippet. So what happens if the Groovy grammar isn't loaded - if we have no idea what Groovy code even is? You often end up with scoring much more like:

C++:  10
SQL:  9
Java: 10
Dart: 8
Go:   11

Our code now poorly matches whatever is left (since the correct answer [of Groovy] is no longer possible). The exact relevance values will of course change (depending on the snippet of code) and may not be this dramatic - but without the correct grammar loaded it's far more likely there is no clear winner... making the final language auto-detected much more of a coin toss.

This isn't a perfect example, but hopefully it is illustrative.

Answer 1

The first proposed alternative has been implemented. A change has now been made so Highlight.js will no longer guess the language if the language provided by the user is not currently supported. The highlighting will defer to the default of lang-none in this case. If no language is provided by the user, Highlight.js will still attempt to guess the language based upon the syntax of the code block.

It should be noted that this change only impacts new posts and edited posts. Existing posts have not been rebaked to implement this change.

Answer 2

This is a hard problem for us due to many years of tech debt and how most users expect the site to act. We have a few potential options here:

1. Support every language hljs supports. This is the ideal solution, but has some barriers due to our scale (network costs).
2. Turn off highlighting entirely unless a valid language is specified or a tag with a language set is added to the post. This essentially kills auto-highlighting entirely. This is the "most correct" solution, but it's far from ideal for reasons animuson mentioned.
3. Make all auto-highlighting fall back to a generic highlighter (prettify did this). This gets us part of the way, but is also far from ideal.
4. Make all auto-highlighting fall back to a generic highlighter if a more suitable language isn't found. This sounds like the most ideal solution, but what value do we use for the confidence interval on the language detection? Is hljs consistent in their relevance scores? Is this something we can even do reliably? There are a lot of unknowns here.

We'd like to explore this further at some point, and I personally think option 4 is interesting, if we can identify and address some of the unknowns. Having said that, I don't anticipate that we'll be able to tackle this anytime soon, so I'm tagging this as status-deferred for the time being.

Answer 3

So, you are discussing a few very different things in this post, and you have some false assumptions in there.

On automatic detection

Completely disabling automatic language detection in Highlight.js is completely off the table. It may be detrimental in the singular case that you have provided, but is not true for many other, much more popular languages.

The most common case is the combination of JavaScript, HTML, and CSS. Because these languages are so frequently mixed together in one question, we do not attempt to tell Highlight.js which language a code block might be, always preferring "default" for those tags. It is up to the highlighter to determine what type of code is in those blocks in a lot of cases, and simply leaving them as plain-text would definitely not be preferable there.

It doesn't sound like that's what you're really asking for here, though, despite some implication that it might be the catch-all solution.

On individual cases

Even if a language identifier is not explicitly aliased in the code, it is still possible to have a tag use another language by default. Any diamond moderator can change the default language for a tag to anything available - it is not hard-coded anywhere and does not need to match anything. If there is a better language that would serve as a default for a tag than "default" then raise the request on the per-site meta to have it set to that.

Tags can even be set to the "<none>" option if no syntax highlighting should ever be used for code blocks under that tag unless explicitly overridden. If you believe Groovy questions should by default not be highlighted at all over having faulty highlighting, then again that is a request that can be made on the per-site meta.

---

So given that, I'm not sure what there really is to do here. We would not turn it off completely because that would break detection for other tags and we already provide the tools to either set it to another similar language or none at all. You just have to ask for the tool to be used. Has anyone posted on Meta Stack Overflow for this case requesting the language hint for Groovy be changed to none?