Unicode mishandling bug in chat starboard truncati�

Question

I see dead charac�...

TD;DR: Just like how when you cut a worm in half, you don’t get two worms because both parts die being non-viable each on its own, so too does cutting a character in half result in dead characters. Characters are atomic and must never be split.

---

In chat when something long is starred, the version that appears on the starboard can be randomly truncated in the middle of a character:

That’s happening because the code is incorrectly counting off 147 code units not 147 code points and splicing the data right there in the middle of a character and so leaving dead character garbage after the knife has had its way with them.

Now, if the code unit size were 32 bits this wouldn’t be so bad since any legal Unicode code point can fit in 32 bits. But they’re working in awkward UTF-16 with its nasty 16-bit code units. So truncating at 147 16-bit code units in UTF-16 is just as wrong as truncating at 147 8-bit code units — that is, bytes — in UTF-8 would be!

In both cases the result is the same; this results in an illegal sequence which the UTF-8 converter for output quite correctly spaces out the airlock by exchanging it with U+FFFD REPLACEMENT CHARACTER. Hence, dead characters.

Please stop counting code units not code points. That’s as bad as confusing pointers with integers. UTF-16 is not UCS-2. UTF-16 is a variable-width encoding just like UTF-8 is, and must be treated in the same fashion. In anything that counts “characters” you must minimally count code points not code units. This includes length checks like for minimum or maximum characters in a line. That happens all over the SE network in anything that thinks it’s counting characters but in fact only counts code units.

By truncating at code-point boundaries at least you won’t be generating illegal Unicode sequences that make the UTF-8 encoder throw in nasty U+FFFD REPLACEMENT CHARACTERs at us. You are, however, still at risk of truncating in the middle of a grapheme sequence due to its combining characters. But at least you will stop mangling the Unicode into illegal sequences and mercilessly murdering the characters.

No more dead characters, pretty please.

Answer 1

As far as I can tell this has been fixed with the new version of the separator code.

However this does not apply to chat, yet.

^{(*) no kittens were harmed in the making of this patch.}

^{(**) unfortunately, we cannot guarantee not to have invoked Cthulhu, the ancient, in the process.}