35

I see dead charac�...

TD;DR: Just like how when you cut a worm in half, you don’t get two worms because both parts die being non-viable each on its own, so too does cutting a character in half result in dead characters. Characters are atomic and must never be split.


In chat when something long is starred, the version that appears on the starboard can be randomly truncated in the middle of a character:

bad chat truncation

That’s happening because the code is incorrectly counting off 147 code units not 147 code points and splicing the data right there in the middle of a character and so leaving dead character garbage after the knife has had its way with them.

Now, if the code unit size were 32 bits this wouldn’t be so bad since any legal Unicode code point can fit in 32 bits. But they’re working in awkward UTF-16 with its nasty 16-bit code units. So truncating at 147 16-bit code units in UTF-16 is just as wrong as truncating at 147 8-bit code units — that is, bytes — in UTF-8 would be!

In both cases the result is the same; this results in an illegal sequence which the UTF-8 converter for output quite correctly spaces out the airlock by exchanging it with U+FFFD REPLACEMENT CHARACTER. Hence, dead characters.

Please stop counting code units not code points. That’s as bad as confusing pointers with integers. UTF-16 is not UCS-2. UTF-16 is a variable-width encoding just like UTF-8 is, and must be treated in the same fashion. In anything that counts “characters” you must minimally count code points not code units. This includes length checks like for minimum or maximum characters in a line. That happens all over the SE network in anything that thinks it’s counting characters but in fact only counts code units.

By truncating at code-point boundaries at least you won’t be generating illegal Unicode sequences that make the UTF-8 encoder throw in nasty U+FFFD REPLACEMENT CHARACTERs at us. You are, however, still at risk of truncating in the middle of a grapheme sequence due to its combining characters. But at least you will stop mangling the Unicode into illegal sequences and mercilessly murdering the characters.

No more dead characters, pretty please.

  • Related. – tchrist Apr 21 '15 at 12:25
  • 4
    This bug also occurs in the comments here. 🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱🐱 – yellowantphil Apr 27 '15 at 2:09
  • So... issue isn't specific to chat... – Braiam Apr 27 '15 at 2:10
  • In my cat spam above, every sequence that looks like �� is U+FFFD U+200C U+200B U+FFFD. Stack Exchange is adding ZERO WIDTH NON-JOINER and ZERO WIDTH SPACE for some reason. U+FFFD is the replacement character. – yellowantphil Apr 27 '15 at 2:17
  • 1
    @Braiam Right: it’s specific to people who mistreat the variable-length encoding UTF-16 as though it were the obsolete fixed-width encoding UCS-2. Not naming any names. It’s part of the lie they tell you when they say 255 characters left in a comment. Those aren’t: they’re lying because they are not counting characters but say that are. – tchrist Apr 27 '15 at 2:18
  • @yellowantphil It’s the same old, same old: somebody was misinformed that UTF-16 was a fixed-width encoding scheme, when it is a variable-width one just like UTF-8. So they are splitting up atomic characters and sticking their U+200C U+200B between them creating illegal code points, which the UTF-8 converter tosses out the airlock just as it is supposed to, leaving U+FFFD in their stead. All SO code that mistreats code units as code points needs to be audited and corrected. They really should be dealing with abstract characters not with physical encodings anyway; that’d fix this. – tchrist Apr 27 '15 at 11:49
9

As far as I can tell this has been fixed with the new version of the separator code.

enter image description here

However this does not apply to chat, yet.

(*) no kittens were harmed in the making of this patch.

(**) unfortunately, we cannot guarantee not to have invoked Cthulhu, the ancient, in the process.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .