🍌 count is wrong in “More to go” for comments

Question

When you go to add a comment, the text incorrectly reads:

enter at least 15 characters

However, it does not enforce this, as only 8 characters are required. This can be readily demonstrated by adding a 🍌 character one at a time.

   Characters  Message                       String
   ----------  --------------                -----------
            0  enter at least 15 characters
            1  13 more to go...              🍌
            2  11 more to go...              🍌🍌
            3  9 more to go...               🍌🍌🍌
            4  7 more to go...               🍌🍌🍌🍌
            5  5 more to go...               🍌🍌🍌🍌🍌
            6  3 more to go...               🍌🍌🍌🍌🍌🍌
            7  1 more to go...               🍌🍌🍌🍌🍌🍌🍌
            8  584 characters left           🍌🍌🍌🍌🍌🍌🍌🍌
            9  582 characters left           🍌🍌🍌🍌🍌🍌🍌🍌🍌
           10  580 characters left           🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌  

The bug is that the code is not counting characters — Unicode code points — but merely some sort of tetranibble artifact of UTF-16 being mishandled as UCS-2.

Like UTF-8, UTF-16 is a variable-width encoding. The code needs to be updated to give accurate counts for actual characters. Nobody should have to know about tetranibbles, or believe lies about needing to enter 15 characters when entering just 8 characters is provably sufficient.

Answer 1

Since comments have to be stored in the database as nvarchar(600) the current behavior accurately measures the remaining number of . . . something. Thanks to UCS-2 and SQL Server's definition of the len() function, 300 bananas has a length of 600, so the count is accurate in terms of what we'll be able to store. The problem here is terminology and the element of surprise that some characters use more of the quota than others.

The simplest solution to the terminology problem would be to leave out the word "characters":

   Characters  Message                       String
   ----------  --------------                -----------
            0  enter at least 15
            1  13 more to go...              🍌
            2  11 more to go...              🍌🍌
            3  9 more to go...               🍌🍌🍌
            4  7 more to go...               🍌🍌🍌🍌
            5  5 more to go...               🍌🍌🍌🍌🍌
            6  3 more to go...               🍌🍌🍌🍌🍌🍌
            7  1 more to go...               🍌🍌🍌🍌🍌🍌🍌
            8  584 left                      🍌🍌🍌🍌🍌🍌🍌🍌
            9  582 left                      🍌🍌🍌🍌🍌🍌🍌🍌🍌
           10  580 left                      🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌  

Yes, it's a bit strange to see your counter go down by two when you've only entered one character. On the other hand, most comments on English-language sites won't have longer characters, such as the banana. For most comments, there won't be any difference between the number of characters remaining and the count displayed.

While researching how other sites manage this problem, I noticed that Twitter accurately counts the number of bananas:

Ideally, we'd do the same thing. But in order to do that, the database field would need to be sized to correctly hold all possible comments. I believe that can be accomplished by using supplementary character collations. The change should be easy enough, but testing the change and not breaking other parts of the system will not be trivial. And, of course, the countdown display would need to be adjusted as well. None of this seems worth the hassle on English-language sites, but it could very well be somewhat more important on スタック・オーバーフロー.

Answer 2

There's a bug with the char-handling only if we assume that they had actually wanted to count Unicode codepoints. But would fixing the code to count Unicode codepoints solve the problem that characters don't match users' expectations?

No, because even after fixing codepoints, and assuming that everything is perfect and "bug-free", Unicode considers q̊q̊q̊ to be 6 codepoints, whereas it is really only 3 user-perceived characters.

So perhaps, the "bug" or problem here is that we are using Unicode. Unicode codepoints do not map directly (one codepoint for each user-perceived character) onto the writing systems taught in schools, because Unicode claims that it's too hard to do so for all languages. As such, a user-perceived character is not the same as Unicode codepoint.

The easiest solution would be to ignore the problem. Users are already used to "weird behavior" for characters outside the so-called "English range". No one's going to be too surprised.

The easier solution is to change the message from "10 characters left" to "10 Unicode codepoints left". (Needless to say, this kind of information is pretty useless to users.) Another easy solution is to convey a vague message, e.g. "about 10 characters left". (Like how Google Search often gives a vague estimate.)

The (extremely) hard but proper solution will be to allow users to enter only user-perceived characters. This is basically white-listing every character that's going to be supported, as opposed to blacklisting anomalies one by one. For example:

• If the use case is to accept English written language, the white-list will accept only the characters a to z and A to Z. Probably adding more if needed (., ,, (, ), !, @, etc) on a case-by-case basis. The white-list will prevent any chances for anomalies to happen.

• If the use case is to accept Japanese written language, the white-list will accept only the 1006 + 1130 + ~40 + ~40 characters. Probably adding more if needed (e.g. supporting obsolete hiragana/katakana/kanji, and non-kanas like 『, 〲, ゝ, etc.) on a case-by-case basis.

• If the use case is to accept PRC Chinese written language, the white-list will accept only the 8105 characters, probably adding more if needed on a case-by-case basis.

• And so on.

With pure white-listing, we can control every single input such that there would be no way for users to enter "weird" stuff like emoji characters and etc.

Answer 3

Actually the bug is that it says "characters" when it should say "bytes".

But that would be ridiculous.

I think everyone knows what the intent is here. Nothing to change.