55

When you go to add a comment, the text incorrectly reads:

enter at least 15 characters

However, it does not enforce this, as only 8 characters are required. This can be readily demonstrated by adding a 🍌 character one at a time.

   Characters  Message                       String
   ----------  --------------                -----------
            0  enter at least 15 characters
            1  13 more to go...              🍌
            2  11 more to go...              🍌🍌
            3  9 more to go...               🍌🍌🍌
            4  7 more to go...               🍌🍌🍌🍌
            5  5 more to go...               🍌🍌🍌🍌🍌
            6  3 more to go...               🍌🍌🍌🍌🍌🍌
            7  1 more to go...               🍌🍌🍌🍌🍌🍌🍌
            8  584 characters left           🍌🍌🍌🍌🍌🍌🍌🍌
            9  582 characters left           🍌🍌🍌🍌🍌🍌🍌🍌🍌
           10  580 characters left           🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌  

The bug is that the code is not counting characters β€” Unicode code points β€” but merely some sort of tetranibble artifact of UTF-16 being mishandled as UCS-2.

Like UTF-8, UTF-16 is a variable-width encoding. The code needs to be updated to give accurate counts for actual characters. Nobody should have to know about tetranibbles, or believe lies about needing to enter 15 characters when entering just 8 characters is provably sufficient.

  • 20
    🍌🍌🍌🍌🍌🍌🍌🍌 – user230705 Apr 29 '15 at 0:02
  • 4
    You're right! 🍌 – user230705 Apr 29 '15 at 0:02
  • 2
    🍌darn... couldn't stop myself, it's true 🍌🍌🍌🍌 – vsoftco Apr 29 '15 at 0:06
  • 3
    Orange you glad I didn't... wait... – Jamal Apr 29 '15 at 1:06
  • 4
    Wow - this is really fruity, and we should certainly peel such things from the char count! We would be bananas not to, and might look like we're being yellow to not enforce this. It Musa be fixed berry quickly! – James Apr 29 '15 at 1:13
  • 6
    ​​​​​​​​​​​​​​​ – Κ‡olɐǝz ǝΙ₯Κ‡ qoq Apr 29 '15 at 3:34
  • 8
    πŸŒΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ΰΉ‰ – Shadow The Princess Wizard Apr 29 '15 at 6:09
  • 2
    +1 just for the art, actually it's really too minor to be worth the amount of time needed to fix. So we'll have more useless comments, not the end of the world. – Shadow The Princess Wizard Apr 29 '15 at 6:10
  • 1
    @Pacerier k 🍌 🍌 – user230705 Apr 29 '15 at 8:14
  • 2
    Up vote from me for the banana in the title - it was the only reason I read your question. Then another +1 for tetranibbles. But also -1 for every fruit I'm going to have to edit out of titles now that you've started it. – slugster Apr 29 '15 at 10:16
  • 6
    🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌I should be able to fit more banana here – user1228 Apr 29 '15 at 12:56
  • 1
    What I'm now trying to figure out is if this is serious or fun... – M.A.R. Apr 29 '15 at 17:15
  • 6
    @Won't Notice the funny goobers that snuck in between some of your 🍌s. That's a bug, too: it makes them very unappetizing. After all, who knows where those goobers came from? They might even be contagious. – tchrist Apr 29 '15 at 17:29
  • 1
    Yes​​​​​​​​​​​​ – JonasCz - Reinstate Monica May 1 '15 at 15:13
  • 2
    🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌 – EKons Aug 28 '16 at 17:57
25

Since comments have to be stored in the database as nvarchar(600) the current behavior accurately measures the remaining number of . . . something. Thanks to UCS-2 and SQL Server's definition of the len() function, 300 bananas has a length of 600, so the count is accurate in terms of what we'll be able to store. The problem here is terminology and the element of surprise that some characters use more of the quota than others.

The simplest solution to the terminology problem would be to leave out the word "characters":

   Characters  Message                       String
   ----------  --------------                -----------
            0  enter at least 15
            1  13 more to go...              🍌
            2  11 more to go...              🍌🍌
            3  9 more to go...               🍌🍌🍌
            4  7 more to go...               🍌🍌🍌🍌
            5  5 more to go...               🍌🍌🍌🍌🍌
            6  3 more to go...               🍌🍌🍌🍌🍌🍌
            7  1 more to go...               🍌🍌🍌🍌🍌🍌🍌
            8  584 left                      🍌🍌🍌🍌🍌🍌🍌🍌
            9  582 left                      🍌🍌🍌🍌🍌🍌🍌🍌🍌
           10  580 left                      🍌🍌🍌🍌🍌🍌🍌🍌🍌🍌  

Yes, it's a bit strange to see your counter go down by two when you've only entered one character. On the other hand, most comments on English-language sites won't have longer characters, such as the banana. For most comments, there won't be any difference between the number of characters remaining and the count displayed.

While researching how other sites manage this problem, I noticed that Twitter accurately counts the number of bananas:

Twitter counts bananas

Ideally, we'd do the same thing. But in order to do that, the database field would need to be sized to correctly hold all possible comments. I believe that can be accomplished by using supplementary character collations. The change should be easy enough, but testing the change and not breaking other parts of the system will not be trivial. And, of course, the countdown display would need to be adjusted as well. None of this seems worth the hassle on English-language sites, but it could very well be somewhat more important on スタック・γ‚ͺーバーフロー.

  • 1
    Your solution seems reasonable to just stop mentioning characters. I was thinking that there might be some database issue here. For my work we use UTF-8 storage; I didn’t know anyone still used the old UCS-2; it takes so much space! Good call about there being more important Unicode problems to worry about for the Japanese site. With all the East_Asian_Width=Wide characters throwing off notions of what fits where, plus the U+3099 and U+309A combining Kana voice marks that must not be split, things are definitely trickier than on an English-language site. Good luck with that, and thank you. – tchrist Apr 29 '15 at 11:15
  • 20
    This is bananas. – Adam Lear Apr 29 '15 at 21:40
4

There's a bug with the char-handling only if we assume that they had actually wanted to count Unicode codepoints. But would fixing the code to count Unicode codepoints solve the problem that characters don't match users' expectations?

No, because even after fixing codepoints, and assuming that everything is perfect and "bug-free", Unicode considers q̊q̊q̊ to be 6 codepoints, whereas it is really only 3 user-perceived characters.

So perhaps, the "bug" or problem here is that we are using Unicode. Unicode codepoints do not map directly (one codepoint for each user-perceived character) onto the writing systems taught in schools, because Unicode claims that it's too hard to do so for all languages. As such, a user-perceived character is not the same as Unicode codepoint.

The easiest solution would be to ignore the problem. Users are already used to "weird behavior" for characters outside the so-called "English range". No one's going to be too surprised.

The easier solution is to change the message from "10 characters left" to "10 Unicode codepoints left". (Needless to say, this kind of information is pretty useless to users.) Another easy solution is to convey a vague message, e.g. "about 10 characters left". (Like how Google Search often gives a vague estimate.)

The (extremely) hard but proper solution will be to allow users to enter only user-perceived characters. This is basically white-listing every character that's going to be supported, as opposed to blacklisting anomalies one by one. For example:

  • If the use case is to accept English written language, the white-list will accept only the characters a to z and A to Z. Probably adding more if needed (., ,, (, ), !, @, etc) on a case-by-case basis. The white-list will prevent any chances for anomalies to happen.

  • If the use case is to accept Japanese written language, the white-list will accept only the 1006 + 1130 + ~40 + ~40 characters. Probably adding more if needed (e.g. supporting obsolete hiragana/katakana/kanji, and non-kanas like γ€Ž, γ€², ゝ, etc.) on a case-by-case basis.

  • If the use case is to accept PRC Chinese written language, the white-list will accept only the 8105 characters, probably adding more if needed on a case-by-case basis.

  • And so on.

With pure white-listing, we can control every single input such that there would be no way for users to enter "weird" stuff like emoji characters and etc.

  • It gets worse. – tchrist Apr 29 '15 at 4:51
  • Why not count graphemes as user-perceived characters? They are well-defined formally so it is easy to code. That also solves the combining character bug in my "it gets worse" link above. This all reminds me of the atomic character bug at some level: splitting what you shouldn't, thanks to miscounting. – tchrist Apr 29 '15 at 5:20
  • @tchrist, Do you mean "grapheme" as defined in Unicode glossary, or "grapheme" in general according to graphemics? If you are talking about Unicode graphemes, then no, it wouldn't solve them problem because z̐ is an invalid character; it simply doesn't exist in any "real" language, yet according to your method you would still consider it valid. Same for zΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•Μš. – Pacerier Apr 29 '15 at 5:49
  • 1
    Pure white-listing is the only solution as per mentioned above. But needless to say this is a "fantasy" problem no one can be bothered to solve. Which is why comments can always be as short as zero characters or as tall as one full pΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“ΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”ΜΝœΝ•Ν“Μ–Μ’Μ“Μ”Μ±Μ²Μ³Μ΄Μ΅ΜΆΜ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ‘Μ–Μ’Μ“Μ”Μ•ΜšΜΝœΝ•Ν“Ν”Μ‘Μ“Μ“Μ“Μ“Μ“Μ“Μ“Μ‘Μ–Μ’Μ“Μ”Μ•Μšage . – Pacerier Apr 29 '15 at 5:53
  • @Pacerrier I don't know why you are talking about alphabets. That's a red herring. People can compose whatever ad hoc graphemes they wish. So of course I am talking about Unicode extended grapheme clusters. Those are open-ended by design. – tchrist Apr 29 '15 at 17:33
  • 1
    @tchrist, You were talking about counting, which is why I raised those points about allocating user input into boxes (white-listing). With white-listing, you have a definite count. How much count should we assign to "tΜ’Ν’hΝ‘ΜšΝ„isͯ͜͜͜͜͜"? The user is going to count it as 4 because it is indeed really only 4 user-perceived characters. Is it right for the system to count it as 4? – Pacerier Apr 30 '15 at 7:27
-5

Actually the bug is that it says "characters" when it should say "bytes".

But that would be ridiculous.

I think everyone knows what the intent is here. Nothing to change.

  • 3
    How is it ridiculous? It's objectively wrong. – bjb568 Apr 29 '15 at 0:05
  • 10
    β€œBytes”? Like seriously? You seem to have misunderstood Unicode. Pray tell, just which byte were you planning on storing something like € in anyway? The byte 0xA4 for € works only in the ISO-8859-15 encoding β€” and down that road lies madness. That € takes 3 bytes in this web page’s UTF-8, so if you had your way it would be what, 3 less in the count? Nobody should be talking about bytes, especially when calling them characters, which they aren’t. Only actual logical characters should be mentioned, not incorrect counts related to low-level bit layouts used in this or that encoding system. – tchrist Apr 29 '15 at 1:05
  • 2
    @tchrist, Actual logical characters? So who decides what's actual logical characters? Unicode is a solution to a problem, but it's not perfect. Unicode characters map onto a language called the Unicode language, they try to resemble real-world languages but they're only about as identical as twins are. Just look at my next comment, written in 15 Unicode "logical characters" but 0 real-world characters: – Pacerier Apr 29 '15 at 3:08
  • 2
    @Pacerier You do not count € as three characters. It’s just one. No user cares that that is 24 bits. To the user it is one character, and that is all that matters. Classic programmer UX failure. – tchrist Apr 29 '15 at 3:13
  • 1
    @tchrist, You do not count "͏͏͏͏͏͏͏͏͏͏͏͏͏͏͏" as 15 characters. It's just zero. No use cares that that's 15 unicode characters. To the user it is zero letters, and that is all that matters. Classic Unicode UX failure. ......... 2) This is not about encoding, because the layer before encoding is already flawed. The writing system provides an interface to the user through user-perceived characters. These are called "letters", not characters. "characters" is merely an abstraction to make handling "letters" easier. No user cares about "characters". – Pacerier Apr 29 '15 at 3:45
  • 2
    @tchrist, Counting column-widths don't make sense for many languages, e.g. Arabic. What people learn in schools is called the writing system, and each language has it's own writing system though there are definitely similarities. The writing system is what Unicode should match. Not some arbitrary concept like "characters" which no user understands. – Pacerier Apr 29 '15 at 3:59
  • 1
    @tchrist: No, I have not "misunderstood Unicode". Conversely, you have misunderstood my answer. – Lightness Races with Monica Apr 29 '15 at 9:40
  • 2
    Bytes aren't exactly correct either. "The storage size, in bytes, is two times the actual length of data entered + 2 bytes." We paper over that by reporting a number that theoretically means something to the user. Unfortunately UCS-2 is a leaky abstraction. – Jon Ericson Apr 29 '15 at 15:36
  • 1
    @JonEricson: That quote talks about a leaky abstraction; the 2 bytes are null termination, which quite rightly isn't included in SO's count of remaining available bytes in the comment box. :) – Lightness Races with Monica Apr 29 '15 at 16:08
  • 2
    The count below the comment box doesn't count bytes. It counts UTF-16 code units. Each character you type takes either two or four bytes in UTF-16. – yellowantphil Apr 29 '15 at 17:39
  • @JonEricson, The leaky abstraction starts right from Unicode itself. It's supposed to map to writing systems used in living languages, but didn't managed to do it in a non-leaky way. – Pacerier Apr 30 '15 at 7:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .