What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

[tag:🍌]

As can readily be seen, tags seem to discriminate against bananas and don't want to play with them. I thought SO was a place of no discrimination against other origins, but it seems the ASCII-Landers are considered first-class citizens, while the Unicodians are seated below them. The same behaviour can be seen in the chat. Here's an evidence photo:

enter image description here (Properly censored, of course.)

I don't think SO should tolerate this behaviour and reeducate the tags. This might be especially problematic on some of the language sites, like French Language & Usage.

share|improve this question
    
getting ready for April 1st? –  Abyx Mar 24 '13 at 11:11
    
@Abyx this is serious, you don't use discriminate unicode. –  bamboon Mar 24 '13 at 11:12
1  
They're already working on internationalization features (see area51.stackexchange.com/proposals/23539/… for ex.). I'm not sure bananas are included in the plan though. –  Mat Mar 24 '13 at 11:12
7  
This is absulutely serious in my opinion. Either you support Unicode or not. If you don't, prepare for being accused of being middle-ages mind :D –  РСТȢѸФХѾЦЧШЩЪЫЬѢѤЮѦѪѨѬѠѺѮѰѲѴ Mar 24 '13 at 12:28
1  
@lechlukasz - but it would look a lot more serious with another sample. Anything but a banana. –  Henk Holterman Mar 24 '13 at 12:45
    
@HenkHolterman: ok, try with a 🐒 then. –  Mat Mar 24 '13 at 12:52
2  
I should add that even if you account for normalization and casing, you will miss things like sœur, soeur, SŒUR, SOEUR, ꜱɶᴜʀ, and ꜱᴏᴇᴜʀ needing to all match one another. If you use the primary-stength–only UCA comparison, however, they shall. Doing Unicode right is considerably harder than most people realize. –  tchrist Mar 24 '13 at 13:21
4  
Guys what are you all talking about? All I see are blank squares! No banana and no shoes. –  Shadow Wizard Mar 24 '13 at 13:38
    
@ShaWizDowArd !SVG –  hjpotter92 Mar 24 '13 at 14:37
    
@Back peeled banana! –  Shadow Wizard Mar 24 '13 at 14:52
2  
I, like @ShaWizDowArd, only see boxes. Strange that Android sees the monkies and bananas fine, but I do not... :( –  Makoto Mar 24 '13 at 15:43
3  
@Makoto 1. Chrome doesn't have font fallback; try a different browser. 2. A browser with font fallback still needs a font with those glyphs somewhere; try updating your system to get newer fonts. 3. Just assume any blank squares are bananas. –  R. Martinho Fernandes Mar 24 '13 at 15:47
    
What's with all the unicode weirdness? Most people can't even see them! –  ɥʇǝS Mar 24 '13 at 16:06
    
@Seth "all the unicode weirdness" is only as weird as what people use to communicate in writing. –  R. Martinho Fernandes Mar 24 '13 at 16:07
2  
@tchrist I kind of glossed the issue there. It does have some sort of thing they call font fallback. But it has a slew of reported bugs that make it seem almost as if it did not. And it seems most issues show up only on the Windows version. –  R. Martinho Fernandes Mar 24 '13 at 16:30

2 Answers 2

picture of 🐵 − (🙈 🙉 🙊) = 🍌⃠

Focussing on bananas and related monkey-business serves only to distract from the real issue here, which is that tags are restricted to only a few possible characters, the letters and digits from old-school 7-bit ASCII.

Bananas are symbols, not alphanumerics, and tags are a sort of identifier. So it makes reasonable sense to restrict tags to identifier characters.

It does not make sense to restrict those identifier characters to the ASCII range only. It is, however, somewhat justifiable, in that as soon as you allow for non-ASCII alphanumerics, all kinds of tricky issues arise. Here’s a brief sampling of things people might care to choose from:

Unicode samples

It is perfectly reasonable to hope to use non-ASCII alphanumerics. In practice, however, it becomes much trickier to get right than most people ever imagine. There are both technical and stylistic considerations that have to be dealt with first.

As I previously wrote in comments, it doesn’t take symbols to show the problem. You can use straight alphabetics.

Try making a tag like café, naïveté, jalapeños, tschüß. Now realize that you must also account for differences in casing (CAFÉ, NAÏVETÉ, JALAPEÑOS, TSCHÜSS) and normalization (unnormalized, normalization form C, normalization form D, etc), and you will see that this is seriously trickier than if it’s limited to the (cave-)manly ASCII alphabet. Probably it is best to compare the strings using an untailored primary-strength–only UCA comparision, which counts only letters not diacritics or case.

Maybe. You might want to tailor for language-based locales; for example, on German L&U, you would want to equate tschüß and tschueß, which is not the normal rule.

Also, even if you account for normalization and casing, you will miss things like sœur, soeur, SŒUR, SOEUR, ꜱɶᴜʀ, and ꜱᴏᴇᴜʀ needing to all match one another. If you use the primary-stength–only UCA comparison, however, they shall. Doing Unicode right really is considerably harder than most people realize.

Sisyphean Tasks

Finally, StackExchange is, for the most part at least, currently written in the Latin and Common scripts. The examples I have given so far are all in one of those two scripts. There may be sites where Greek or Arabic or what not are commonly used. Arabic is a problem because of bidirectional issues, but English text has traditionally allowed for Greek characters in it, and even that poses trickinesses.

Suppose someone wants to use ΣΊΣΥΦΟΣ or Σίσυφος as a tag. Default casing rules would cause the first one to be lowercased to σίσυφοσ instead of to σίσυφος, which is its own problem.

If you say that StackExchange sites are all English-only, except perhaps for the various non-English Language & Usage sites, then perhaps you will allow only English words, or at least, English-looking words. Not counting the various letters used by Old English or Middle English like thorn and ash and wynn, you still have lot to choose from, even in Modern English. The Oxford English Dictionary has entries for all of these terms in it, including the ones with Greek in them:

OED Unicode examples

So we can’t even use arbitrary English words, not just Old English words like wrǽððu or wrǽþþu, but Modern English words like the ones in the table above. That seems . . . unfortunate.


NOTE: I have for the most part used embedded images here instead of the literal code points, because I’m tired of people who read StackOverflow on their Dick-Tracy smart-watches complaining of wee boxes. All images used by the kind permission of the author.

share|improve this answer

Unicode support is enabled on a per-site basis. The language sites that need it have it turned on already, it is not enabled on Stack Overflow.

We haven't seen a large need for them there with nearly 5 million questions asked...to change the tag rules and introduce them at this point we'd have to see a very convincing argument.

share|improve this answer
    
🍌 should be a good enough argument. –  Xeo Mar 24 '13 at 16:53
7  
Is it Friday again already? –  Nick Craver Mar 24 '13 at 16:54
2  
This is clearly a bug, not a missing feature, at least in chat. It's the UTF-16 curse again. Or maybe the UCS-2 curse from JavaScript. I don't know which part of the code handles this. In chat, characters like ç or á can be used just fine in tags. The trouble comes up with non-BMP characters like 𐌀. A tag with one of these characters ends up as two question marks. Not even replacement characters, but question marks. I challenge the notion that that was designed. –  R. Martinho Fernandes Mar 24 '13 at 21:11
    
@R.MartinhoFernandes - While I agree it should be supported for chat.stackexchange.com which hosts chat for sites with tags that have unicode characters(and we'll look at that), I don't consider it a bug on chat.stackoverflow.com. On Stack Overflow these are invalid tags to begin with. –  Nick Craver Mar 24 '13 at 23:45
1  
@R.MartinhoFernandes Pretty sure that ever rendering single code points as a pair of question-mark oopsy-doopsy glyphs is a guaranteed UCS-2 (or mishandled UTF-16) bug. Seems to happen a lot here. –  tchrist Mar 25 '13 at 2:59
    
“The language sites that need it have it turned on already”. Oh? On French Language, it's technically possible to edit accented letters into tags, but this messes up the database, it seems to still use é and à as tag name separators (which is unfortunate as é is the most common accented letter in tag names in French). –  Gilles Apr 20 at 23:47
    
@Gilles those separators are no longer used in the database, you can read a detailed post about it here: marcgravell.blogspot.co.uk/2014/04/… –  Nick Craver Apr 21 at 0:12
    
Nice! So are meta.stackexchange.com/questions/224158/… and meta.french.stackexchange.com/questions/105/… status-completed? What should we do with the questions with broken tagging due to attempts at introducing étiquettes? –  Gilles Apr 21 at 0:18
    
@Gilles I'm not sure on the status for the bad data introduced from the admin tools - I'll relay to Marc who's heading up the changes there and he'll give it love tomorrow or Tuesday, we can very likely do some automated cleanup there –  Nick Craver Apr 21 at 0:21
    
Any word from Marc? Should we wait 6–8 weeks? –  Gilles May 11 at 20:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .