The introduction of multiple language-specific sites to the SE network has brought to light the shortcoming of the current tag system: lack of support for non-English characters. Non-English characters are either silently dropped or converted to the nearest English equivalent. I propose that the system be changed to allow language-specific sites a greater degree of self-determination in the matter of tagging. Some of the affected sites are: Russian.SE, Spanish.SE, Chinese.SE, Japanese.SE, French.SE, Judaism.SE.

This question has initially been about the Russian.SE site, but seeing how other sites are affected by it I decided to broaden it.

See also:

share|improve this question
@kotekzot site specific feature requests/bug reports should go on site metas. SE staff monitors all site metas. Particularly private betas and early public betas when there are no mods. –  wax eagle Jun 14 '12 at 2:19
@kotekzot It's a conscious choice. For your information, the existing language sites have taken different decisions about tags: French Language settled on French (with English synonyms), German Language settled on English (with German synonyms), Spanish Language initially went with English but changed to Spanish, Chinese Language and Japanese Language both use English. –  Gilles Jun 14 '12 at 2:22
And it's worth noting that those posts actually have more visibility to the people who need to see them on site metas than they do here as here they may get lost in the shuffle, but on site metas where they have much less traffic the community team is more likely to see them and address them on an individual site meta. –  wax eagle Jun 14 '12 at 2:22
@Gilles you don't see the pattern here? Sites devoted to languages that use a latic script have tags (or synonyms) in their own language, sites that don't - don't. That seems like a technological problem to me. –  kotekzot Jun 14 '12 at 2:26
@kotekzot The pattern is not clear. For a long time, French Language was the only site with non-English tag names that people saw. I do think it's the right way to go, but it is not yet established. –  Gilles Jun 14 '12 at 2:29
@Gilles it is pretty clear if you consider synonyms. Sites dedicated to languages that aren't lucky enough to use a latin script don't get a choice. –  kotekzot Jun 14 '12 at 2:35
@casperOne, this could be edited as general question about allowing Unicode tags and synonyms. –  theUg Jun 14 '12 at 2:37
@Gilles, even Latin script language sites do not seem to be using language-specific characters like eñe in Spanish. –  theUg Jun 14 '12 at 2:41
@theUg good idea, done. –  kotekzot Jun 14 '12 at 2:55
@Gilles The Japanese site uses English tags because there isn't really a feasible alternative. You can't auto-convert 漢字 to a supported character like you can auto-convert é to e, you have to use the English word. –  Troyen Jun 14 '12 at 4:26
@Troyen thanks, I've added French.SE to the list of affected sites and linked the 2 questions. –  kotekzot Jun 14 '12 at 5:53
Judaism.SE would also certainly want it, as would many proposals still in a51 (Hebrew, Yiddish, etc.) –  HodofHod Jun 14 '12 at 12:40
Spanish.SE would also like this:…… this is important as we want our tags to be in Spanish. –  JoulSauron Jun 15 '12 at 14:02
add comment

1 Answer

up vote 12 down vote accepted

Our original implementation of tags was written with stackoverflow (and similar sites) in mind, and due to technical reasons would not support much by way of non-latin characters. This has seen a lot of work lately, in particular changing the implementation so that it theoretically supports a wider character set. So: we're working on it.

We still have a little way to go, and we might need to still think about what characters we allow per-site (in particular, to avoid 7 versions of the same tag, with/without accents - there's only so much synonyms can do).

Update: this is now enabled on russian.SE, as an initial proving ground, and will be released to additional sites where it makes sense assuming no huge "eek" issues are reported.

share|improve this answer
Good to know it's being worked on! Don't suppose you could give us a ballpark estimate of when it would be ready? –  kotekzot Jun 14 '12 at 6:37
@kotekzot there's a running joke here on meta, that every time estimate is always 6-8 weeks, regardless of magnitude or existing time elapsed... –  Marc Gravell Jun 14 '12 at 6:48
I'll make sure to come a-knockin' in 8 weeks then! –  kotekzot Jun 14 '12 at 6:50
In French, it would be fine if you couldn't have two tags that differ only by accentuation. There are words that differ only by accentuation, but they wouldn't tend to be tag names. I can't speak for other languages. –  Gilles Jun 14 '12 at 7:33
"technical reasons would not support much by way of non-latin characters" intrigues me as to how you're currently storing them. varchar with some obscure collation? –  AakashM Jun 14 '12 at 8:23
@Gilles - I recall that (according to French grammar) if a word is capitalized, then accentuation is not required. –  Adam Mosheh Jun 14 '12 at 17:36
@AdamMosheh cringe No, accents are required. The lack of accents is tolerated if the typographical medium doesn't permit it, and word games such as crosswords ignore accents, but spelling rules apply to words in uppercase as much as to words in lowercase. (Besides, we were speaking about tags, which are in lowercase.) –  Gilles Jun 14 '12 at 17:45
@AdamMosheh Omitting accent's on uppercase letter's is like the grocer's apostrophe in English. Some people do it, but its not to everyones liking. –  Gilles Jun 14 '12 at 17:45
@Gilles careful - I realise the above is intentional, but you accidentally got one of them right –  Marc Gravell Jun 14 '12 at 18:18
@AakashM you really want to know? Well... We were kind of abusing sql-server's full-text search by tricking it into working on a munged version of the combined tags. This let us do a range of "all of" / "any of" tag-based queries against a fast index. This falls to pieces when multiple cultures get involved - our hacks no longer work. Most of the code has been tweaked to compensate for this (meaning: not using this hacky hackety hack-hack): in many ways, it is now a case of removing code that is there to enforce simple tags –  Marc Gravell Jun 14 '12 at 18:22
It seems to me that canonicalizing to the lowercase form of the C normalization form is not going to be enough, because you aren’t actually doing Unicode casefolding for matches then. So you will have a problem getting says tchüss and tshüß to match, or Σίσυφος and σίσυφος and σίσυφοσ. I suggest running everything through a level-1 comparison under the Unicode Collation Algorithm, as the primary sort keys ignore both full Unicode case as well as diacritics. You already seem to ignore case correctly on chat searches, in that looking for tschüß finds TSCHÜSS correctly. –  tchrist Oct 1 '12 at 17:22
@tchrist happy to get the feedback, but in many ways we dodge the most complex scenarios through a: auto-complete, b: synonyms, and c: the human factor. If there's something I'm missing, I'd love to hear the scenario (in the context of tags). Tagging != text search. –  Marc Gravell Oct 1 '12 at 17:56
This answer needs updating. –  msh210 Apr 3 '13 at 7:49
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .