What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

I have characters which are outside 7-bit ASCII in my username, "Jakub Narębski". Characters outside US-ASCII are dropped from the full profile URL: http://meta.stackoverflow.com/users/130454/jakub-narbski (observe that it is 'narbski', not 'narebski' or 'narębski').

I'm not quite sure if it is a bug or not. I think that the last part of URL is purely informational; http://meta.stackoverflow.com/users/130454 works as well, I suspect to allow to change one's name field.


Added 2009-07-30:
This feature request, in the form of stripping diacritical marks, got implemented (). Wouldn't it be a good idea to separate this Unicode transliteration code and put it as ASP.NET snippet or mini-library somewhere?

See also: This Is America, Take Your Unicode Somewhere Else blog post by Ted Dziuba, which mentions Text::Unidecode Perl module (which does US-ASCII transliterations of Unicode text), and mentions Stack Overflow in passing.


Added 2009-08-30
Perl 6 has :ignoreaccent modifier, and there is also the Text::Unaccent Perl module (which uses unac C library).

share|improve this question
1  
The right way to do accent-insensitive comparisons is by comparing things at the primary strenth in the Unicode Collation Algorithm. That is what it is there for. You will never get them all otherwise. –  tchrist Apr 29 '11 at 4:15

2 Answers 2

up vote 26 down vote accepted

Here's what we have in the substitution table

public static string RemapInternationalCharToAscii(char c)
{
    string s = c.ToString().ToLowerInvariant();
    if ("àåáâäãåą".Contains(s))
    {
        return "a";
    }
    else if ("èéêëę".Contains(s))
    {
        return "e";
    }
    else if ("ìíîïı".Contains(s))
    {
        return "i";
    }
    else if ("òóôõöøőð".Contains(s))
    {
        return "o";
    }
    else if ("ùúûüŭů".Contains(s))
    {
        return "u";
    }
    else if ("çćčĉ".Contains(s))
    {
        return "c";
    }
    else if ("żźž".Contains(s))
    {
        return "z";
    }
    else if ("śşšŝ".Contains(s))
    {
        return "s";
    }
    else if ("ñń".Contains(s))
    {
        return "n";
    }
    else if ("ýÿ".Contains(s))
    {
        return "y";
    }
    else if ("ğĝ".Contains(s))
    {
        return "g";
    }
    else if (c == 'ř')
    {
        return "r";
    }
    else if (c == 'ł')
    {
        return "l";
    }
    else if (c == 'đ')
    {
        return "d";
    }
    else if (c == 'ß')
    {
        return "ss";
    }
    else if (c == 'Þ')
    {
        return "th";
    }
    else if (c == 'ĥ')
    {
        return "h";
    }
    else if (c == 'ĵ')
    {
        return "j";
    }
    else
    {
        return "";
    }
}

Updated a few times now; does this cover it?

share|improve this answer
1  
Māori has ā. There's also æ and œ (--> ae and oe) though I'm not sure if they turn up in names. You could add Þ for when we get our first Anglo-Saxon users :-) –  John Fouhy Jul 30 '09 at 22:57
    
I also can't find the uppercase variants L with stroke (Ł) and D with stroke (Đ). –  Marcel Korpel Aug 31 '10 at 12:02
    
@John: There's @aether, which turns a single character to "ae" accoring to meta.stackoverflow.com/questions/38288/… –  Andrew Grimm Apr 28 '11 at 2:14
18  
@Jeff, why in the world are you hardcoding all those? Use the Unicode Collation Algorithm and compare at the primary strength, which works even better than canonically decomposing and then removing Grapheme_Extend characters. The UCA1("ð") is the same as UCA1("d"), etc. Otherwise you fight a losing battle. Use the Unicode facilities; don’t hardcode stuff. –  tchrist Apr 29 '11 at 4:12
    
@tchrist this algorithm is quite fast, and it needs to be since it is potentially called hundreds of times per page. Do you have any c# source code examples? –  Jeff Atwood Apr 29 '11 at 5:07
    
@Jeff: Well, the UCA is supposed to be fast, too; there are big lookup tables, which may be a memory issue. But I don’t do C#; I'm not a Microsoft guy. I could have probably winged it in Java if you’d asked since I do Java stuff from time to time, but my primary language is Perl. I agree you’re doing better than the naïve stripping of Mark characters that I normally see done. It just seems painfully ad hoc, and we have tables for this stuff. –  tchrist Apr 29 '11 at 12:08
    
@Jeff: is there a reason you went with If-Else-If vs Switch? (did you test/compare?) posts here (e.g. stackoverflow.com/questions/445067/if-vs-switch-speed) and elsewhere (e.g. blackwasp.co.uk/SpeedTestIfElseSwitch.aspx) indicate that switch is generally faster. –  Faust Jun 8 '11 at 11:26
1  
@faust kind of irrelevant for a number of reasons, the primary reason is that the sequence of if..then here is ordered by frequency that those letters occur in English. So it hits the most common cases first, and almost never goes all the way to the bottom. –  Jeff Atwood Jul 13 '11 at 12:07
1  
@Jeff I've got a version using a unicode collation function in .net on this answer : stackoverflow.com/questions/25259/… - I haven't measured the performance impact, but coverage is better, although I still need an exceptions table. I also delay load the hyphens which might be useful even if you prefer the hardcode lookup table for perf. –  DanH Jul 19 '11 at 12:59
    
@dan great tip, I was looking for that code! I'll check it out. –  Jeff Atwood Jul 20 '11 at 5:15
    
@Jeff will be v. interested in any results if you benchmark it, especially as you have real data to work against. I'm pretty curious if the delay hyphen makes any diff for you by itself. –  DanH Jul 22 '11 at 11:02
    
That doesn't work on my name. Please contain uppercase versions of that chars.(For example: Ç ) –  Murat Corlu Jan 10 '13 at 13:04
    
Using ToLowerInvariant might result in some strange behavior in character normalization, as "İ".ToLowerInvariant() gives us "İ" (Yes, the same character: Unicode Character 'LATIN CAPITAL LETTER I WITH DOT ABOVE' (U+0130)) The issue seems to be related to yet another Turkish Locale Horror (love the article, btw.), and according to this article, using ToUpperInvariant for string normalization is the recommended approach. What do you think? –  mono blaine Dec 27 '13 at 23:46
    
I guess it's no use struggling: "ı".ToUpperInvariant() is ı –  mono blaine Dec 28 '13 at 0:27
2  
I see this problem persists:my comment from three years ago still applies in full. –  tchrist May 4 at 20:55

Those characters are often encoding in URLs, but it works just as well to drop them, too. That last part is purely informational (SEO ftw), so if you felt like it, you could even call yourself Mary, Queen of Scots and it still would go to the right page.

share|improve this answer
1  
well, sort of. We realized that having lots of people link to a question as /12345/blah-foo-bar and /12345/baz-shell and /12345/extra-words was confusing Google in some cases, so we force-redirect to the 'correct' title now. –  Jeff Atwood Mar 15 '11 at 5:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .