What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

Attempting to link to a site that uses non-alphanumeric characters in its domain name is incorrectly encoded to Hexadecimal "safe" characters instead of punycode.

For instance, clicking on this links to encodes the link instead of linking to http://domaintest.xn--q9jyb4c/.

Also, attempting to write something with non-alphanumeric characters does not recognize anything past those characters as part of the URL.

share|improve this question
1  
I must be missing something, as I end up in the same place, and the same URL with both links. –  Oded Jun 27 at 12:57
2  
@Oded depends on the browser. Works in Chrome, the first link breaks in IE11 and FF30 for me. Second link brings me to a testpage in all browsers. –  rene Jun 27 at 13:06
    
Still wondering if this is a problem with the software of Stack Exchange? –  rene Jun 27 at 13:09
2  
What browser are you using? Both IE and Firefox have the issue for me. This is how my page source looks. Note the difference in the URLS. No issue in Chrome though. –  Yserbius Jun 27 at 13:09

1 Answer 1

up vote 5 down vote accepted

The real problem here is not the lack of punycode encoding, but rather the inappropriate application of percent-encoding to the domain part of the URL. Simply leaving the Unicode characters in the domain name un-encoded, as in:

<a href="http://www.domaintest.みんな/">this</a>

makes the URL into a valid IRI, which all modern browsers support.

The fact that Chrome, apparently, also accepts the invalid percent-encoded domain names (and decodes and re-encodes them as punycode) does not make this behavior correct or desirable.


Addendum: Another related problem, which I did not realize earlier, is that the SE Markdown converter does not support un-encoded Unicode in URLs.

This should really be fixed properly, by adding true RFC 3987 IRI support to the converter, but in the mean time, a sensible work-around would be to at least make the "Insert Hyperlink" button in the Markdown editor use Punycode instead of broken percent-encoding for IDNs, as suggested above. This does require some non-trivial processing, but fortunately, there are existing libraries available for it.

Ps. Version 1.18 of the Stack Overflow Unofficial Patch user script now contains a client-side fix for this issue. The fix consists of two parts:

  1. A monkey-patch to the Markdown editor "Insert Hyperlink" dialog that converts any URLs entered through it to use Punycode host names (using the punycode.js library by Mathias Bynens).

  2. A content filter that automatically decodes any inappropriately percent-encoded host names in existing links within posts and comments.

share|improve this answer
    
That's correct, as I soon realized. This is a very widespread problem on the Web and the sites that do not have this problem are the ones that allow Unicode characters in their links (i.e. Facebook). Presumably, the StackExchange does not want to open the door to possible XSS exploits. The solution would be using a mixed encoding system where the domain name is encoded with Punycode (non trivial but very doable) and the path, query and fragment as percent-encoded. –  Yserbius Jun 27 at 17:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .