9

Attempting to link to a site that uses non-alphanumeric characters in its domain name is incorrectly encoded to Hexadecimal "safe" characters instead of punycode.

For instance, clicking on this links to encodes the link instead of linking to http://domaintest.xn--q9jyb4c/.

Also, attempting to write something with non-alphanumeric characters does not recognize anything past those characters as part of the URL.

6
  • 1
    I must be missing something, as I end up in the same place, and the same URL with both links.
    – Oded StaffMod
    Commented Jun 27, 2014 at 12:57
  • 2
    @Oded depends on the browser. Works in Chrome, the first link breaks in IE11 and FF30 for me. Second link brings me to a testpage in all browsers.
    – rene Mod
    Commented Jun 27, 2014 at 13:06
  • Still wondering if this is a problem with the software of Stack Exchange?
    – rene Mod
    Commented Jun 27, 2014 at 13:09
  • 2
    What browser are you using? Both IE and Firefox have the issue for me. This is how my page source looks. Note the difference in the URLS. No issue in Chrome though.
    – Yserbius
    Commented Jun 27, 2014 at 13:09
  • 3
    To close voters: I can still reproduce the problem in IE11 and Edge. The fact that it "works" in Chrome is because of non-standard behavior on its end; see the answer below. Commented Nov 6, 2018 at 19:09
  • If you still cannot reproduce the issue, make sure that you've disabled the SOUP userscript. Commented Nov 6, 2018 at 19:28

1 Answer 1

8

The real problem here is not the lack of punycode encoding, but rather the inappropriate application of percent-encoding to the domain part of the URL. Simply leaving the Unicode characters in the domain name un-encoded, as in:

<a href="http://www.domaintest.みんな/">this</a>

makes the URL into a valid IRI, which all modern browsers support.

The fact that Chrome, apparently, also accepts the invalid percent-encoded domain names (and decodes and re-encodes them as punycode) does not make this behavior correct or desirable.


Addendum: Another related problem, which I did not realize earlier, is that the SE Markdown converter does not support un-encoded Unicode in URLs.

This should really be fixed properly, by adding true RFC 3987 IRI support to the converter, but in the mean time, a sensible work-around would be to at least make the "Insert Hyperlink" button in the Markdown editor use Punycode instead of broken percent-encoding for IDNs, as suggested above. This does require some non-trivial processing, but fortunately, there are existing libraries available for it.

Ps. Version 1.18 of the Stack Overflow Unofficial Patch user script now contains a client-side fix for this issue. The fix consists of two parts:

  1. A monkey-patch to the Markdown editor "Insert Hyperlink" dialog that converts any URLs entered through it to use Punycode host names (using the punycode.js library by Mathias Bynens).

  2. A content filter that automatically decodes any inappropriately percent-encoded host names in existing links within posts and comments.

1
  • That's correct, as I soon realized. This is a very widespread problem on the Web and the sites that do not have this problem are the ones that allow Unicode characters in their links (i.e. Facebook). Presumably, the StackExchange does not want to open the door to possible XSS exploits. The solution would be using a mixed encoding system where the domain name is encoded with Punycode (non trivial but very doable) and the path, query and fragment as percent-encoded.
    – Yserbius
    Commented Jun 27, 2014 at 17:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .