Emoji in URL breaks "insert hyperlink" tool in editor

Question

I just happened to notice that entering a URL that contains emoji or other non-BMP Unicode characters into the "Insert Hyperlink" dialog in the SE Markdown editor will fail with an uncaught URIError. This happens both with the old-style hyperlink dialog (as currently used here on meta.SE) and with the new-style one (as currently used on SO).

Steps to reproduce:

1. Open a post editor, e.g. by clicking the Ask Question link. The "Your Answer" editor at the bottom of this page will also work.

2. Open the "Insert Hyperlink" dialog, either by clicking the icon showing a pair of chain links above the edit box, or by pressing Ctrl+L while the editor has focus.

3. Enter a URL that contains non-BMP Unicode characters, like https://en.wikipedia.org/wiki/😃, into the dialog.

4. Submit the dialog by pressing Enter or clicking the "OK" button.

5. Observe that no link was actually inserted into the post (and that an uncaught URIError: malformed URI sequence was logged to the browser's JS console instead).

Looking at the Markdown editor code, the problem appears to occur in the properlyEncoded() function, on the following line:

link = link.replace(/%(?:[\da-fA-F]{2})|\?|\+|[^\w\d-./[\]]/g, function (match) {

The regexp on that line is supposed to match (valid % escape sequences and) any single characters in the URL outside the character class \w\d-./[\], and feed them one by one to the following callback function which URL-encodes them. However, it instead ends up matching individual UTF-16 code units, including halves of a surrogate pair. Since these individual surrogates do not represent valid Unicode characters, and so cannot be correctly URL-encoded without knowing the other half of the pair, passing them to encodeURI() causes it to throw an error.

Answer 1

There are several ways in which this bug could be fixed:

1. One solution, for modern browsers, would be to add the u flag to the regexp to make it treat surrogate pairs as single characters, but alas, browser support for that ES2015 feature may still be insufficient.

2. An alternative fix would be to make the regexp explicitly match surrogate pairs, like this:

   link = link.replace(/%(?:[\da-fA-F]{2})|[\uD800-\uDBFF][\uDC00-\uDFFF]|[^\w\d-./[\]]/g, function (match) {
   

   Note that the \?|\+ part of the original regexp is redundant, so I've omitted it above. The surrogate pair pattern must come before the [^\w\d-./[\]] part, so that the regexp engine tries it first. Alternatively, individual surrogate characters could be completely excluded from the match by changing the last part into [^\w\d\-./[\]\uD800-\uDFFF].

3. An even simpler fix (which would likely also speed up the replacement a little) would be to allow the regexp to match an arbitrary number of non-excluded code units at once, like this:

   link = link.replace(/%(?:[\da-fA-F]{2})|[?+]|[^\w\d-./[\]%?+]+/g, function (match) {
   

   Note that this solution requires explicitly excluding the special characters %, ? and + from the last part of the regexp, since the callback code below assumes that those characters will always be matched individually.

Any of these methods should solve the problem (although, as noted, the first one might not work on all supported browsers yet). Personally, I'd recommend the last method for its simplicity and efficiency, although it probably needs to be explicitly commented so that future maintainers won't accidentally break it e.g. by removing the +.

---

Ps. My SOUP user script currently fixes this issue as a side effect of its workaround for the IDN percent-encoding bug. The code I'm using in that work-around is basically the same as my third suggestion above, except somewhat simplified (since it's just a pre-processing step before the URL is fed into the actual SE properlyEncoded() function):

// Separate URL and optional title, fix possibly broken % encoding in URL
var m = /^\s*(.*?)(?:\s+"(.*)")?\s*$/.exec(text);
var url = m[1], title = m[2];
var normalized = url.replace(/%(?:[\da-fA-F]{2})|[^\w\d\-./[\]%?+]+/g, function (match) {
    if (match.length === 3 && match.charAt(0) == "%") return match;
    else return encodeURI(match);
} );