4

I just happened to notice that entering a URL that contains emoji or other non-BMP Unicode characters into the "Insert Hyperlink" dialog in the SE Markdown editor will fail with an uncaught URIError. This happens both with the old-style hyperlink dialog (as currently used here on meta.SE) and with the new-style one (as currently used on SO).

Steps to reproduce:

  1. Open a post editor, e.g. by clicking the Ask Question link. The "Your Answer" editor at the bottom of this page will also work.

  2. Open the "Insert Hyperlink" dialog, either by clicking the icon showing a pair of chain links above the edit box, or by pressing Ctrl+L while the editor has focus.

  3. Enter a URL that contains non-BMP Unicode characters, like https://en.wikipedia.org/wiki/😃, into the dialog.

  4. Submit the dialog by pressing Enter or clicking the "OK" button.

  5. Observe that no link was actually inserted into the post (and that an uncaught URIError: malformed URI sequence was logged to the browser's JS console instead).

Looking at the Markdown editor code, the problem appears to occur in the properlyEncoded() function, on the following line:

link = link.replace(/%(?:[\da-fA-F]{2})|\?|\+|[^\w\d-./[\]]/g, function (match) {

The regexp on that line is supposed to match (valid % escape sequences and) any single characters in the URL outside the character class \w\d-./[\], and feed them one by one to the following callback function which URL-encodes them. However, it instead ends up matching individual UTF-16 code units, including halves of a surrogate pair. Since these individual surrogates do not represent valid Unicode characters, and so cannot be correctly URL-encoded without knowing the other half of the pair, passing them to encodeURI() causes it to throw an error.

  • 1
    Why should SE support invalid URL's in the first place? – Shadow The Curly Braced Wizard Oct 5 '16 at 14:10
  • "I just happened to notice" hmmm – Cai Oct 5 '16 at 14:11
  • @ShadowWizard: These are perfectly valid URLs, and are being used in practice. For example, https://en.wikipedia.org/wiki/😃 works and (currently) redirects to en.wikipedia.org/wiki/Emoticon, even if SE's parser doesn't currently handle it properly. Try pasting it into your browser's URL bar if you don't believe me. – Ilmari Karonen Oct 5 '16 at 14:15
  • @Cai: In fact, I was reading this question on security.SE and decided to test how SE's editor (and SOUP) would handle such a URL. As it turns out, they both break, but at least I can fix SOUP on my own. 😃‎ – Ilmari Karonen Oct 5 '16 at 14:17
  • As far as I can tell, the browser itself is encoding it to https://en.wikipedia.org/wiki/%F0%9F%98%83 before actually going to this address, so it's not relevant. – Shadow The Curly Braced Wizard Oct 5 '16 at 14:17
  • @IlmariKaronen makes sense. Had no idea emoji URLs were a thing... I don't think they should be allowed (at all, ever, anywhere). How about instead of fixing this we throw a polite notice telling people to stop it? – Cai Oct 5 '16 at 14:33
  • @ShadowWizard: Yes, but that's an internal detail for backwards compatibility, not something that users should need to care about. See RFC 3987 §3 for more details. It makes no sense to assume that links entered by users will always be in %-encoded form. We already fixed this bug for Unicode characters on the BMP, it just apparently never got properly tested with non-BMP characters. – Ilmari Karonen Oct 5 '16 at 14:33
  • 1
    @Cai: Note that there are a lot of Unicode characters outside the BMP besides emoji, and this bug affects all of them. Do you really feel that none of them should ever be used in URLs "(at all, ever, anywhere)"? – Ilmari Karonen Oct 5 '16 at 14:45
  • @IlmariKaronen in all honestly probably yes. I don't think URLs with playing cards, dominos or hieroglyphics should be a thing. Anyway, I was being facetious—I'm not seriously suggesting anything—just voicing my disdain for emoji. – Cai Oct 5 '16 at 14:54
2

There are several ways in which this bug could be fixed:

  1. One solution, for modern browsers, would be to add the u flag to the regexp to make it treat surrogate pairs as single characters, but alas, browser support for that ES2015 feature may still be insufficient.

  2. An alternative fix would be to make the regexp explicitly match surrogate pairs, like this:

    link = link.replace(/%(?:[\da-fA-F]{2})|[\uD800-\uDBFF][\uDC00-\uDFFF]|[^\w\d-./[\]]/g, function (match) {
    

    Note that the \?|\+ part of the original regexp is redundant, so I've omitted it above. The surrogate pair pattern must come before the [^\w\d-./[\]] part, so that the regexp engine tries it first. Alternatively, individual surrogate characters could be completely excluded from the match by changing the last part into [^\w\d\-./[\]\uD800-\uDFFF].

  3. An even simpler fix (which would likely also speed up the replacement a little) would be to allow the regexp to match an arbitrary number of non-excluded code units at once, like this:

    link = link.replace(/%(?:[\da-fA-F]{2})|[?+]|[^\w\d-./[\]%?+]+/g, function (match) {
    

    Note that this solution requires explicitly excluding the special characters %, ? and + from the last part of the regexp, since the callback code below assumes that those characters will always be matched individually.

Any of these methods should solve the problem (although, as noted, the first one might not work on all supported browsers yet). Personally, I'd recommend the last method for its simplicity and efficiency, although it probably needs to be explicitly commented so that future maintainers won't accidentally break it e.g. by removing the +.


Ps. My SOUP user script currently fixes this issue as a side effect of its workaround for the IDN percent-encoding bug. The code I'm using in that work-around is basically the same as my third suggestion above, except somewhat simplified (since it's just a pre-processing step before the URL is fed into the actual SE properlyEncoded() function):

// Separate URL and optional title, fix possibly broken % encoding in URL
var m = /^\s*(.*?)(?:\s+"(.*)")?\s*$/.exec(text);
var url = m[1], title = m[2];
var normalized = url.replace(/%(?:[\da-fA-F]{2})|[^\w\d\-./[\]%?+]+/g, function (match) {
    if (match.length === 3 && match.charAt(0) == "%") return match;
    else return encodeURI(match);
} );

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .