29

There was a question on SciFi.SE where I wanted to mention the Japanese ronin, and to make sure people knew what I was talking about, I also wanted to link to the appropriate Wikipedia page.

Wikipedia, being what it is, has the page for ronin at the pretentiously diacritically-laden Rōnin. However, Stack Exchange does not share Wikipedia's pedantic use of diacritics and does the following:

  • In questions, it completely ignores the URL and acts like the link markdown isn't there
  • In comments, it strips out the ō and replaces it with a gaping void, sending people to the page for Rnin.

The question problem has come up before for other, similar URL detection issues, but the comment problem is, I believe, a different (and potentially more serious) issue as it presents the wrong URL instead of just ignoring it entirely.

Can the URL filter be adjusted to allow for valid Unicode characters, either by being left alone or URL encoded as per RFC 3986?

Improve this question
5
  • 2
    Example of the ō being stripped out: Rōnin.
    – user149432
    Apr 21, 2011 at 9:28
  • test - ja.wikipedia.org/wiki/浪人 OR en.wikipedia.org/wiki/Rōnin
    – YOU
    Apr 21, 2011 at 10:20
  • 5
    en.wikipedia.org/wiki/R%C5%8Dnin -- looks like Chrome now follows Firefox in urlencoding non-ASCII characters when copying a URL. So... Get a better browser?
    – Shog9
    Apr 21, 2011 at 20:11
  • 3
    At first glance, I thought your title was "URL filter goes rogue on unicorn characters in URLs" ... maybe this place is affecting me.
    – Brian Reichle
    Feb 9, 2012 at 7:13
  • @Shog9 This remains a bug on ELU; please see my comment below to balpha, along with an actual use-case example that just arose there.
    – tchrist
    May 4, 2014 at 17:17

2 Answers 2

Reset to default
7

This is fixed from the next build on (and since comments are rendered on the fly, it's fixed retroactively, so your first example comment under your question will then be correct).

Two things should be mentioned.

  1. This is comment-only at the moment. There's nothing inherently problematic with doing the same thing in posts (i.e. questions and answers), and we may do it at some point, but since

    • posts (unlike comments) have the markdown preview, so you immediately see what happens,

    • not-allowed characters in URLs aren't silently castrated in posts, but rather cause the sanitizer to eat the whole link, so it's much more obvious,

    • entering a link into a post via Ctrl-L or the "Hyperlink" button will do all the encoding for you anyway,

    • and this would require changing both Markdown implementations (which also are maintained as open source projects, thus requiring extra thoughts regarding compatibility),

    this is not immediately planned.

  2. There has been no change in the freeform link recognition ("you typed some text that looks like you may want it to be linked"); rules for this are as strict as always. Anything beyond the "standard" characters will cause an automatic link recognition to fail (at this point, I like to link to the canonical example of how recognizing too much is just as bad).

    But this failure is of course much more obvious than secretly removing characters from a link that otherwise looks as expected.

    Anyway, if your link contains crazy moon characters, as Jeff calls them, you have to tell us explicitly that it's a link, using the inline comment syntax.

    [Rōnin](http://en.wikipedia.org/wiki/Rōnin)
[text](http://somesite.com/index.php?x[]=y)
[images](http://www.mediawiki.org/wiki/Manual:$wgAllowExternalImages)
[Fisher-Yates shuffle](http://en.wikipedia.org/wiki/Fisher–Yates_shuffle)
[浪人](http://ja.wikipedia.org/wiki/浪人)
[markdown things](http://meta.stackexchange.com/questions/tagged/markdown*)

Oh, whitespace still isn't allowed. Because that's just crazy (well, and because it may be a syntax ambiguity). %20 your spaces yourself, if you have to use them.

Improve this answer
10
  • 1
    Snrk "Crazy moon characters"
    – Ben Brocka
    Apr 10, 2012 at 14:16
  • This continues to be a serious problem, and is not fixed. For example, one cannot even cite [the use–mention distinction](http://en.wikipedia.org/wiki/Use–mention_distinction) without URI-encoding each separate UTF-8 byte in the link proper, requiring the user to type [the use–mention distinction](http://en.wikipedia.org/wiki/Use%E2%80%93mention_distinction) instead. This is far from obvious to programmers, and even they might not know how to do it “properly”. And it is impossible for non-programmers, such as those on ELU, where the use–mention distinction actually matters quite a bit.
    – tchrist
    May 4, 2014 at 17:15
  • the use–mention distinction Looks good to me @tchrist -- that's copied directly from the non-encoded version in your comment. What exactly do you see not working?
    – balpha StaffMod
    May 4, 2014 at 17:45
  • When I write en.wikipedia.org/wiki/Use–mention_distinction it does not work. I intentionally re-encoded it myself in this answer because the raw version with actual UTF-8 characters didn’t work.
    – tchrist
    May 4, 2014 at 17:48
  • @tchrist Yes, that's point 2 in my answer. "You have to tell us explicitly it's a link". Your comment wasn't about that, though.
    – balpha StaffMod
    May 4, 2014 at 17:50
  • Even when I tell it explicitly that it is a link using square brackets and round ones, like by writing [use–mention distinction](http://en.wikipedia.org/wiki/Use–mention_distinction), it still does not work on ELU or its meta. I have to URI-encode the UTF-8 manually, even in the parens. Similarly in chat, where the problem is even worse: epic fail. Here I do not have to do that, but there I do.
    – tchrist
    May 4, 2014 at 17:52
  • @tchrist meta.english.stackexchange.com/questions/2706/… Can you give me a link to a comment where it doesn't work? (I'll look at updating chat to behave like comments, though)
    – balpha StaffMod
    May 4, 2014 at 17:57
  • It works in a comment, but not in a post. Scroll to the bottom here and look at the Addendum, then look at the source. Thanks a bunch!
    – tchrist
    May 4, 2014 at 18:06
  • @tchrist This post here is about comments. Point 1 of my answer addresses posts.
    – balpha StaffMod
    May 4, 2014 at 18:08
  • Ok, so should I submit a new bug for posts then? And also one for chat? One cannot expect nonprogrammers to know how to manually URI-encode UTF-8 bytes, especially when they cannot even see them. That EN DASH looks very much like a HYPHEN-MINUS, which doesn’t have the issue. And they’ll never figure out how it would have to be escaped.
    – tchrist
    May 4, 2014 at 18:09
2

It should be noted that we have 2 implementations here; a full RFC-3986 implementation is in place on some sites, in particular our foreign language sites. The UTF-8 rules are not currently enabled on scifi. I can confirm that "ō" is not handled currently, although we have gradually added a few others (for downgrade to "o" etc). I'm not sure that this is a common enough issue to flip the switch for UTF-8, though - thoughts?

Improve this answer
6
  • 2
    Is there an ongoing cost to having it enabled on more sites? I couldn't pretend to estimate how prevalent a problem it is on the English-language sites (although I suspect it's a very small percentage of links), but there's no indication what's wrong if you happen to copy-paste such URLs and aren't familiar with this quirk.
    – user149432
    Feb 9, 2012 at 6:20
  • @Mark I assume the biggest issue is that existing URLs would all suddenly be redirects; not sure that is a show-stopper, though. Let me check.
    – Marc Gravell
    Feb 9, 2012 at 7:13
  • Maybe just a simple alert is all that's needed... If non-ascii is detected in a link, merely tell the user that he must URL encode it (or upgrade browsers). No silent fail == no problem.
    – Awesome Poodles
    Feb 9, 2012 at 7:37
  • @Mark update; we can enable it for the questions, but it looks like comments are still a bit impacted by this
    – Marc Gravell
    Feb 9, 2012 at 7:38
  • @MarcGravell That'd be good: at least with comments it's immediately obvious something went wrong. In questions, it just silently removes the link (or did, haven't checked in a while).
    – user149432
    Feb 9, 2012 at 7:39
  • It’s also a problem on ELU. See my comment to balpha.
    – tchrist
    May 4, 2014 at 17:16

You must log in to answer this question.