9

I have seen that attempting to navigate to a URL such as http://web.archive.org/http://iwataasks.nintendo.com/interview-club-nintendo.html#/game-and-watch-ball-reward/0/0 posted in a comment will actually lead to a malformed URL of the format http://[site].stackexchange.com/questions/[questionID]/http://main-board-of-the-original-game-watch-ball/http://iwataasks.nintendo.com/interview-club-nintendo.html#/game-and-watch-ball-reward/0/0

Reproduced on Retrocomputing and on Meta Stack Exchange. Bug first reported on Retrocomputing Meta before I realized it was network-wide.

Also note that this is not reproducible in the iOS app but is reproducible on mobile and desktop web.

Note that this is not a duplicate of archive.org URL not handled properly in comment or Asterisks break URLs because those involve URLs with asterisks. This URL does not contain asterisks.

  • Took a quick look in the Chrome debugger. It looks like archive.org itself is returning a redirect to that URL on Stack Exchange's site. This could have been based off of the referer header. It still returns the stack exchange URL when I remove that header, but that seems to be caching: changing other aspects of the request breaks the cache and gets a redirect to the right URL. It seems like this may be a bug on the Internet Archive's side; I don't see anything wrong with the links we generate. – Jeremy Banks Oct 28 '16 at 19:46
  • @jeremyBanks Weird, so how come the link in this comment works? Is it because the link was formatted to now show the complete URL? – JAL Oct 28 '16 at 22:06
  • I'm not sure. It's https, but I didn't think https prevented sites from receiving referers, just from sending them to http sites. (i.e. I don't think this bug happens if you're using https to read Stack Exchange.) – Jeremy Banks Oct 28 '16 at 22:17
  • Test comment: web.archive.org/http://iwataasks.nintendo.com/… – Ben N Oct 29 '16 at 19:18
  • @BenN that should work now (no need to re-post etc) – Marc Gravell Nov 1 '16 at 10:18
7

Thanks to @IlmariKaronen's excellent detective work, it looks like refer[r]er plays a strong role here. The good news is that we've already done the work to make markdown-mini emit rel="nofollow noreferrer" links (rather than just rel="nofollow"); this should have the happy accident of fixing this issue. The bad news is that there's something banjaxed in the packaging of that component, so those changes didn't get deployed cleanly. I'll get that straightened out and deploy when I can.

8

Wow, that's a weird one!

This is clearly a bug at archive.org — I can reproduce it by manually sending HTTP requests to their servers, without involving SE in any way. What appears to trigger it is the combination of:

  1. using a short archive URL of the form http://web.archive.org/..., as opposed to the longer form http://web.archive.org/web/*/..., and;

  2. having the browser send a HTTP Referer (sic) header with a URL whose path portion begins with some letters, a slash, some numbers and another slash (optionally followed by more text).

It just happens that SE question page URLs like http://meta.stackexchange.com/questions/286411/... happen to match that pattern, and so will trigger the bug, but a referrer URL like http://example.com/a/1/b will trigger it just as well. For example, here's the output of a quick command-line test using lwp-request:

$ GET -USed -H Referer:http://example.com/a/1/b http://web.archive.org/http://foo.bar/test
GET http://web.archive.org/http://foo.bar/test
Referer: http://example.com/a/1/b
User-Agent: lwp-request/6.03 libwww-perl/6.05

302 Found
Connection: close
Date: Mon, 31 Oct 2016 01:18:29 GMT
Location: http://example.com/a/1/http://b/http://foo.bar/test
Server: Tengine/2.1.0
Vary: Referer
Content-Length: 0
Content-Type: text/html
Client-Date: Mon, 31 Oct 2016 01:18:29 GMT
Client-Peer: 207.241.225.186:80
Client-Response-Num: 1
Set-Cookie: wayback_server=12; Domain=archive.org; Path=/; Expires=Wed, 30-Nov-16 01:18:29 GMT;
X-Archive-Playback: 0
X-Page-Cache: MISS

Also, note that these responses are cached pretty aggressively by the archive.org servers, so sending a second request for the same URL with a different Referer header will still return the original response (with X-Page-Cache: HIT instead of MISS in the response headers).

The reason this doesn't (normally) happen with https: links is because, for security reasons, most browsers won't send cross-protocol referrer information. So if a plain http: page links to an https: URL, no Referer header will be sent. Of course, if you happened to be reading SE over HTTPS (which mostly works), you'd see the same bug with https: links too.

Fortunately, while this is really a bug on archive.org's side, the workaround is pretty simple: always include the /web/* part in archive.org URLs.

Of course, the SE automatic URL detector won't handle URLs containing an asterisk correctly, so you'll have to use the Markdown [link text](URL) syntax. But that's a good habit to get into anyway.

  • 1
    That's some above-and-beyond investigation; I offer a quiet clap of appreciation. But, yeah, that's tricky! I wonder whether it would be acceptable to add rel="nofollow noreferrer" in the auto-linking (rather than just rel="nofollow") – Marc Gravell Oct 31 '16 at 13:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .