13

Until yesterday it was possible to save hot questions page in Wayback Machine as discussed eg here.

Today robots.txt appears to be changed to block this.

I wonder what could be reasons for such a change?

https://i.stack.imgur.com/QOHCQ.png


In case if it matters, current contents of robots.txt file is as follows:

User-Agent: *
Disallow: /feeds/
Disallow: /users/login/
Disallow: /users/chat-login/
Disallow: /users/logout/
Disallow: /users/authenticate/
Disallow: /users/flair/
Disallow: /api/
Disallow: /*/ivc/*
Disallow: /*?
Disallow: /leagues/*/week/*/20
Disallow: /leagues/*/month/*/20
Disallow: /leagues/*/quarter/*/20
Disallow: /leagues/*/year/*/20
Disallow: /leagues/*/alltime/*/20
Disallow: /filters/*/*
Allow: /

# for "/*?", refer to http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

#
# beware, the sections below WILL NOT INHERIT from the above!
# http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
#

#
# disallow adsense bot, as we no longer do adsense.
#
User-agent: Mediapartners-Google
Disallow: / 

User-agent: Spinn3r
Disallow: /

#
# Yahoo Pipes is for feeds not web pages.
#
User-agent: Yahoo Pipes 1.0
Disallow: /

#
# This isn't really an image
#
User-agent: Googlebot-Image
Disallow: /*/ivc/*

#
# KSCrawler - we don't need help from you
#
User-agent: KSCrawler
Disallow: /
6
  • side note for those interested in maintaining historical log of HNQ at Wayback Machine, this page still can be saved over there if you enter Stack Exchange homepage URL https://stackexchange.com (which redirects to questions?tab=hot)
    – gnat
    Apr 15 '17 at 7:07
  • 3
    Guess they want to make it harder for people to archive it, thus making it harder to complain about them. Apr 15 '17 at 9:30
  • 2
    "a tool to feed speculation and paranoia..." @ShadowWizard :)
    – gnat
    Apr 15 '17 at 9:33
  • 2
    I'm a little confused here - that file hasn't changed since May 2015 and the line in question has been there since 2010... Apr 17 '17 at 20:47
  • 1
    Maybe WBM adjusted their spiders? (Clearly part of the great HNQ conspiracy...)
    – Journeyman Geek Mod
    Apr 18 '17 at 2:53
  • 1
    and... blog.archive.org/2017/04/17/… internet archives might be ignoring robots.txt in future. Which should probably make you very happy
    – Journeyman Geek Mod
    Apr 22 '17 at 14:12
14
+50

robots.txt was changed to block saving hot questions page in Wayback Machine, why?

Nope! The robots.txt hasn't changed since May, 2015. And the Disallow: /*? directive has been there since the file was created in 2010.

So...this wasn't us. The Wayback Machine changed something on their end.

4

The answer appears to be hidden in the robots.txt file itself.

Disallow: /*?

Seems to be the appropriate rule

and is documented in a comment

# for "/*?", refer to http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

Which might not be the right link. Dang, I was hoping that it would document itself

The line incidentally also prevents scrapers from accidentally running queries for almost any tab or search type thing which might be the actual reason. Its a lot simpler than a dozen seperate per-tab-type rules. Of course, we'd need a dev for a definitive answer but this seems plausible. According to Nick Craver, the files haven't actually been changed recently - with the line being there since 2010, so If that's changed, it might be at wayback machine's end.

5
  • wonder why they decided to change to this way so suddenly (until yesterday access went just fine)
    – gnat
    Apr 15 '17 at 9:35
  • 1
    Beats me, I don't exactly hang around watching the HNQ list all the time ;)
    – Journeyman Geek Mod
    Apr 15 '17 at 9:35
  • You can see on main sites (e.g. MSE's) there are lots of separate rules with "?" query strings; I guess having a single catch-all rule is a bit easier.. you generally don't want different tabs indexed I guess
    – Cai
    Apr 15 '17 at 9:43
  • Yeah, that's what I'm thinking. Also makes it more maintainable.
    – Journeyman Geek Mod
    Apr 15 '17 at 9:45
  • "To block access to all URLs that include question marks (?). For example, the sample code blocks URLs that begin with your domain name, followed by any string, followed by a question mark, and ending with any string: User-agent: Googlebot Disallow: /*?" Check the "Block URLs with robots.txt" section, "Pattern-matching rules to streamline your robots.txt code" subsection.
    – Braiam
    Apr 16 '17 at 17:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .