13

Until yesterday it was possible to save hot questions page in Wayback Machine as discussed eg here.

Today robots.txt appears to be changed to block this.

I wonder what could be reasons for such a change?

https://i.sstatic.net/QOHCQ.png


In case if it matters, current contents of robots.txt file is as follows:

User-Agent: *
Disallow: /feeds/
Disallow: /users/login/
Disallow: /users/chat-login/
Disallow: /users/logout/
Disallow: /users/authenticate/
Disallow: /users/flair/
Disallow: /api/
Disallow: /*/ivc/*
Disallow: /*?
Disallow: /leagues/*/week/*/20
Disallow: /leagues/*/month/*/20
Disallow: /leagues/*/quarter/*/20
Disallow: /leagues/*/year/*/20
Disallow: /leagues/*/alltime/*/20
Disallow: /filters/*/*
Allow: /

# for "/*?", refer to http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

#
# beware, the sections below WILL NOT INHERIT from the above!
# http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
#

#
# disallow adsense bot, as we no longer do adsense.
#
User-agent: Mediapartners-Google
Disallow: / 

User-agent: Spinn3r
Disallow: /

#
# Yahoo Pipes is for feeds not web pages.
#
User-agent: Yahoo Pipes 1.0
Disallow: /

#
# This isn't really an image
#
User-agent: Googlebot-Image
Disallow: /*/ivc/*

#
# KSCrawler - we don't need help from you
#
User-agent: KSCrawler
Disallow: /
7
  • side note for those interested in maintaining historical log of HNQ at Wayback Machine, this page still can be saved over there if you enter Stack Exchange homepage URL https://stackexchange.com (which redirects to questions?tab=hot)
    – gnat
    Commented Apr 15, 2017 at 7:07
  • 3
    Guess they want to make it harder for people to archive it, thus making it harder to complain about them. Commented Apr 15, 2017 at 9:30
  • 2
    "a tool to feed speculation and paranoia..." @ShadowWizard :)
    – gnat
    Commented Apr 15, 2017 at 9:33
  • 2
    I'm a little confused here - that file hasn't changed since May 2015 and the line in question has been there since 2010... Commented Apr 17, 2017 at 20:47
  • 1
    Maybe WBM adjusted their spiders? (Clearly part of the great HNQ conspiracy...) Commented Apr 18, 2017 at 2:53
  • 1
    and... blog.archive.org/2017/04/17/… internet archives might be ignoring robots.txt in future. Which should probably make you very happy Commented Apr 22, 2017 at 14:12
  • Any explanation as to what the KSCrawler comment is about?
    – jmoreno
    Commented Feb 13, 2022 at 0:31

2 Answers 2

14
+50

robots.txt was changed to block saving hot questions page in Wayback Machine, why?

Nope! The robots.txt hasn't changed since May, 2015. And the Disallow: /*? directive has been there since the file was created in 2010.

So...this wasn't us. The Wayback Machine changed something on their end.

4

The answer appears to be hidden in the robots.txt file itself.

Disallow: /*?

Seems to be the appropriate rule

and is documented in a comment

# for "/*?", refer to http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360

Which might not be the right link. Dang, I was hoping that it would document itself

The line incidentally also prevents scrapers from accidentally running queries for almost any tab or search type thing which might be the actual reason. Its a lot simpler than a dozen seperate per-tab-type rules. Of course, we'd need a dev for a definitive answer but this seems plausible. According to Nick Craver, the files haven't actually been changed recently - with the line being there since 2010, so If that's changed, it might be at wayback machine's end.

5
  • wonder why they decided to change to this way so suddenly (until yesterday access went just fine)
    – gnat
    Commented Apr 15, 2017 at 9:35
  • 1
    Beats me, I don't exactly hang around watching the HNQ list all the time ;) Commented Apr 15, 2017 at 9:35
  • You can see on main sites (e.g. MSE's) there are lots of separate rules with "?" query strings; I guess having a single catch-all rule is a bit easier.. you generally don't want different tabs indexed I guess
    – Cai
    Commented Apr 15, 2017 at 9:43
  • Yeah, that's what I'm thinking. Also makes it more maintainable. Commented Apr 15, 2017 at 9:45
  • "To block access to all URLs that include question marks (?). For example, the sample code blocks URLs that begin with your domain name, followed by any string, followed by a question mark, and ending with any string: User-agent: Googlebot Disallow: /*?" Check the "Block URLs with robots.txt" section, "Pattern-matching rules to streamline your robots.txt code" subsection.
    – Braiam
    Commented Apr 16, 2017 at 17:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .