10

This year's Winter Bash site has robots.txt which prevents the whole1 site from being crawled.

User-agent: *
Disallow: /

Is there a reason why robots.txt was added to this year's WB site?

1The main page is crawled and archived on the Wayback Machine.

14

Google's spider was hitting it... A lot...

This was causing a lot of unnecessary load on the server. Best to just block the bots so as to let real people enjoy it.

  • 4
    Ah I see, thanks for the response! Would it be possible to only allow Wayback Machine to crawl it (maybe after the whole event ends)? – Panda Dec 21 '18 at 5:03
3

There are possibly 2 other explanations for this (apart from what Shog mentioned already):

  • It is to prevent using search engines to answer searches about secret hats.
  • It's a way to implement the equivalent of I hate hats for search engines, whereas:

    • they seem to not have a choice to change their preference.
    • the default for them is the opposite of the default (= I like hats) for regular SE users.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .