25

It apparently used to be the case that the robots.txt file on chat.stackexchange.com allowed search engines to index some parts of that domain; this robots.txt file was apparently intended to allow crawling of chat transcripts, though that question suggests it didn't actually work.

Now, however, it is set to disallow crawling of the entire domain, explicitly preventing chat transcripts from being indexed.

However, this does not work; a Google search for site:chat.stackexchange.com transcripts shows that chat transcripts in fact are being indexed thoroughly. I think that the intended behavior--telling crawlers not to index chat transcripts--is marginally preferable from a privacy perspective (not, of course, that SE chat is really intended to be private).

I believe that the reason for this not working is fairly simple; although that robots.txt file is valid, it is returned with HTTP status code 418 ("I'm a Teapot") rather than 200. Crawlers generally ignore robots.txt files returned with error status codes and I believe that this is why Google is not respecting that robots.txt file.

I suggest fixing this by changing the status code to a 200, either keeping the current policy (disallowing indexing) in place or committing to allowing the indexing of chat transcripts. I'd prefer the former but regardless the current behavior is a bug.

Improve this question
6
  • 5
    .... That's a rather hilarious misconfiguration if intentional
    – Journeyman Geek
    Commented Oct 19 at 0:21
  • 4
    @JourneymanGeek Its hilarity would be little lessened by lack of intent.
    – tchrist
    Commented Oct 19 at 0:56
  • 2
    Love error code 418! Also amusing is error code 503, which is when a combined coffee pot/teapot is temporarily out of coffee.
    – DaemonsMercy
    Commented Oct 19 at 2:29
  • 12
    To be honest, I would prefer that chat is indexed, the search there is terrible, and to find anything more complex than single word we have to use external search. But meh, you're of course correct, and that would be one bug I won't be happy to see fixed. :)
    – Shadow Wizard
    Commented Oct 19 at 7:18
  • 1
    Kind of related: Preventing unauthorized automated access to the network (#3)
    – Meta Andrew T.
    Commented Oct 21 at 12:24
  • 1
    Conspiracy theory,. The robots.text file is a teapot.
    – Journeyman Geek
    Commented Oct 28 at 15:03

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .