User-Agent: *
Allow: /
Disallow: /posts
Disallow: /posts/
Disallow: /tags
Disallow: /tags/

Wikipedia says:

Some major crawlers support an Allow directive which can counteract a following Disallow directive. This is useful when you disallow an entire directory but still want some HTML documents in that directory crawled and indexed. While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that it first evaluates all Allow patterns and only then all Disallow patterns. (irrelevant in this case since Allow is the first, so google's special behavior does not apply here)

So the Allow: / directive overrides all the following disallows rendering robots.txt effectively useless.

This is the reason all major search engines (bing, google, etc) crawl pages they are not supposed to, like post revisions.

See also
Please include question title in title of revision history pages

  • 9
    i'm just barely resisting the urge to change the title to "Stack Overflow's robots are completely useless" – quack quixote Mar 25 '10 at 21:58
  • 3
    @quack quixote: I, for one, welcome our new robot overlords. – Randolpho Mar 26 '10 at 2:22

Well, that's interesting, since the Google Webmaster Tools robots.txt generator looks like

User-agent: *
Allow: /

But you're totally right that adding a disallow moves it above the Allow:

User-agent: *
Disallow: /foo/
Allow: /

So, thanks for digging this up.. solves a long-standing mystery!

| improve this answer | |
  • you might also consider noindex,noarchive meta tag on the pages you don't want indexed... just to have redundancy – Kip Mar 26 '10 at 2:53

So if this is true, why doesn't Google have search results for revisions?

I'm not saying you're wrong, just that I'm not understanding.

For example, I tried searching Google for site:stackoverflow.com inurl:posts/9033*, and while it had the revision page linked, it didn't crawl the page, since, as you can see, there is no summary blab for the revision page.

*That question is the most upvoted SO question, so it should also be the most linked. Thus, since the revisions page is linked from that page, Google is almost definitely guaranteed to find it and index it if it was allowed to.

| improve this answer | |
  • 1
    yes, you would expect there to be lots of these pages in the results if it was completely "useless" – Jeff Atwood Mar 25 '10 at 23:56
  • @waiwai933: if you look at the google screenshot on this question you can see revision pages that were cached with summary: meta.stackexchange.com/questions/44041 in that screenshot, i had searched for "locked by jeff atwood" site:stackoverflow.com – Kip Mar 26 '10 at 2:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .