When using Google to search Stack Overflow today, I noticed that the revision history pages all share the title "Revisions - Stack Overflow". This is not very friendly to the search engine user:


I would suggest using the current title of the post in the page's <title> tag.

In addition, a few SEO questions arise:

  • Should revision history be exposed to search engines? It is not very useful to new visitors, but it might include terms that the user searched for that have been edited away.
  • Should the revision history include a rel="canonical" link back to the actual question?


    User-Agent: *
    Allow: /
    Disallow: /posts
    Disallow: /posts/


PS: rel=canonical should be used only for identical or vastly similar (layout differences only) content. In this case it would be mis-using it and google will ignore it for sure, and maybe even penalize the search rankings.

EDIT: mystery solved. It's the Allow: /; from wikipedia:

Some major crawlers support an Allow directive which can counteract a following Disallow directive. This is useful when you disallow an entire directory but still want some HTML documents in that directory crawled and indexed. While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that it first evaluates all Allow patterns and only then all Disallow patterns.

In other words Allow: / overrides all the disallows and gives google access to the entire website.

| improve this answer | |
  • 1
    yep, we disallow it from Google but somehow they slip in anyway. – Jeff Atwood Mar 25 '10 at 20:14
  • @Jeff: maybe do the reverse of what you did in the sitemap? ie, if it's google send a 404 – Thomas Bonini Mar 25 '10 at 20:16
  • @Jeff: read my edit – Thomas Bonini Mar 25 '10 at 20:23
  • maybe you should make it a new feature request (or bug?). it seems like a pretty big deal, as it makes the site's current robots.txt useless – Kip Mar 25 '10 at 20:49
  • well this part (thanks Kop!) is [status-completed] anyway.. – Jeff Atwood Apr 23 '10 at 7:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .