4

When using Google to search Stack Overflow today, I noticed that the revision history pages all share the title "Revisions - Stack Overflow". This is not very friendly to the search engine user:

http://img641.imageshack.us/img641/5190/revisiontitles.png

I would suggest using the current title of the post in the page's <title> tag.

In addition, a few SEO questions arise:

  • Should revision history be exposed to search engines? It is not very useful to new visitors, but it might include terms that the user searched for that have been edited away.
  • Should the revision history include a rel="canonical" link back to the actual question?

1 Answer 1

3

https://stackoverflow.com/robots.txt

    User-Agent: *
    Allow: /
    Disallow: /posts
    Disallow: /posts/

Weird.

PS: rel=canonical should be used only for identical or vastly similar (layout differences only) content. In this case it would be mis-using it and google will ignore it for sure, and maybe even penalize the search rankings.


EDIT: mystery solved. It's the Allow: /; from wikipedia:

Some major crawlers support an Allow directive which can counteract a following Disallow directive. This is useful when you disallow an entire directory but still want some HTML documents in that directory crawled and indexed. While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that it first evaluates all Allow patterns and only then all Disallow patterns.

In other words Allow: / overrides all the disallows and gives google access to the entire website.

5
  • 1
    yep, we disallow it from Google but somehow they slip in anyway. Mar 25, 2010 at 20:14
  • @Jeff: maybe do the reverse of what you did in the sitemap? ie, if it's google send a 404 Mar 25, 2010 at 20:16
  • @Jeff: read my edit Mar 25, 2010 at 20:23
  • maybe you should make it a new feature request (or bug?). it seems like a pretty big deal, as it makes the site's current robots.txt useless
    – Kip
    Mar 25, 2010 at 20:49
  • well this part (thanks Kop!) is [status-completed] anyway.. Apr 23, 2010 at 7:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .