Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 157 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

If you do a Google search for text edited or removed from a post, you get the "revisions" page that shows the edit history. Isn't there a way to discourage or prevent Google from indexing these pages?

Only latest, complete version of posts should reflect the actual content of the site. The revision history is just administrative overhead.

Technology changes, information becomes obsolete, and code becomes deprecated. Hopefully, the archive of Stack Overflow will continue to be updated, augmented, and incorrect information removed.

As a matter of prestige, Stack Overflow should not accumulate endless search hits to information that has been explicitly removed or corrected. There is also the illegal information (Links to pirated material in the edit history) that will be hosted on Stack Overflow forever.

share|improve this question
Isn't the pirated material what's illegal, not the links to them? Google probably links to tons of pirated material. – Kip Aug 17 '09 at 13:09
up vote 4 down vote accepted

This was a mistake in robots.txt having to do with the interpretation of trailing slashes.. should be fixed now.

Well, it wasn't fixed. Apparently, per Google Webmaster support ..

Each section in the robots.txt file is separate and does not build upon previous sections. For example:

User-agent: *
Disallow: /folder1/

User-Agent: Googlebot
Disallow: /folder2/

In this example only the URLs matching /folder2/ would be disallowed for Googlebot.

I expected the * part to be inherited by Googlebot, but it wasn't. Stunningly counter-intuitive, but maybe my programmer background was tricking me.

I restructured robots.txt for the fifth and hopefully final time ... maybe now it's fixed?

share|improve this answer
interesting... but why specify specific bots at all? – Kip Aug 17 '09 at 13:05

To me this is a benefit. If google indexes information that is added and removed, SO gets a larger 'web' to catch traffic. Google has its own algorithms to try and deliver users with the most relevant content. If this content is a revisions page, so be it.

Perhaps if you're linked to a revisions page from google you could be invisibly redirected to the questions page, but you should never try to prevent google from indexing useful information.

share|improve this answer
a <link rel="canonical" ... />, to the question page, would also make sense. – Brad Gilbert Aug 12 '09 at 17:52
That strikes me as a bit Experts-Exchange-esque (i.e. links to non-content polluting Google for the sake of visibility). I'm talking about the future. Casting a large net is innocuous now, but as more and more "non-content" is returned in searches, it will become increasingly "spammy". – Robert Cartaino Aug 12 '09 at 18:43
Exposing the revision list to Google and everyone else is nothing like the hyphen site. – Brad Gilbert Aug 12 '09 at 18:57
@rcataino, it's not spammy, there was something important in the revisions page to trigger the match. SO isn't flooding the internet with pseudo content and mash-ups. – Ian Elliott Aug 12 '09 at 19:02
Maybe I over-stated my case. My comment was that getting more search engine traffic by indexing content not on the site (i.e. deleted) should not be a seen as a "benefit." I don't see how anything "important" can end up on revisions page that is no longer in the main post. The evolving nature of wiki will keep the content fresh on SO. But Google's view of SO is that increasingly irrelevant content will continue to accumulate forever. That's all. – Robert Cartaino Aug 12 '09 at 20:54
I don't see this as a benefit since it's indexing effectively duplicate content. We block a bunch of stuff (or TRIED to, given my weak understanding of robots.txt) under this same philosophy. – Jeff Atwood Aug 17 '09 at 7:02

I think you can tell google in a site map which pages you think are most important; therefore, getting Google to show the main post (not the edits). Maybe votes should drive this as well.

share|improve this answer

I disagree with this request. I have found posts via Google in the past, because of text parts I could remember, which were revised in the meantime (the statement of the post didn't change). So Google did me a favour indexing the revision pages.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .