28

Stack Exchange has a pretty good relationship with Google. Often, new questions on Stack Overflow may get indexed by Google within seconds of being posted, or within minutes at most.

Usually, this is a good thing, but it also makes SE a very tempting target for spammers. If Google happens to crawl a spam post before it gets deleted, there's a good chance that it will stay in Google's index for several days or even months before Google finally decides that it's really gone and drops it.

Part of the problem is that Google's crawling and indexing (like that of most search engines; but let's face it, Google is the 800 pound gorilla in the room) is very well optimized for quickly finding and indexing new content, but not particularly efficient at forgetting removed content. There are several things that SE might be able to do to speed things up, but in this particular feature-request, I'll suggest just one change that should be relatively simple to implement:

When a question is deleted via spam flags, return a HTTP "410 Gone" response instead of "404 Not Found" for it.

This should make Google drop the page from its index a bit faster. Of course, there will still be some delay before Google recrawls the page and notices the deletion in the first place, but at least the explicit 410 response should make it drop the page sooner once that happens. It won't be a panacea for spam, but it should help a little bit.


Note: This is not quite the same as this declined request, which asked for all deleted questions to return 410 instead of 404. In particular, there are two major differences between spam and other deleted questions:

  • Spam is never going to be undeleted. While mods technically can do that, in practice the only case where it may happen is if a valid question is mistakenly deleted as spam. And we're already pretty good at making sure that doesn't happen.

  • For most deleted questions, there isn't much harm in letting Google keep them in their index for a few days or weeks or months after deletion. For "index-stuffing" spam, there is.

In any case, 410 isn't a suicide pact. The HTTP spec may say that a 410 status means "that this condition is likely to be permanent", but in practice, it's no more permanent than a 301 "permanent" redirect. If a page that formerly returned 410 starts working again, Google will index it just fine. It might take them a little longer to recrawl it, but no sane search engine is going to completely ignore a valid URL just because of what it used to return. (In fact, as far as I know, the only real difference between the way Google treats 404 and 410 is in how long they'll wait for the page to come back before dropping it from their index.)


See also:

  • Just putting a reference from Official Google Webmaster about how Google handles 404s (and its sibling, 410), and the suggested approach to deleted content. "If you’re getting rid of that content entirely and don’t have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found), so it’s immaterial to us whether you return one or the other." – No Distraction Wizard Nov 18 '15 at 1:25
  • @AndrewT: The page you linked to seems to be a bit outdated. In this video from last year, Matt Cutts says that there nowadays is a (minor) difference, as I described above. – Ilmari Karonen Nov 18 '15 at 1:34

You must log in to answer this question.

Browse other questions tagged .