How to clean up dead links

Question

I was reading some answers about Ruby and there are a lot of links to _why's sites. _why has recently left the community and took all his sites with him.

Almost every introduction question about Ruby will include at least one or two links to one of his sites. Examples:

• https://stackoverflow.com/questions/6806/what-is-the-best-way-to-learn-ruby
• https://stackoverflow.com/questions/32516/getting-started-with-ruby-development
• Is there a tutorial that teaches common Ruby programming idioms used by experienced programmers, but may not be obvious to newcomers?
• Learning Ruby: recommended blogs to read?
• What's the best/easiest GUI Library for Ruby?

With a bit more effort I could probably find more questions with a lot of dead links.

My suggestion for discussion was, what would be a good way to solve dead links in the answers/comments? Not every site _why created has resurfaced somewhere else, but some could be replaced by new links.

But I wanted the discussion to go beyond this specific problem of dead links in the Ruby answers, an automatic solution would be to let the links be scanned in a batch and when found dead be replaced by a Google cache or Wayback Machine version of the page, and perhaps mark it so that the community can replace the link with an alternative?

Answer 1

Well, if you come up with a fix, please apply it to the whole of the internet.

Answer 2

A possible solution could look like this:

• Extract links from all posts with an age over a ceratin threshold (6 months?)
• Check if the link still exists
• If it doesn't, notify the OP on his next logon to review this

This would have several problems:

• Performing the check comes with a cost (I'm more concerned about bandwidth than CPU usage here, as Jeff likes to point out: Hardware is cheap, bandwidth can get really expensive)
• The test is unreliable. You can send a GET command to the server. In case of _why this is pretty straightforward, the sites don't exist any longer, so the GET fails. But what if the site is down temporarily? You would need to perform a DNS lookup to know for sure. And if the page has just been removed from the site, but they don't send a 404 but provide an error page?
• Users who often provide links in their answers would at some point get spammed with dead link notices. The more you contribute, the more notices you receive. Does the S[OFU] team really want to annoy users who contribute a lot?

OTOH, your question here shows that the system is not broken, but actually working: A user (i.e. you) detected those broken links. Maybe you already fixed some of them yourself, but I think now that you alerted the meta task force, I am confident that there will be enough volunteers willing to to some community service to clean up this mess.