3

I was reading some answers about Ruby and there are a lot of links to _why's sites. _why has recently left the community and took all his sites with him.

Almost every introduction question about Ruby will include at least one or two links to one of his sites. Examples:

With a bit more effort I could probably find more questions with a lot of dead links.

My suggestion for discussion was, what would be a good way to solve dead links in the answers/comments? Not every site _why created has resurfaced somewhere else, but some could be replaced by new links.

But I wanted the discussion to go beyond this specific problem of dead links in the Ruby answers, an automatic solution would be to let the links be scanned in a batch and when found dead be replaced by a Google cache or Wayback Machine version of the page, and perhaps mark it so that the community can replace the link with an alternative?

3
  • What a fine example of linkrot this is.
    – random
    Aug 24, 2009 at 8:07
  • Just did a fast question count using google.. whythelucklystiff: 85, hobix: 21, poignantguide.net: 90, hackety.org: 23, shoooes.net: 18, hacketyhack.net: 30, adding up to 267 questions (off course a lot overlap!) Aug 24, 2009 at 12:58
  • I think webarchive is more a neutral cache source than Google. Beyond replacing dead links, I would go so far as to advocate replacing the submitted URLs with the webarchive version. If you go with a less aggressive approach, then have a background server job that submits new URLs to webarchive to take a snapshot. When there is linkrot, you have a reliable source to fall back on.
    – Sun
    Nov 28, 2014 at 5:36

2 Answers 2

6

Well, if you come up with a fix, please apply it to the whole of the internet.

4
  • I know this is a universal problem, but contrary to the whole of the internet, we do have a community which could help solve some of our deadlinks. I'm not saying it would be possible to solve all of them, just that we could indicate where linkrot is occuring. For bandwidth/time it might be a good idea to limit it to the top 5 answer+comments? Aug 24, 2009 at 8:15
  • 1
    @Jeff: Maybe your team can provide a list of questions containing _why links to the community so we can do a little clean up?
    – Treb
    Aug 24, 2009 at 8:29
  • 1
    @Treb: Or you could do it yourself with the data dump: blog.stackoverflow.com/category/cc-wiki-dump Aug 24, 2009 at 13:51
  • @Stu: Good point! It's just so much easier to ask others to do the work for you...
    – Treb
    Aug 24, 2009 at 17:41
3

A possible solution could look like this:

  • Extract links from all posts with an age over a ceratin threshold (6 months?)
  • Check if the link still exists
  • If it doesn't, notify the OP on his next logon to review this

This would have several problems:

  • Performing the check comes with a cost (I'm more concerned about bandwidth than CPU usage here, as Jeff likes to point out: Hardware is cheap, bandwidth can get really expensive)
  • The test is unreliable. You can send a GET command to the server. In case of _why this is pretty straightforward, the sites don't exist any longer, so the GET fails. But what if the site is down temporarily? You would need to perform a DNS lookup to know for sure. And if the page has just been removed from the site, but they don't send a 404 but provide an error page?
  • Users who often provide links in their answers would at some point get spammed with dead link notices. The more you contribute, the more notices you receive. Does the S[OFU] team really want to annoy users who contribute a lot?

OTOH, your question here shows that the system is not broken, but actually working: A user (i.e. you) detected those broken links. Maybe you already fixed some of them yourself, but I think now that you alerted the meta task force, I am confident that there will be enough volunteers willing to to some community service to clean up this mess.

3
  • agreed, perhaps it should be the other way around, make it possible to "link-scan" the site. (although a good google query could solve this) And let the community signal linkrot occuring. I haven't fixed the links because I'm not familiar enough with ruby to suggest the best alternative. And some of the links are in upvoted comments, so hard to overrule those. Aug 24, 2009 at 9:26
  • 1
    I like the idea (hence the upvote), but don't think it's particularly feasible. Your idea that the system is working because users notice it is actually effective, though. I recently got DV'ed on a very old question on StackOverflow, because the links I put in the article posted to my own blog, which I shut down earlier this year. A user noticed it, DV'ed the answer and commented. I got the comment notification and cleaned up the answer. System worked. This is how I'd recommend dealing with those scenarios -- DV, comment.
    – John Rudy
    Aug 24, 2009 at 10:57
  • I think suggesting alternative content pages to replace a 404 page is a very different task (requires more thought) than referencing a cached page on a different web site.
    – Sun
    Nov 28, 2014 at 5:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .