What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

I was reading some answers about ruby and there are a lot of links to _why's sites. _why has recently left the community and took all his sites with him.

Almost every introduction question about ruby will include at least one or two links to one of his sites. Examples:

With a bit more effort I could probably find more questions with a lot of deadlinks.

My suggestion for discussion was, what would be a good way to solve deadlinks in the answers/comments? Not every site _why created has resurfaced somewhere else, but some could be replaced by new links.

But I wanted the discussion to go beyond this specific problem of deadlinks in the ruby answers, an automatic solution would be to let the links be scanned in a batch and when found dead be replaced by a google cache or webarchive version of the page, and perhaps mark it so that the community can replace the link with an alternative?

share|improve this question
    
What a fine example of linkrot this is. –  random Aug 24 '09 at 8:07
    
Just did a fast question count using google.. whythelucklystiff: 85, hobix: 21, poignantguide.net: 90, hackety.org: 23, shoooes.net: 18, hacketyhack.net: 30, adding up to 267 questions (off course a lot overlap!) –  Davy Landman Aug 24 '09 at 12:58
    
I think webarchive is more a neutral cache source than Google. Beyond replacing dead links, I would go so far as to advocate replacing the submitted URLs with the webarchive version. If you go with a less aggressive approach, then have a background server job that submits new URLs to webarchive to take a snapshot. When there is linkrot, you have a reliable source to fall back on. –  sunk818 Nov 28 at 5:36

2 Answers 2

Well, if you come up with a fix, please apply it to the whole of the internet.

share|improve this answer
    
I know this is a universal problem, but contrary to the whole of the internet, we do have a community which could help solve some of our deadlinks. I'm not saying it would be possible to solve all of them, just that we could indicate where linkrot is occuring. For bandwidth/time it might be a good idea to limit it to the top 5 answer+comments? –  Davy Landman Aug 24 '09 at 8:15
1  
@Jeff: Maybe your team can provide a list of questions containing _why links to the community so we can do a little clean up? –  Treb Aug 24 '09 at 8:29
1  
@Treb: Or you could do it yourself with the data dump: blog.stackoverflow.com/category/cc-wiki-dump –  Stu Thompson Aug 24 '09 at 13:51
    
@Stu: Good point! It's just so much easier to ask others to do the work for you... –  Treb Aug 24 '09 at 17:41

A possible solution could look like this:

  • Extract links from all posts with an age over a ceratin threshold (6 months?)
  • Check if the link still exists
  • If it doesn't, notify the OP on his next logon to review this

This would have several problems:

  • Performing the check comes with a cost (I'm more concerned about bandwidth than CPU usage here, as Jeff likes to point out: Hardware is cheap, bandwidth can get really expensive)
  • The test is unreliable. You can send a GET command to the server. In case of *why this is pretty straightforward, the sites don't exist any longer, so the GET fails. But what if the site is down temporarily? You would need to perform a DNS lookup to know for sure. And if the page has just been removed from the site, but they don't send a 404 but provide an error page?
  • Users who often provide links in their answers would at some point get spammed with dead link notices. The more you contribute, the more notices you receive. Does the S[OFU] team really want to annoy users who contribute a lot?

OTOH, your question here shows that the system is not broken, but actually working: A user (i.e. you) detected those broken links. Maybe you already fixed some of them yourself, but I think now that you alerted the meta task force, I am confident that there will be enough volunteers willing to to some community service to clean up this mess.

share|improve this answer
    
agreed, perhaps it should be the other way around, make it possible to "link-scan" the site. (although a good google query could solve this) And let the community signal linkrot occuring. I haven't fixed the links because I'm not familiar enough with ruby to suggest the best alternative. And some of the links are in upvoted comments, so hard to overrule those. –  Davy Landman Aug 24 '09 at 9:26
1  
I like the idea (hence the upvote), but don't think it's particularly feasible. Your idea that the system is working because users notice it is actually effective, though. I recently got DV'ed on a very old question on StackOverflow, because the links I put in the article posted to my own blog, which I shut down earlier this year. A user noticed it, DV'ed the answer and commented. I got the comment notification and cleaned up the answer. System worked. This is how I'd recommend dealing with those scenarios -- DV, comment. –  John Rudy Aug 24 '09 at 10:57
    
I think suggesting alternative content pages to replace a 404 page is a very different task (requires more thought) than referencing a cached page on a different web site. –  sunk818 Nov 28 at 5:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .