What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

There are many links posted in answers, and some in questions, that are either stale or broken.

An example is the change to Ant on-line documentation this year to do away with separate 'Optional' and 'Core' tasks areas in the documentation - the old links 404 now. I've fixed a bunch of 'optionaltasks' broken links, but searching for 'coretasks' in Stack Overflow questions returns over 300 matches...

Are there any tools for us to review and fix these - with an emphasis on the fixing - without sifting through completely manually?

Is it time for some refactoring tools for Stack Overflow content, in particular, for links?

Also, I'd be interested to know if there is an analysis of links (as found in the Stack Overflow data dumps) out there.
Answer: there is a rough analysis here of a sample of links and images, but it's getting a bit out-of-date.

share|improve this question
    
...also: images? Today I learned about the new subscription model for the Mac screen capturing and image sharing tool Skitch. I was happy to learn, if I understand correctly, that existing images will not be deleted, even when not becoming a subscriber. But there's a lot of images out there that do not use the Imgur hosting. The API is still read-only, right, or could that be used? –  Arjan Dec 6 '10 at 0:21
    
@Arjan: Yup, see my answer below. –  uɐɯsO uɐɥʇɐN Dec 6 '10 at 0:22
    
Well, @George, that was actually your request for some work with the API that I referred to. What's up with that? ;-) –  Arjan Dec 6 '10 at 0:25
    
@Arjan: I totally forgot about that question! –  uɐɯsO uɐɥʇɐN Dec 6 '10 at 0:34
3  
Unexpected type of link rot; apparently some kind of referral/affiliate links can also go bad: I hate these just because someone uses them to answer a question, then the link dies, and the answer is now useless. Just had an accepted answer become useless because of this not too long ago. –  Arjan Jan 21 '11 at 14:08
    
related: can we have a flag allowing reader to indicate link rot? Given that moderator attention -> other is not mentioned I assume it's not (not widely) used to indicate such issues –  gnat Aug 16 '11 at 1:35
    
What about links in comments, which are not generally editable? How can they be fixed? –  Craig McQueen Sep 12 '13 at 4:57

7 Answers 7

up vote 10 down vote accepted

Nice: since April 2012 Stack Exchange is experimenting with automatically following links. If a broken link is found, that might make Community add a "special comment" to the post asking the creator to fix it, and might add it to a list in /review. Also some badges to help drive the fixing might be introduced.

Details at Does Stack Exchange crawl websites?

share|improve this answer
    
Although, of note it appears they no longer have a public review of these messages. –  user7116 Mar 11 '13 at 19:34
    
This has been aborted. –  Shadow Wizard Nov 2 at 19:30

I think the pragmatic solution might be to have a page in the 10K tools section for posts with broken links. We could then go in pick half a dozen and fix them.

Not sure what the best way to identify the links is though.

Fixing them would be a way to get [Copy Editor] badge.

share|improve this answer
    
I'd be very happy to earn the badge, but I'd rather do it by editing copy than doing a manual search-and-replace. –  martin clayton Dec 5 '10 at 23:57
    
@martin - I wasn't suggesting a manual search and replace (unless I've missed the point of your comment). Use tools to identify the broken links but people to replace them with the correct content as it won't usually be as your example in the question. –  ChrisF Dec 6 '10 at 0:01
    
Proposals to check links have not proved popular before - I agree that would be beneficial. I marked the question with the discussion tag because I'm unsure what form the new feature aught to have. Perhaps a list of posts with hanging links would be enough. But for the Ant example it seems labourious to fix them manually. –  martin clayton Dec 6 '10 at 0:14
    
Identifying them is only half the problem. Once they are the busted links (in bad answers to boot) get ignored if the perp has enough "juice". –  Awesome Poodles Aug 16 '11 at 1:22

The API and/or data dumps would be an excellent way to retrieve this information and then parse it in some way looking for broken links.

Something like this:

  • Retrieve 100 questions and include their answers.
  • Use a DOM tool to enumerate all linked content and send an HTTP HEAD request to those URLs to make sure they are valid.
  • Take note of all of the ones that return 404 and store the post IDs somewhere.
  • Repeat the above steps until all posts have been processed.

After that, a real person (with 3k+ rep) would have to manually edit the posts to remove the content.


Oh, and you forgot the biggest nuisance of them all... broken image URLs.

share|improve this answer
3  
Too bad there's many 404 pages out there that actually send a HTTP 200 OK code. Ha! Maybe whenever posting a link, SOFU should make a request to that same domain with some non-existing URL. If that returns 200 OK then that domain should be banned from linking. ;-) –  Arjan Dec 6 '10 at 0:27
    
With over 1 million (que Dr. Evil voice) questions to sift through it would be nice if there were some way for the retrieval queue to be populated by users finding broken links as well as by a continuous scan. –  martin clayton Dec 6 '10 at 0:35
1  
Notice I omitted requests that fail - because servers go down now and then and equipment malfunctions. –  uɐɯsO uɐɥʇɐN Dec 6 '10 at 0:35
2  
Sure, we could do it with the data dump - but this really should be something stackoverflow does in the background, so that all links get checked periodically –  Joel Coehoorn Dec 6 '10 at 23:45
    
@Joel - I'd be interested in seeing it done effectively in the background too, the problem is just the ever (and rapidly) expanded volume of SO. –  Tim Post Jan 21 '11 at 22:30
    
Image rot is something we've started to plan for. –  Rebecca Chernoff Aug 16 '11 at 1:27
    
You can always use a link checking tool like XENU to do it yourself. –  sunk818 Nov 6 at 17:01
    
@RebeccaChernoff I read images on imgur expire after 6 months. These images are not crawled systematically to keep them alive? –  sunk818 Nov 6 at 17:04

The neatest tool to combat link rot is the Wayback Machine - http://web.archive.org/web/*/http://stackexchange.com . Would be nice if mods or 3k+ers could have the option to browse through the Web Archive (of course, it is always possible to do this manually, but the aforementioned persons are already busy...)

share|improve this answer

If anyone knows of URLs that are often seen, but might need special care if some solution would differentiate between domains, then maybe add those here?

  • Dropbox uses URLs like dl.dropbox.com/u/userid/filename and the shortened db.tt/xxxx for public Dropbox files. These files clutter the user's Public folder, and count towards the user's disk space, so might be removed relatively soon.
share|improve this answer

My related question was closed as a dupe, so here's an idea for SE internal deleted links. External links are not reasonably manageable.

If SE had a link-table with an internal origin and an internal destination link page (I don't know if this table exists). A [deleted] boolean column could be added to the destination link column. This way, whenever a page is rendered for -10Krep user the dead links could be disabled. When the same page is rendered for +10Krep users, the dead links are rendered normally.

share|improve this answer

To prevent link rot, what about a cronjob that submits external links to the Wayback Machine? The external link is then replaced with the Wayback Machine link. This way, you have a snapshot of the web page at the time it was saved.

I personally use a one line javascript to save the current page to the Wayback Machine, then link the Wayback Machine URL in my answers.

javascript:q=(document.location.href);void(open('http://web.archive.org/save/'+location.href.replace(/https?:\/\//i, ""),'_self ','resizable,location,menubar,toolbar,scrollbars,status'));

This has proven to be good way to mitigate link rot.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .