57

There are many links posted in answers, and some in questions, that are either stale or broken.

An example is the change to Ant on-line documentation this year to do away with separate 'Optional' and 'Core' tasks areas in the documentation - the old links 404 now. I've fixed a bunch of 'optionaltasks' broken links, but searching for 'coretasks' in Stack Overflow questions returns over 300 matches...

Are there any tools for us to review and fix these - with an emphasis on the fixing - without sifting through completely manually?

Is it time for some refactoring tools for Stack Overflow content, in particular, for links?

Also, I'd be interested to know if there is an analysis of links (as found in the Stack Overflow data dumps) out there.
Answer: there is a rough analysis here of a sample of links and images, but it's getting a bit out-of-date.

Improve this question
7
  • ...also: images? Today I learned about the new subscription model for the Mac screen capturing and image sharing tool Skitch. I was happy to learn, if I understand correctly, that existing images will not be deleted, even when not becoming a subscriber. But there's a lot of images out there that do not use the Imgur hosting. The API is still read-only, right, or could that be used?
    – Arjan
    Commented Dec 6, 2010 at 0:21
  • @Arjan: Yup, see my answer below.
    – uɐɯsO uɐɥʇɐN
    Commented Dec 6, 2010 at 0:22
  • Well, @George, that was actually your request for some work with the API that I referred to. What's up with that? ;-)
    – Arjan
    Commented Dec 6, 2010 at 0:25
  • @Arjan: I totally forgot about that question!
    – uɐɯsO uɐɥʇɐN
    Commented Dec 6, 2010 at 0:34
  • 4
    Unexpected type of link rot; apparently some kind of referral/affiliate links can also go bad: I hate these just because someone uses them to answer a question, then the link dies, and the answer is now useless. Just had an accepted answer become useless because of this not too long ago.
    – Arjan
    Commented Jan 21, 2011 at 14:08
  • related: can we have a flag allowing reader to indicate link rot? Given that moderator attention -> other is not mentioned I assume it's not (not widely) used to indicate such issues
    – gnat
    Commented Aug 16, 2011 at 1:35
  • 2
    What about links in comments, which are not generally editable? How can they be fixed?
    – Craig McQueen
    Commented Sep 12, 2013 at 4:57

7 Answers 7

Reset to default
16

Nice: since around April 2012 Stack Exchange was experimenting with automatically following links. If a broken link was found, that might make Community add a "special comment" to the post asking the creator to fix it, and might add it to a list in /review. Also some badges to help drive the fixing might be introduced.

Details at Does Stack Exchange crawl websites? but it seems this is no longer active.

Improve this answer
7
  • Although, of note it appears they no longer have a public review of these messages.
    – user7116
    Commented Mar 11, 2013 at 19:34
  • This has been aborted.
    – Shadow Wizard
    Commented Nov 2, 2014 at 19:30
  • 1
    Why it was aborted, is there any follow-up?
    – kenorb
    Commented Apr 21, 2015 at 21:58
  • @kenorb nothing official was posted, guess they found it was too faulty and not worth the efforts.
    – Shadow Wizard
    Commented Aug 10, 2015 at 6:02
  • Looks like you missed some comments here... :)
    – Shadow Wizard
    Commented Oct 11, 2015 at 10:19
  • @ShadowWizard. I edited because of those comments; did I miss something? Or, maybe you mean that I'm a bit late? Yes, very true then: though I guess I did read them earlier, I failed to act in a timely matter. :-)
    – Arjan
    Commented Oct 11, 2015 at 11:11
  • Better late than never, I say. :-)
    – Shadow Wizard
    Commented Oct 11, 2015 at 12:15
15

I think the pragmatic solution might be to have a page in the 10K tools section for posts with broken links. We could then go in pick half a dozen and fix them.

Not sure what the best way to identify the links is though.

Fixing them would be a way to get [Copy Editor] badge.

Improve this answer
3
  • I'd be very happy to earn the badge, but I'd rather do it by editing copy than doing a manual search-and-replace.
    – martin clayton
    Commented Dec 5, 2010 at 23:57
  • @martin - I wasn't suggesting a manual search and replace (unless I've missed the point of your comment). Use tools to identify the broken links but people to replace them with the correct content as it won't usually be as your example in the question.
    – ChrisF Mod
    Commented Dec 6, 2010 at 0:01
  • Proposals to check links have not proved popular before - I agree that would be beneficial. I marked the question with the discussion tag because I'm unsure what form the new feature aught to have. Perhaps a list of posts with hanging links would be enough. But for the Ant example it seems labourious to fix them manually.
    – martin clayton
    Commented Dec 6, 2010 at 0:14
11
+50

The API and/or data dumps would be an excellent way to retrieve this information and then parse it in some way looking for broken links.

Something like this:

  • Retrieve 100 questions and include their answers.
  • Use a DOM tool to enumerate all linked content and send an HTTP HEAD request to those URLs to make sure they are valid.
  • Take note of all of the ones that return 404 and store the post IDs somewhere.
  • Repeat the above steps until all posts have been processed.

After that, a real person (with 3k+ rep) would have to manually edit the posts to remove the content.

Oh, and you forgot the biggest nuisance of them all... broken image URLs.

Improve this answer
8
  • 3
    Too bad there's many 404 pages out there that actually send a HTTP 200 OK code. Ha! Maybe whenever posting a link, SOFU should make a request to that same domain with some non-existing URL. If that returns 200 OK then that domain should be banned from linking. ;-)
    – Arjan
    Commented Dec 6, 2010 at 0:27
  • With over 1 million (que Dr. Evil voice) questions to sift through it would be nice if there were some way for the retrieval queue to be populated by users finding broken links as well as by a continuous scan.
    – martin clayton
    Commented Dec 6, 2010 at 0:35
  • 1
    Notice I omitted requests that fail - because servers go down now and then and equipment malfunctions.
    – uɐɯsO uɐɥʇɐN
    Commented Dec 6, 2010 at 0:35
  • 2
    Sure, we could do it with the data dump - but this really should be something stackoverflow does in the background, so that all links get checked periodically
    – Joel Coehoorn
    Commented Dec 6, 2010 at 23:45
  • @Joel - I'd be interested in seeing it done effectively in the background too, the problem is just the ever (and rapidly) expanded volume of SO.
    – user50049
    Commented Jan 21, 2011 at 22:30
  • Image rot is something we've started to plan for.
    – Rebecca Chernoff
    Commented Aug 16, 2011 at 1:27
  • 1
    You can always use a link checking tool like XENU to do it yourself.
    – Sun
    Commented Nov 6, 2014 at 17:01
  • @RebeccaChernoff I read images on imgur expire after 6 months. These images are not crawled systematically to keep them alive?
    – Sun
    Commented Nov 6, 2014 at 17:04
7

The neatest tool to combat link rot is the Wayback Machine - http://web.archive.org/web/*/http://stackexchange.com . Would be nice if mods or 3k+ers could have the option to browse through the Web Archive (of course, it is always possible to do this manually, but the aforementioned persons are already busy...)

Improve this answer
1
  • 2
    Perhaps SE could encourage link-providing users to also provide an additional link to an archive. That's what Wikipedia does with its cite web template. archive.org also makes "saving" (i.e., archiving a previously unarchived page) easy, which reinforces the habit.
    – TomRoche
    Commented Mar 27, 2016 at 23:02
4

To prevent link rot, what about a cronjob that submits external links to the Wayback Machine? The external link is then replaced with the Wayback Machine link. This way, you have a snapshot of the web page at the time it was saved.

I personally use a one line javascript to save the current page to the Wayback Machine, then link the Wayback Machine URL in my answers.

javascript:q=(document.location.href);void(open('http://web.archive.org/save/'+location.href.replace(/https?:\/\//i, ""),'_self ','resizable,location,menubar,toolbar,scrollbars,status'));

This has proven to be good way to mitigate link rot.

Improve this answer
2
  • Are you saying a cron job should replace links with wayback machine links pre-emptively? There is a problem with this plan: while most websites work find in wayback machine there are some that do not save correctly for some reason. The cron job you are proposing would occasionally replace a working link with a link to a page that looks wrong or even lacks content.
    – Andrey Bienkowski
    Commented Dec 6, 2023 at 11:08
  • Good point. As I think about this almost 10 years later, I'd still want wayback submissions of URLs cited on SE. If the original link has a 404, SE could fall back to a wayback archive. It'd mean SE would have to scan the server header status and redirect to wayback if necessary.
    – Sun
    Commented Dec 7, 2023 at 4:24
2

If anyone knows of URLs that are often seen, but might need special care if some solution would differentiate between domains, then maybe add those here?

  • Dropbox uses URLs like dl.dropbox.com/u/userid/filename and the shortened db.tt/xxxx for public Dropbox files. These files clutter the user's Public folder, and count towards the user's disk space, so might be removed relatively soon.
Improve this answer
1

My related question was closed as a dupe, so here's an idea for SE internal deleted links. External links are not reasonably manageable.

If SE had a link-table with an internal origin and an internal destination link page (I don't know if this table exists). A [deleted] boolean column could be added to the destination link column. This way, whenever a page is rendered for -10Krep user the dead links could be disabled. When the same page is rendered for +10Krep users, the dead links are rendered normally.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .