Some of the internet links on the site are not working, e.g. link #1 in the answer to this post.

Is there an automatic routine check that all links on the site are active? Would it be useful for authors to get a message if a link in one of their posts is not working correctly?

  • 1
    I am rolling back because your edit is a totally different question. Since you already got answer to your original question, it's not a good thing to do. Please start a new bug report. Aug 11 '15 at 12:31
  • OK. I agree the original question is closely related to the link rot question. Aug 11 '15 at 12:52

We have something that does this, but it's not optimal. When activated, the system will routinely send HEAD requests in an extremely polite manner, and put anything that doesn't come back with a HTTP 200 response into a queue for users to check, as well as notifying the author of the post about a broken link.

In theory, this should work well, in reality it doesn't. When a considerable amount of context for a post depends on an external resource and that resource goes away, it can be very difficult to find a replacement for it - fixing it often means just rewriting the post. The Internet Archive's Wayback Machine helps, if you can find a copy of the resource at the time the post was written. If it's not there, and the answer isn't useful any longer without a working link, you're looking at a lot of work or deleting the post if it's no longer of use.

Wayback does have an API and we experimented with automating this to some extent. We can ask the archive if it has a given resource, and the dates of the snapshots it has available. If it has a copy, and a snapshot in reasonably close proximity to the date the post was written then it's likely a good match. But, this just prolongs a semi-broken state - what happens if the archive no longer has that resource one day? It doesn't fix the underlying issue. It also needs human supervision.

Posts that people actually care about are generally well-maintained, even when it comes to link rot. What the automated system tends to churn are links on stuff that hasn't been viewed in a long time, and the effort required to fix this stuff is generally more than the value of the stuff being fixed. That's why we didn't continue putting it in a queue for folks to work on, it just piles up, there are few 'easy' fixes, and it takes time that could be given to more important content.

tl;dr; - the suggested edit system is actually a better fit for this as people find things opportunistically and fix them. It's clear signal that someone found something, found it useful, and made sure it remained useful for others. In other words, content that at least someone still cares about.

Tasking people to fix a link to a bug report surrounding an ancient version of some language that has long fallen out of relevancy is just not an ideal use of anyone's time.

That said, I do want to experiment with it some more, as time permits. Squelching alerts on everything under a certain amount of recent views would drop a pretty big chunk of noise. I won't say that link rot isn't a problem, but there are more pressing and important ones to tackle first.

  • Tim, I'm going to toot my own horn just a bit. I posted on MSO last week about link rot on SO. Several answers (including one I posted) suggested bringing back the broken link queue or utilizing the wayback machine's API as possible fixes.
    – Andy
    Aug 11 '15 at 5:37
  • "What the automated system tends to churn are links on stuff that hasn't been viewed in a long time..." - I didn't have time since last view, but I did find that roughly 2/3s of the posts in my sample that had broken links had been viewed less than once a day since the time of their creation.
    – Andy
    Aug 11 '15 at 5:39
  • @Andy I've pursued both ideas internally (and somewhat aggressively) after talking to Jeff Atwood about it. Nobody ever intended to task really smart people with fixing broken links, and the wayback API does offer a high degree of sanity. The resistance is partly technical (community would be making and owning a lot of automated edits), and partly philosophical (as in, would we be missing a ton of weeds that really need pulling?) [1/2]
    – Tim Post
    Aug 11 '15 at 5:44
  • I am adding an initiative to our queue to revisit this, but we need to finish the overhaul of the moderator tooling first, along with tweaks to the help & improvement / triage queues. [2/2]
    – Tim Post
    Aug 11 '15 at 5:45

The link does not work because it has a typo in it. The editor does not automatically validate/checks the URL entered by OP. But, the community users (high-reps. or with edit privilege), moderators and OP can "edit" the post and fix the URL. Edit also earns you reputations and badges for being especially helpful. I think it would be a tremendous burden on the system to go through each and every post to validate the URL's in them. IMHO, let the users do these tiny editing type jobs and leave system for the important stuff.

Here is the source code of it: (notice "/" before http)

<a href="/http://en.wikipedia.org/wiki/Simple_linear_regression">the wikipedia article on simple linear regression</a>

It should be as:

<a href="http://en.wikipedia.org/wiki/Simple_linear_regression">the wikipedia article on simple linear regression</a>

Works: the wikipedia article on simple linear regression

  • I tried to edit the link, but edits are limited to 6 characters minimum, and in this case the necessary edit is only one character. Is there a way to distinguish typo's from other edits? Aug 11 '15 at 12:22
  • 1
    @HillelBar-Gera no there's no such way, and several feature requests asking for this have been declined. Aug 11 '15 at 12:44
  • @HillelBar-Gera I agree with ShadowWizard. The system is designed "6 characters minimum" requirement and we can't change that. I know you you can't edit but high-reps, moderators and OP can edit his/her own post. I think leaving a comment for OP is the best one can do when no edit is allowed due to 6 char. limit. I agree with what Tim Post suggested in his answer "tl;dr; - the suggested edit system is actually a better fit..." Aug 11 '15 at 13:13

Not the answer you're looking for? Browse other questions tagged .