How should we handle "conditionally" broken links?

Question

As part of my big mission to rid us of broken links I have come across an edge case I would like to address.

Links such as: http://stackexchange.com/filters/new are broken, unless one is logged on to Stack Exchange.

I am not done my crawl, but am pretty sure we are going to find a fair amount of links that give you a 404 unless a user is registered. "Go here ... (requires registration)"

There are 2 parts to this question:

1. Should we internally return a status code that is not 404 (or 418 for that matter) when people hit resources they do not have permission to see?

2. Are this links considered "valid" should we allow them as a community? (I tend to think that yes on a case by case basis - and would need to amend the validation process to amend)

Thoughts?

Answer 1

I say, if we have to register in order to view the content that they're pointing us to, then the link is broken, because that's very unlikely to happen. It's bad enough they're already pointing us off-site to view it, but now we have to register too? Don't think so. Those links should be subject to removal just as any broken link should be.

As far as I'm aware, the crawler only puts a comment there saying "this link is broken" right? I'm fairly sure you can exclude Stack Exchange links, especially on meta sites. Any other sites that fall under that edge-case category should still be reviewed anyways, so the comment is helpful. If the link is a valid edge-case, just remove the comment and let it continue on. I'd be much more worried about the bad links we'd be letting continue on because we're trying to avoid the good links.

This seems to follow the same issue that gets complained about with the "low quality posts" tab. Sure, sometimes it turns up false positive (ok, lots of the time), but it's better than letting the ones that need reviewed slip through the cracks.

Answer 2

1. Semantically, if a user is not supposed to see a particular resource, 401 or 403 would make more sense (I'm not sure which one applies: 401 implies that basic authentication could work, 403 implies that nobody can see the resource). But specific HTTP error codes seem to have gone the way of the dinosaur, so, shrug, you might as well return 404.

2. How would you know whether a link is conditionally broken? If it returns an error status to the crawler, include it in the review list. I expect that we'll eventually determine a list of domains where 404 should be whitelisted. This list will probably be site-specific; for example, http://SITE.stackexchange.com/* should be whitelisted on meta.SITE.stackexchange.com, as it's common to link to deleted questions. Links to scientific papers behind a paywall may be considered legitimate in certain communities; it should be up to each site's community to decide. Links to sites that require registration should be considered on a case-by-case basis; your example with http://stackexchange.com/filters/new can be generalized to many sites (answer to a [facebook] question on Webapps.SE: “go to your user options at …”; answer to a [some-hosting-provider] question on Webmasters.SE: “go to your control paenl at …”; etc). Again, I expect some whitelists to emerge.

I hope you aren't considering banning dead links upon posting. A warning would probably a good thing though: the bad link could be a typo, or it could be a bad idea (e.g. a link behind a paywall); but let the user say “yes, it's legitimate”.

Answer 3

I've been in the habit of labeling links that I install to pages that have conditional visibility. Usually that means (10k only) to make it clear that the link is not useful for everyone.