Validating that which can not be validated, how do we deal with robots exclusions?

Question

Many high profile sites have taken a "white-list" based approach to robots.txt

Examples are:

• Github
• Facebook
• Delicious

The result is that we are not allowed to automatically check that any external links to these sites are kosher.

The scale of the problem is fairly big, for example (for a sample of 5000 of the earliest links)

• 3.5% are uncrawlable due to robots exclusion.
• 5.8% are broken

---

How should we be dealing with these uncrawlable links?

Answer 1

Here's how we feel about it at GitHub, from Aman Gupta:

Normal crawlers wreak havoc on our servers, because they can find content that isn't actually commonly seen (i.e. a persistent crawler can walk an entire Git history by following parent links), and it blows any attempt at caching by filling the cache with bogus entries.

In this case, because you're not crawling, you're not quite the same as a robot - the worries we have about GoogleBot don't apply here.

Answer 2

I would not worry about it for now. What you're doing isn't very much different than a user interface doing an AJAX check to validate a link to show a little green check mark next to it in a UI. Due to the same origin policy, those have to go through a proxy and checking robots.txt in that scenario would seem extremely silly.

For the long term, you could set up a proper bot (just like Googlebot) and send the corresponding user agent, give it time to circulate and then check robots.txt for exclusions, but I really think doing so is going way above and beyond what a good netizen should be doing. You aren't crawling or indexing, you're just sending a HEAD request to see the status of any given link, and you're doing it nicely. There's no good reason to make two requests instead of one initially, or deal with the complexity of caching a bunch of robots.txt files to avoid it.

Finally, I think most site owners would want Stack Overflow to make every effort to verify inbound links to their site, as the only alternative would be to remove those that can't be verified wholesale.

Answer 3

Since robots.txt doesn't force us to follow it:

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

^(source)

Basically, you want to be the good-guy cooperating crawler, right? Well, we don't crawl links. Crawlers spread out through every link they get and basically end up parsing nearly the whole site. That's bad when a site has tons of links that don't need to be crawled.

On the other hand, we take a link, point, and shoot. There's no "crawling" involved--nothing that can spiral into bigtime server load if we hit a page with millions of links. That's pretty much why we have robots.txt-- to prevent the robots from taking over the world overloading the site.

Though, if you notice, there's the "other web robots" in the Wikipedia snippet. Not complying with robots.txt will mean that we are complying with the intentions of robots.txt, but not with the actual rules. Which is OK, since overly broad rules are meant to be broken. And, it's not really a "rule" per se--it's a "listen if you know what's good for you" thingy:

Despite the use of the terms "allow" and "disallow", the protocol is purely advisory.

And, we're old enough to know what's good for us without having them tell us, right? ;-)

So just point-and-shoot. Don't even fetch robots.txt, it unecessarily increases server load a teensy bit. Though, as @Dennis suggested, you can consult robots.txt if you have a truckload of links on the same domain that you need to verify.