Many high profile sites have taken a "white-list" based approach to robots.txt

Examples are:

The result is that we are not allowed to automatically check that any external links to these sites are kosher.

The scale of the problem is fairly big, for example (for a sample of 5000 of the earliest links)

  • 3.5% are uncrawlable due to robots exclusion.
  • 5.8% are broken

How should we be dealing with these uncrawlable links?

  • What is the error code the excluded sites are returning? – user102937 May 16 '12 at 2:38
  • 3
    @RobertHarvey the exclusion in robots bans us from checking the links ... there is no error. – waffles May 16 '12 at 2:40
  • How could that be? If you try to hit the link, there has to be some sort of response code. – user102937 May 16 '12 at 2:41
  • 6
    @waffles: Change your user-agent to Googlebot. Simple. – Manishearth May 16 '12 at 2:41
  • @RobertHarvey we hit robots.txt first, parse it, it tells us we are not allowed to check ANY links on the site, so we stop. – waffles May 16 '12 at 2:41
  • @TimManishEarth A WINNING suggestion :) – waffles May 16 '12 at 2:42
  • 2
    @RobertHarvey: Basically, it's a standard that you're expected to follow. You can sidestep it, of course--I don't think the system forces you to parse robots.txt – Manishearth May 16 '12 at 2:42
  • 2
    @TimManishEarth: That's what I think you should do. It's not like we're crawling the entire site, or even a substantial part of it. Nor are we scraping for email addresses. Google will respect it, of course, because they must crawl the entire site, or at least that part that's publicly visible. – user102937 May 16 '12 at 2:43
  • @RobertHarvey: (Source:Wikipedia) The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. – Manishearth May 16 '12 at 2:43
  • 1
    @RobertHarvey: Yeah, me too. We don't exactly crawl the web page, we just point-and-shoot. – Manishearth May 16 '12 at 2:44
  • 3
    For context I am performing a throttled HEAD request to each link, respecting crawl delay and all. – waffles May 16 '12 at 2:45
  • 4
    You're not really crawling if you're just poking Abe Vigoda – random May 16 '12 at 2:51
  • 1
    So uh, maybe move this to webmasters as it's awfully not SO specific? – Kevin Montrose May 16 '12 at 4:24
  • 1
    Don't cross post, just migrate. This discussion shouldn't be "hidden away" on meta; webmasters is the correct place for this. – Kevin Montrose May 16 '12 at 4:42
  • 2
    It's correct here, but if you remove the implicit "we, Stack Exchange" it fits elsewhere and should go there as people looking for this information are unlikely to look on a meta site. – Kevin Montrose May 16 '12 at 4:45

Here's how we feel about it at GitHub, from Aman Gupta:

alt text

Normal crawlers wreak havoc on our servers, because they can find content that isn't actually commonly seen (i.e. a persistent crawler can walk an entire Git history by following parent links), and it blows any attempt at caching by filling the cache with bogus entries.

In this case, because you're not crawling, you're not quite the same as a robot - the worries we have about GoogleBot don't apply here.

| improve this answer | |
  • What about abnormal bot that don't respect the robots.txt, because you are somehow pushing robots to not respect your robots.txt – dvhh May 16 '12 at 4:02

I would not worry about it for now. What you're doing isn't very much different than a user interface doing an AJAX check to validate a link to show a little green check mark next to it in a UI. Due to the same origin policy, those have to go through a proxy and checking robots.txt in that scenario would seem extremely silly.

For the long term, you could set up a proper bot (just like Googlebot) and send the corresponding user agent, give it time to circulate and then check robots.txt for exclusions, but I really think doing so is going way above and beyond what a good netizen should be doing. You aren't crawling or indexing, you're just sending a HEAD request to see the status of any given link, and you're doing it nicely. There's no good reason to make two requests instead of one initially, or deal with the complexity of caching a bunch of robots.txt files to avoid it.

Finally, I think most site owners would want Stack Overflow to make every effort to verify inbound links to their site, as the only alternative would be to remove those that can't be verified wholesale.

| improve this answer | |

Since robots.txt doesn't force us to follow it:

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.


Basically, you want to be the good-guy cooperating crawler, right? Well, we don't crawl links. Crawlers spread out through every link they get and basically end up parsing nearly the whole site. That's bad when a site has tons of links that don't need to be crawled.

On the other hand, we take a link, point, and shoot. There's no "crawling" involved--nothing that can spiral into bigtime server load if we hit a page with millions of links. That's pretty much why we have robots.txt-- to prevent the robots from taking over the world overloading the site.

Though, if you notice, there's the "other web robots" in the Wikipedia snippet. Not complying with robots.txt will mean that we are complying with the intentions of robots.txt, but not with the actual rules. Which is OK, since overly broad rules are meant to be broken. And, it's not really a "rule" per se--it's a "listen if you know what's good for you" thingy:

Despite the use of the terms "allow" and "disallow", the protocol is purely advisory.

And, we're old enough to know what's good for us without having them tell us, right? ;-)

So just point-and-shoot. Don't even fetch robots.txt, it unecessarily increases server load a teensy bit. Though, as @Dennis suggested, you can consult robots.txt if you have a truckload of links on the same domain that you need to verify.

| improve this answer | |
  • Well, with different highlighting, this becomes is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. – Dennis May 16 '12 at 2:51
  • 4
    one could argue that even hitting robots.txt for our context is a bad way to act ... especially if we only need to point and shoot at a "single" link – waffles May 16 '12 at 2:51
  • @waffles: If you hadn't mentioned it, I wouldn't have even thought of checking robots.txt. – user102937 May 16 '12 at 2:53
  • @Dennis: I know.. Actually I'll add that.. Issue is, we're complying with the motivation behind robots.txt, but not the rules behind it. – Manishearth May 16 '12 at 2:54
  • @waffles: Single link sounds nice. But how many Facebook links are there? – Dennis May 16 '12 at 2:56
  • 3
    @Dennis: why do we have FB links on SO?? – Manishearth May 16 '12 at 2:56
  • @Dennis: Relative to the total traffic on Facebook? It's infinitesimally small. – user102937 May 16 '12 at 2:58
  • @TimManishEarth: Nevermind. I misunderstood parts of the question. Unless there is a big amount of links on a single website that have to be verified, I agree that robots.txt shouldn't be consulted. – Dennis May 16 '12 at 2:58
  • @Dennis: And even robots need to spend time on FB, don't they? ;-) – Manishearth May 16 '12 at 3:00
  • 1
    @TimManishEarth: I don't know. I never was a robot nor have I ever spent time on Facebook (sorted by likelihood to happen, decreasing). – Dennis May 16 '12 at 3:02
  • @Dennis: Same here, actually :P – Manishearth May 16 '12 at 3:06
  • 2
    @waffles I don't see anything wrong with just hitting a link to verify it exists, and I see no reason to consult robots.txt prior to doing it, even if you have a large amount of links to check on a single domain. Just stagger them appropriately and be done with it. Consulting robots.txt is in no way useful, as it will never give you explicit permission to do it. The alternate is removing any questionable link on SO that can't be verified, which I'm sure the domain owner would not want. – Tim Post May 16 '12 at 3:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .