15

Many high profile sites have taken a "white-list" based approach to robots.txt

Examples are:

The result is that we are not allowed to automatically check that any external links to these sites are kosher.

The scale of the problem is fairly big, for example (for a sample of 5000 of the earliest links)

  • 3.5% are uncrawlable due to robots exclusion.
  • 5.8% are broken

How should we be dealing with these uncrawlable links?

20
  • What is the error code the excluded sites are returning?
    – user102937
    Commented May 16, 2012 at 2:38
  • 3
    @RobertHarvey the exclusion in robots bans us from checking the links ... there is no error.
    – waffles
    Commented May 16, 2012 at 2:40
  • How could that be? If you try to hit the link, there has to be some sort of response code.
    – user102937
    Commented May 16, 2012 at 2:41
  • 6
    @waffles: Change your user-agent to Googlebot. Simple. Commented May 16, 2012 at 2:41
  • @RobertHarvey we hit robots.txt first, parse it, it tells us we are not allowed to check ANY links on the site, so we stop.
    – waffles
    Commented May 16, 2012 at 2:41
  • @TimManishEarth A WINNING suggestion :)
    – waffles
    Commented May 16, 2012 at 2:42
  • 2
    @RobertHarvey: Basically, it's a standard that you're expected to follow. You can sidestep it, of course--I don't think the system forces you to parse robots.txt Commented May 16, 2012 at 2:42
  • 2
    @TimManishEarth: That's what I think you should do. It's not like we're crawling the entire site, or even a substantial part of it. Nor are we scraping for email addresses. Google will respect it, of course, because they must crawl the entire site, or at least that part that's publicly visible.
    – user102937
    Commented May 16, 2012 at 2:43
  • @RobertHarvey: (Source:Wikipedia) The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Commented May 16, 2012 at 2:43
  • 1
    @RobertHarvey: Yeah, me too. We don't exactly crawl the web page, we just point-and-shoot. Commented May 16, 2012 at 2:44
  • 3
    For context I am performing a throttled HEAD request to each link, respecting crawl delay and all.
    – waffles
    Commented May 16, 2012 at 2:45
  • 4
    You're not really crawling if you're just poking Abe Vigoda
    – random
    Commented May 16, 2012 at 2:51
  • 1
    So uh, maybe move this to webmasters as it's awfully not SO specific? Commented May 16, 2012 at 4:24
  • 1
    Don't cross post, just migrate. This discussion shouldn't be "hidden away" on meta; webmasters is the correct place for this. Commented May 16, 2012 at 4:42
  • 2
    It's correct here, but if you remove the implicit "we, Stack Exchange" it fits elsewhere and should go there as people looking for this information are unlikely to look on a meta site. Commented May 16, 2012 at 4:45

3 Answers 3

21

Here's how we feel about it at GitHub, from Aman Gupta:

alt text

Normal crawlers wreak havoc on our servers, because they can find content that isn't actually commonly seen (i.e. a persistent crawler can walk an entire Git history by following parent links), and it blows any attempt at caching by filling the cache with bogus entries.

In this case, because you're not crawling, you're not quite the same as a robot - the worries we have about GoogleBot don't apply here.

1
  • What about abnormal bot that don't respect the robots.txt, because you are somehow pushing robots to not respect your robots.txt
    – dvhh
    Commented May 16, 2012 at 4:02
14

I would not worry about it for now. What you're doing isn't very much different than a user interface doing an AJAX check to validate a link to show a little green check mark next to it in a UI. Due to the same origin policy, those have to go through a proxy and checking robots.txt in that scenario would seem extremely silly.

For the long term, you could set up a proper bot (just like Googlebot) and send the corresponding user agent, give it time to circulate and then check robots.txt for exclusions, but I really think doing so is going way above and beyond what a good netizen should be doing. You aren't crawling or indexing, you're just sending a HEAD request to see the status of any given link, and you're doing it nicely. There's no good reason to make two requests instead of one initially, or deal with the complexity of caching a bunch of robots.txt files to avoid it.

Finally, I think most site owners would want Stack Overflow to make every effort to verify inbound links to their site, as the only alternative would be to remove those that can't be verified wholesale.

12

Since robots.txt doesn't force us to follow it:

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

(source)

Basically, you want to be the good-guy cooperating crawler, right? Well, we don't crawl links. Crawlers spread out through every link they get and basically end up parsing nearly the whole site. That's bad when a site has tons of links that don't need to be crawled.

On the other hand, we take a link, point, and shoot. There's no "crawling" involved--nothing that can spiral into bigtime server load if we hit a page with millions of links. That's pretty much why we have robots.txt-- to prevent the robots from taking over the world overloading the site.

Though, if you notice, there's the "other web robots" in the Wikipedia snippet. Not complying with robots.txt will mean that we are complying with the intentions of robots.txt, but not with the actual rules. Which is OK, since overly broad rules are meant to be broken. And, it's not really a "rule" per se--it's a "listen if you know what's good for you" thingy:

Despite the use of the terms "allow" and "disallow", the protocol is purely advisory.

And, we're old enough to know what's good for us without having them tell us, right? ;-)

So just point-and-shoot. Don't even fetch robots.txt, it unecessarily increases server load a teensy bit. Though, as @Dennis suggested, you can consult robots.txt if you have a truckload of links on the same domain that you need to verify.

12
  • Well, with different highlighting, this becomes is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.
    – Dennis
    Commented May 16, 2012 at 2:51
  • 4
    one could argue that even hitting robots.txt for our context is a bad way to act ... especially if we only need to point and shoot at a "single" link
    – waffles
    Commented May 16, 2012 at 2:51
  • @waffles: If you hadn't mentioned it, I wouldn't have even thought of checking robots.txt.
    – user102937
    Commented May 16, 2012 at 2:53
  • @Dennis: I know.. Actually I'll add that.. Issue is, we're complying with the motivation behind robots.txt, but not the rules behind it. Commented May 16, 2012 at 2:54
  • @waffles: Single link sounds nice. But how many Facebook links are there?
    – Dennis
    Commented May 16, 2012 at 2:56
  • 3
    @Dennis: why do we have FB links on SO?? Commented May 16, 2012 at 2:56
  • @Dennis: Relative to the total traffic on Facebook? It's infinitesimally small.
    – user102937
    Commented May 16, 2012 at 2:58
  • @TimManishEarth: Nevermind. I misunderstood parts of the question. Unless there is a big amount of links on a single website that have to be verified, I agree that robots.txt shouldn't be consulted.
    – Dennis
    Commented May 16, 2012 at 2:58
  • @Dennis: And even robots need to spend time on FB, don't they? ;-) Commented May 16, 2012 at 3:00
  • 1
    @TimManishEarth: I don't know. I never was a robot nor have I ever spent time on Facebook (sorted by likelihood to happen, decreasing).
    – Dennis
    Commented May 16, 2012 at 3:02
  • @Dennis: Same here, actually :P Commented May 16, 2012 at 3:06
  • 2
    @waffles I don't see anything wrong with just hitting a link to verify it exists, and I see no reason to consult robots.txt prior to doing it, even if you have a large amount of links to check on a single domain. Just stagger them appropriately and be done with it. Consulting robots.txt is in no way useful, as it will never give you explicit permission to do it. The alternate is removing any questionable link on SO that can't be verified, which I'm sure the domain owner would not want.
    – user50049
    Commented May 16, 2012 at 3:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .