My crawler follows Stack's robots.txt directives (it is a general web crawler so hitting many domains). Yet, when I am done crawling I frequently get blocked from accessing your website from a browser for a period of time. I would rather not be blocked and am happy to follow a crawl-delay, but none exists in your robots.txt file (see https://stackoverflow.com/robots.txt)

Your "official" guide to rate limiting does not answer how many requests we are allowed: The Complete Rate-Limiting Guide

Sooo, can you please put in a crawl-delay for your robots.txt file? I understand you may want to be a bit opaque about how many requests people are allowed but for the robots.txt file you could be conservative: e.g. if the "real" limit is 1 request every 5 seconds you could put a crawl-delay of 10 seconds, or 7 seconds, etc.

  • 1
    User name needs moar camelCase. – Robert Harvey Apr 28 '17 at 0:19
  • 1
    In other news, couldn't you just figure this out by trial and error, or by using one of the other limits as a proxy? For example, the rate limiting guide says that you can do 30 searches in a 60 second window, which works out to one page request every 2 seconds, which seems like a reasonable throttle to me. – Robert Harvey Apr 28 '17 at 0:23
  • Also, if the amount of data you're scraping is reasonably limited (which it should be), and you're letting it run overnight (which you should be), you can figure this out as a simple math problem. – Robert Harvey Apr 28 '17 at 0:25
  • 5
    So, you want me to hit ALL of their hosts multiple extra times just to try to figure out their delay? That doesn't seem practical. Since the crawler is a general crawler it would be very inefficient to set a crawl delay for each host like this. Yes, I saw the API is different and could figure it out by trial and error..I would assume each of stack's hosts has the same crawl delay but maybe they're not and if they change then I have to do another trial and error and hit their mutliple hosts multiple more times to figure something that could be laid out in their robots.txt. No camelCase for you! – Brent Apr 28 '17 at 0:36
  • 5
    @RobertHarvey I've tried to figure out the rate limits by "trial and error". It sucks. I also have wished it was documented on way more than one occasion. Trial and error isn't really feasible, and definitely not a reasonable substitute for documenting recommendations somewhere. Plus for general crawlers it's not a realistic approach to manually add exceptions for SE sites when you're already reading robots.txt. – Jason C Apr 28 '17 at 0:49
  • @llamawithabowlcut That said I think the limits themselves may be somewhat complex and dynamic, and I think there's a few layers of it, too. I usually don't run into problems with one request per 5 seconds, unless I'm also doing something weird with authorization cookies, but that's a different story from your crawler. – Jason C Apr 28 '17 at 0:50
  • Even if their limit is complex and dynamic they should just put a conservative number for the delay. Problem solved. Still no camelCase. – Brent Apr 28 '17 at 2:09
  • @Robert are you assuming that all Camelids are alike? – Andrew Grimm Apr 28 '17 at 10:17

We provide regular data-dumps for access to the vast amount of historical data and an API that provides up-to-date information. The latter has a well-documented set of rate limits. Generally speaking, you should prefer either or both of those to scraping the site; we build them with the intention of enabling machine access.

If you must scrape the site, you're on your own; robots.txt provides some basic information on what you can hit, but makes no promises as to how often you can hit it because unless you're a human using the site or a search engine sending lots humans to useful information on the site you're almost certainly chewing up a lot more than your fair share of resources by scraping.

Implement adaptive throttling; it's not that hard. Google is the 900lbs gorilla of scrapers and they manage a sensible rate; unless you're gonna start sending more traffic this way than Google, you've no excuse for being less considerate.

  • 4
    So, I now have to figure out what "sensible rate" means for each and every website out there? Great, I'll get working on that /s – Brent May 4 '17 at 1:36
  • You are making a lot of assumptions about why we choose to crawl the website instead of using the API. Please provide an answer without assuming information that is outside the scope of the question. – Brent May 4 '17 at 1:41
  • If the goal is to direct crawlers to the API then simply set a long crawl-delay in the robots.txt file. – Brent May 6 '17 at 21:46

It seems pretty clear that they are unwilling to add a crawl-delay to their robots.txt file(s). The answer given by @Shog9 does not address the question and instead goes off on a tangent. His answer depends on a lot of assumptions about the crawler that do not apply in most cases. I can understand why they'd like to direct crawlers to the API. In that case, simply set a long crawl-delay. Instead, his answer imagines a scenario and assumes all bots fall into that category.

Also, he directs people to use Google's standard for crawling (which they don't explicitly publish). We are seemingly supposed to guess what Google does. I think not! The robots.txt standard IS the standard. While crawl-delay is an optional parameter for robots.txt, I am still looking for a reason to leave that item out. As stated, my crawler obeys the crawl-delay. Set it to 1s, 1,000,000 seconds. It doesn't matter to me as my goal isn't to scrape the website.

Until SE includes the parameter or gives a better explanation I will mark this as the accepted answer.

  • 1
    Shog suggested using adaptive throttling. This isn't some sort of proprietary Google algorithm. It's what you do in situations where throttle rates are not known ahead of time, under the assumption that throttle bans are not permanent. If you implement this it will do you good in the long run, too, as it will allow you to support other sites that may have missing / incorrect crawl delays as well. Of course a specific delay is way better, and I don't defend SE's decision not to add it, but Shog's post was pretty good advice even if you remove the parts that don't directly apply. – Jason C May 6 '17 at 22:03
  • No, he's suggesting that Google sets the standard. We have a standard called robots.txt. – Brent May 6 '17 at 23:31
  • Shog still doesn't answer the question. My answer does. – Brent May 6 '17 at 23:34
  • 1
    I think you might have misinterpreted his Google reference slightly. What he meant was "Google does a ton of scraping, and if they can still manage to not get throttled by SE, you should be able to, too, given that you probably do a lot less scraping than Google does, or at least you likely don't do more." Google was more an arbitrary example of an extreme for comparison, rather than a statement of a standard. – Jason C May 6 '17 at 23:48
  • My point is that by the time adaptive throttling kicks in you have ALREADY been throttled. To say that Google doesn't get throttled is FALSE. If they receive a 200 response it says "keep going". The only way to know to backoff is to send the 429 status and, by that time, you have already been throttled. – Brent May 7 '17 at 0:15
  • Yeah, that's the algorithm. Basically you'd do this: Pick a starting reasonable rate (either something arbitrary, or from robots.txt if it's there, not the case for SE of course). If you get a 429, obey Retry-After if it's present and try again with a slower rate next time. For sites where you had to do this, keep track of the rate you're using in a persistent db somewhere. It's annoying, and it means you'll have a few minutes of initial throttling and guessing, but it only has to be done the first time you're scraping a new site. After you settle you're theoretically good forever. – Jason C May 7 '17 at 0:20
  • (And, if you want, over time if you never receive more errors you can gradually increase your rate until you get throttled again. Then you can store that as the max allowable rate, so you'll back off dynamically then slowly increase back to that max rate and stay there, and now you'll have found the fastest rate you can scrape without throttling.) Btw check out some of the tips here -- some are relevant; also you have the option of asking for permission for a higher rate limit. – Jason C May 7 '17 at 0:25
  • You have been very helpful. BTW, SE has Retry-After of 0. So there goes that. – Brent May 7 '17 at 3:12
  • 1
    @JasonC's advice is good, apart from the "good forever" bit. I would recommend you always have logic in place to back off if a request is failing; this protects you against dynamic limits on our side and bugs in your own logic. I've seen a lot of naive scrapers that get concurrency wrong & end up doubling up on requests - and then accelerating from a short ban to a long one by assuming they can keep retrying at a constant rate through the ban period. Also, log everything - you'll never see the bug in your logic if you can't track the requests you're sending when it fails. – Shog9 May 9 '17 at 19:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .