6

My crawler follows Stack's robots.txt directives (it is a general web crawler so hitting many domains). Yet, when I am done crawling I frequently get blocked from accessing your website from a browser for a period of time. I would rather not be blocked and am happy to follow a crawl-delay, but none exists in your robots.txt file (see https://stackoverflow.com/robots.txt)

Your "official" guide to rate limiting does not answer how many requests we are allowed: The Complete Rate-Limiting Guide

Sooo, can you please put in a crawl-delay for your robots.txt file? I understand you may want to be a bit opaque about how many requests people are allowed but for the robots.txt file you could be conservative: e.g. if the "real" limit is 1 request every 5 seconds you could put a crawl-delay of 10 seconds, or 7 seconds, etc.

Improve this question
8
  • 1
    User name needs moar camelCase.
    – user102937
    Apr 28, 2017 at 0:19
  • 1
    In other news, couldn't you just figure this out by trial and error, or by using one of the other limits as a proxy? For example, the rate limiting guide says that you can do 30 searches in a 60 second window, which works out to one page request every 2 seconds, which seems like a reasonable throttle to me.
    – user102937
    Apr 28, 2017 at 0:23
  • Also, if the amount of data you're scraping is reasonably limited (which it should be), and you're letting it run overnight (which you should be), you can figure this out as a simple math problem.
    – user102937
    Apr 28, 2017 at 0:25
  • 5
    So, you want me to hit ALL of their hosts multiple extra times just to try to figure out their delay? That doesn't seem practical. Since the crawler is a general crawler it would be very inefficient to set a crawl delay for each host like this. Yes, I saw the API is different and could figure it out by trial and error..I would assume each of stack's hosts has the same crawl delay but maybe they're not and if they change then I have to do another trial and error and hit their mutliple hosts multiple more times to figure something that could be laid out in their robots.txt. No camelCase for you!
    – user_78361084
    Apr 28, 2017 at 0:36
  • 5
    @RobertHarvey I've tried to figure out the rate limits by "trial and error". It sucks. I also have wished it was documented on way more than one occasion. Trial and error isn't really feasible, and definitely not a reasonable substitute for documenting recommendations somewhere. Plus for general crawlers it's not a realistic approach to manually add exceptions for SE sites when you're already reading robots.txt.
    – Jason C
    Apr 28, 2017 at 0:49
  • @llamawithabowlcut That said I think the limits themselves may be somewhat complex and dynamic, and I think there's a few layers of it, too. I usually don't run into problems with one request per 5 seconds, unless I'm also doing something weird with authorization cookies, but that's a different story from your crawler.
    – Jason C
    Apr 28, 2017 at 0:50
  • Even if their limit is complex and dynamic they should just put a conservative number for the delay. Problem solved. Still no camelCase.
    – user_78361084
    Apr 28, 2017 at 2:09
  • @Robert are you assuming that all Camelids are alike?
    – Andrew Grimm
    Apr 28, 2017 at 10:17

2 Answers 2

Reset to default
8

We provide regular data-dumps for access to the vast amount of historical data and an API that provides up-to-date information. The latter has a well-documented set of rate limits. Generally speaking, you should prefer either or both of those to scraping the site; we build them with the intention of enabling machine access.

If you must scrape the site, you're on your own; robots.txt provides some basic information on what you can hit, but makes no promises as to how often you can hit it because unless you're a human using the site or a search engine sending lots humans to useful information on the site you're almost certainly chewing up a lot more than your fair share of resources by scraping.

Implement adaptive throttling; it's not that hard. Google is the 900lbs gorilla of scrapers and they manage a sensible rate; unless you're gonna start sending more traffic this way than Google, you've no excuse for being less considerate.

Improve this answer
3
  • 4
    So, I now have to figure out what "sensible rate" means for each and every website out there? Great, I'll get working on that /s
    – user_78361084
    May 4, 2017 at 1:36
  • You are making a lot of assumptions about why we choose to crawl the website instead of using the API. Please provide an answer without assuming information that is outside the scope of the question.
    – user_78361084
    May 4, 2017 at 1:41
  • If the goal is to direct crawlers to the API then simply set a long crawl-delay in the robots.txt file.
    – user_78361084
    May 6, 2017 at 21:46
3

It seems pretty clear that they are unwilling to add a crawl-delay to their robots.txt file(s). The answer given by @Shog9 does not address the question and instead goes off on a tangent. His answer depends on a lot of assumptions about the crawler that do not apply in most cases. I can understand why they'd like to direct crawlers to the API. In that case, simply set a long crawl-delay. Instead, his answer imagines a scenario and assumes all bots fall into that category.

Also, he directs people to use Google's standard for crawling (which they don't explicitly publish). We are seemingly supposed to guess what Google does. I think not! The robots.txt standard IS the standard. While crawl-delay is an optional parameter for robots.txt, I am still looking for a reason to leave that item out. As stated, my crawler obeys the crawl-delay. Set it to 1s, 1,000,000 seconds. It doesn't matter to me as my goal isn't to scrape the website.

Until SE includes the parameter or gives a better explanation I will mark this as the accepted answer.

Improve this answer
9
  • 2
    Shog suggested using adaptive throttling. This isn't some sort of proprietary Google algorithm. It's what you do in situations where throttle rates are not known ahead of time, under the assumption that throttle bans are not permanent. If you implement this it will do you good in the long run, too, as it will allow you to support other sites that may have missing / incorrect crawl delays as well. Of course a specific delay is way better, and I don't defend SE's decision not to add it, but Shog's post was pretty good advice even if you remove the parts that don't directly apply.
    – Jason C
    May 6, 2017 at 22:03
  • No, he's suggesting that Google sets the standard. We have a standard called robots.txt.
    – user_78361084
    May 6, 2017 at 23:31
  • Shog still doesn't answer the question. My answer does.
    – user_78361084
    May 6, 2017 at 23:34
  • 2
    I think you might have misinterpreted his Google reference slightly. What he meant was "Google does a ton of scraping, and if they can still manage to not get throttled by SE, you should be able to, too, given that you probably do a lot less scraping than Google does, or at least you likely don't do more." Google was more an arbitrary example of an extreme for comparison, rather than a statement of a standard.
    – Jason C
    May 6, 2017 at 23:48
  • My point is that by the time adaptive throttling kicks in you have ALREADY been throttled. To say that Google doesn't get throttled is FALSE. If they receive a 200 response it says "keep going". The only way to know to backoff is to send the 429 status and, by that time, you have already been throttled.
    – user_78361084
    May 7, 2017 at 0:15
  • Yeah, that's the algorithm. Basically you'd do this: Pick a starting reasonable rate (either something arbitrary, or from robots.txt if it's there, not the case for SE of course). If you get a 429, obey Retry-After if it's present and try again with a slower rate next time. For sites where you had to do this, keep track of the rate you're using in a persistent db somewhere. It's annoying, and it means you'll have a few minutes of initial throttling and guessing, but it only has to be done the first time you're scraping a new site. After you settle you're theoretically good forever.
    – Jason C
    May 7, 2017 at 0:20
  • (And, if you want, over time if you never receive more errors you can gradually increase your rate until you get throttled again. Then you can store that as the max allowable rate, so you'll back off dynamically then slowly increase back to that max rate and stay there, and now you'll have found the fastest rate you can scrape without throttling.) Btw check out some of the tips here -- some are relevant; also you have the option of asking for permission for a higher rate limit.
    – Jason C
    May 7, 2017 at 0:25
  • You have been very helpful. BTW, SE has Retry-After of 0. So there goes that.
    – user_78361084
    May 7, 2017 at 3:12
  • 1
    @JasonC's advice is good, apart from the "good forever" bit. I would recommend you always have logic in place to back off if a request is failing; this protects you against dynamic limits on our side and bugs in your own logic. I've seen a lot of naive scrapers that get concurrency wrong & end up doubling up on requests - and then accelerating from a short ban to a long one by assuming they can keep retrying at a constant rate through the ban period. Also, log everything - you'll never see the bug in your logic if you can't track the requests you're sending when it fails.
    – Shog9
    May 9, 2017 at 19:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .