30

I'm writing a small script that makes a large amount of requests in order to obtain data that can't be acquired via the API (for example, getting a user's network profile URL, scraping the network reputation graph, scraping moderator election pages, etc.).

However, I am aware that it is possible to run into a brick wall when making lots of requests in quick succession (I also have personal experience accidentally hitting this limit by inadvertently holding F5 down...).

I'd like to leave this running over a long period of time (there's a lot of data that I'm trying to scrape), but I'd also prefer that it not take weeks to complete. So I'd like for it to go as fast as possible.

Is there any general guideline for this? I'd like to get a response something like

If you wait 1337 seconds between every request, you should be fine.

(although with a much lower number obviously).

Improve this question
5
  • 2
    Why don't you make this dynamic? You get an html page from the proxy if I'm not mistaken telling you to back-off. Then you know your last rate was too high. I recall that you have to wait a minute or two when you're auto-ip-banned.
    – rene Mod
    Commented Feb 22, 2015 at 12:15
  • 6
    The trial and error method of rene seems an effective way to get to know this, but it would be very nice indeed if they just make it public. I don't think it is much of a secret, most services have a guideline on this.
    – Patrick Hofman
    Commented Feb 22, 2015 at 14:19
  • 1
    Quick note: I ended up going ahead with a sleep 1 between every request, and never hit any errors. The question still stands, though.
    – Doorknob
    Commented Feb 22, 2015 at 18:42
  • 1
    If I hit a 503, waiting for a minute is enough to be able to get requests served again. You simply want to know what this line looks like in the haproxy: stick-table type ip size 200k expire 3m store conn_rate(100s),bytes_out_rate(60s)
    – rene Mod
    Commented Feb 22, 2015 at 19:24
  • 1
    I can drop to just above 509 ms between calls but after 80 calls I get a 503 and have to back-off for a minute. I managed one run of 177 calls before a 503 with a wait time dropping from 920ms to 745ms between calls.
    – rene Mod
    Commented Feb 22, 2015 at 21:12

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .