I'm writing a small script that makes a large amount of requests in order to obtain data that can't be acquired via the API (for example, getting a user's network profile URL, scraping the network reputation graph, scraping moderator election pages, etc.).
However, I am aware that it is possible to run into a brick wall when making lots of requests in quick succession (I also have personal experience accidentally hitting this limit by inadvertently holding F5 down...).
I'd like to leave this running over a long period of time (there's a lot of data that I'm trying to scrape), but I'd also prefer that it not take weeks to complete. So I'd like for it to go as fast as possible.
Is there any general guideline for this? I'd like to get a response something like
If you wait 1337 seconds between every request, you should be fine.
(although with a much lower number obviously).