What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

I'm doing some research work on Stack Overflow and I need to analyze some HTML pages from this site. So I need to do some HTTP requests.

How many requests can I do during one minute or during one second without Stack Overflow stopping me?

After having made a request, how long must I wait for doing another?

Is there a document that specifies all rules (for example possible searches during one minute)?

share|improve this question
12  
Are you aware of the data dump containing all SO content (anonymized)? –  interjay Feb 24 '13 at 14:35
5  
Also there's an api - I'll guess that SE people would rather you use that than scrape the main sites. (I think the rate limits are documented for that too.) –  Mat Feb 24 '13 at 14:44

1 Answer 1

You'll never get an answer for this exact question. Everything is rate limited on the site, and that includes what serves up the final rendered page; the servers are behind nginx installs which will throttle overzealous clients.

Indicating what the throttle would let everyone know just where the line is without crossing over it, and allow people to place much greater load on the site.

That said, your best bet is to use either the data dump or use the API to get the data that you need.

There's nothing that's in the rendered HTML that you can't get from these data sources, and on top of that, you don't have to worry about scraping possibly changing HTML structures.

share|improve this answer
    
are the review queue lengths available via the API? I haven't seen them documented –  Jan Dvorak Feb 24 '13 at 17:31
    
@JanDvorak It doesn't look like it probably because it's too ephemeral. –  casperOne Feb 24 '13 at 17:36
    
Thank you for the answer, and thank you for providing non-scraping ways of getting at the data. –  vonbrand Feb 25 '13 at 5:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .