I want to try doing a data analysis project based on a large amount of SO question data. Specifically, I want to take a year worth of question data (from 2018/01/01 to 2019/01/01), and look for certain trends.

I looked at the question API and realized I could do something like this which gives me all the data points I need.

The problem is, I don't think I'm allowed to pull this much data this way in any reasonable amount of time. Looking over the throttling page, I'm limited to 10,000 requests a day, which is problematic when combined with the fact that page sizes max out at 100.

If I go to the 20,000th page, that only takes me back to like September. That means I'd have to max out my daily requests for 3 days to gather the data.

Is there a better way of going about this, and/or is gathering this much data considered abusive?


The Stack Exchange Data Dump allows you to download all the data locally so you can query it as much as you like.

You can read more about that here: Database schema documentation for the public data dump and SEDE

  • I missed that data could be exported from SEDEs. That'll work actually. Thanks. – Carcigenicate Feb 25 '19 at 2:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .