0

I want to try doing a data analysis project based on a large amount of SO question data. Specifically, I want to take a year worth of question data (from 2018/01/01 to 2019/01/01), and look for certain trends.

I looked at the question API and realized I could do something like this which gives me all the data points I need.

The problem is, I don't think I'm allowed to pull this much data this way in any reasonable amount of time. Looking over the throttling page, I'm limited to 10,000 requests a day, which is problematic when combined with the fact that page sizes max out at 100.

If I go to the 20,000th page, that only takes me back to like September. That means I'd have to max out my daily requests for 3 days to gather the data.

Is there a better way of going about this, and/or is gathering this much data considered abusive?

3

The Stack Exchange Data Dump allows you to download all the data locally so you can query it as much as you like.

You can read more about that here: Database schema documentation for the public data dump and SEDE

  • I missed that data could be exported from SEDEs. That'll work actually. Thanks. – Carcigenicate Feb 25 at 2:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .