I want to try doing a data analysis project based on a large amount of SO question data. Specifically, I want to take a year worth of question data (from 2018/01/01 to 2019/01/01), and look for certain trends.
I looked at the question API and realized I could do something like this which gives me all the data points I need.
The problem is, I don't think I'm allowed to pull this much data this way in any reasonable amount of time. Looking over the throttling page, I'm limited to 10,000 requests a day, which is problematic when combined with the fact that page sizes max out at 100.
If I go to the 20,000th page, that only takes me back to like September. That means I'd have to max out my daily requests for 3 days to gather the data.
Is there a better way of going about this, and/or is gathering this much data considered abusive?