1

I am using the StackAPI library for Python to access Stack Overflow questions. I need to collect all question IDs attached with a particular tag (for example, regex).

I used the search bar on Stack Overflow with the query [regex]. The Stack Overflow search engine indicated that there are 725,098 questions.

With StackAPI, I set the page_size to 100 and max_pages to 7251. Since I have an API key, my quota is 10000 per day. When I run the code, it throws an error code for throttle violation. I am not sure why.

Note: I am using a custom filter with back_off checked.

Enter image description here

Improve this question
7
  • Related: Stack Exchange API Optimal, Acceptable Request Frequency
    – Glorfindel Mod
    Oct 19, 2021 at 6:48
  • 4
    Did you obey the throttle? Note: even if you obey the throttle you need to handle all other types of responses from the SE API, as the responses which the SE API give are undefined under common circumstances (e.g. when your IP is temporarily blocked/throttled, which is related to but separate from a throttle response).
    – Makyen
    Oct 19, 2021 at 6:57
  • Since I had the remaining_quota over 9000, setting the max_pages to less than should technically obey the throttle. Unless I didn't account for anything else on top of the number of pages.
    – akalanka
    Oct 19, 2021 at 15:46
  • What do you do if you receive a value in backoff? How many calls do you make per 90 seconds?
    – rene
    Oct 19, 2021 at 17:07
  • Unfortunately, I didn't find any option in the library to deal with backoff? So, I went old school and used the requests module in Python and manually add a sleep time for 2 secs between each page request. I go to a new iteration to access the next page in a loop if has_more is true and remaining_quota is over 100
    – akalanka
    Oct 19, 2021 at 18:11
  • 1
    Unless you need it in realtime or up to date, I wonder if the datadumps or SEDE might be a better option...
    – Journeyman Geek Mod
    Oct 20, 2021 at 8:21
  • Using the data dump was my only option.
    – akalanka
    Oct 20, 2021 at 16:14

0

Reset to default