2

I am using the Stack Exchange API to get Stack Overflow data, but I am getting "too many request from this ip" after making only five requests. I got JSON content like this:

{u'error_id': 502, u'error_message': u'too many requests from this IP, more requests available in 68492 seconds', u'error_name': u'throttle_violation'}

I made the following requests to Stack Exchange without an access token:

  1. "https://api.stackexchange.com/2.2/posts?order=desc&sort=activity&site=stackoverflow" ---> 2 times called this API

  2. "https://api.stackexchange.com/2.2/search?order=desc&min=1416268800&sort=activity&tagged=python&site=stackoverflow" --> 3 times called this API and on the 4th call I got an exception

  • Shared IP? Memory failure? A rogue loop in your code? – John Dvorak Nov 18 '14 at 11:23
  • No loop in my code I am making one request at a time and IP is not shared – Yogesh Nov 18 '14 at 11:27
  • Also, you shouldn't be requesting the same URL twice in less than a minute. Also, have you got a backoff header? – John Dvorak Nov 18 '14 at 11:28
  • but requesting twice in less than a minute is not mentioned in rate limiting api.stackexchange.com/docs/throttle right ? – Yogesh Nov 18 '14 at 11:30
  • "While not strictly a throttle, the Stack Exchange API employs heavy caching and as such no application should make semantically identical requests more than once a minute. This is generally a waste of bandwidth as, more often than not, the exact same result will be returned." - last paragraph – John Dvorak Nov 18 '14 at 12:39
  • 1
    Pure speculation: they may have tightened up the rules after recent DDOS attack on SO. I suggest registering on Stack Apps to get a key for your application. You don't need to make anything about your code public there: the software automatically generates a private page for your app and gives you a key parameter. Passing &key=... with your requests should solve the problem. – user259867 Nov 18 '14 at 13:46
  • Related: The API is being very standoffish – Tim Stone Nov 18 '14 at 20:13
11

It looks like you selectively read the throttle guidance

A dynamic throttle is also in place on a per-method level. If an application receives a response with the backoff field set, it must wait that many seconds before hitting the same method again. For the purposes of throttling, all /me routes are considered to be identical to their /users/{ids} equivalent. Note that backoff is set based on a combination of factors, and may not be consistently returned for the same arguments to the same method. Additionally, all methods (even seemingly trivial ones) may return backoff.

If I take your first URL on my first response I immediately get a backoff for 10 seconds:

{ ... ,"has_more":true,
       "backoff":10,
       "quota_max":300,
       "quota_remaining":299}

So don't go down the route to guesstimate how many calls per minute you're allowed. Instead always honor the backoff.

  • 2
    This is the correct answer. If I recall correctly using a key also gives you less strict throttling, so register for one in case you can't actually meet this backoff period. – Kasra Rahjerdi Nov 18 '14 at 21:34
  • Unfortunately the API doesn't always return the backoff parameter and sometime just block without any explanation. – laurent Nov 20 '16 at 14:14
  • @this.lau_ yes, that is possible if you happen to be on an IP address that has many connections. You'll be blocked by the HA-proxy in that case with a 503 http error. You can recover from those by backing off for 60 seconds. – rene Nov 20 '16 at 15:37

You must log in to answer this question.

protected by Community May 19 at 1:11

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged .