I am using the Stack Exchange API to get Stack Overflow data, but I am getting "too many request from this ip" after making only five requests. I got JSON content like this:

{u'error_id': 502, u'error_message': u'too many requests from this IP, more requests available in 68492 seconds', u'error_name': u'throttle_violation'}

I made the following requests to Stack Exchange without an access token:

  1. "https://api.stackexchange.com/2.2/posts?order=desc&sort=activity&site=stackoverflow" ---> 2 times called this API

  2. "https://api.stackexchange.com/2.2/search?order=desc&min=1416268800&sort=activity&tagged=python&site=stackoverflow" --> 3 times called this API and on the 4th call I got an exception

  • Shared IP? Memory failure? A rogue loop in your code? – John Dvorak Nov 18 '14 at 11:23
  • No loop in my code I am making one request at a time and IP is not shared – Yogesh Nov 18 '14 at 11:27
  • Also, you shouldn't be requesting the same URL twice in less than a minute. Also, have you got a backoff header? – John Dvorak Nov 18 '14 at 11:28
  • but requesting twice in less than a minute is not mentioned in rate limiting api.stackexchange.com/docs/throttle right ? – Yogesh Nov 18 '14 at 11:30
  • "While not strictly a throttle, the Stack Exchange API employs heavy caching and as such no application should make semantically identical requests more than once a minute. This is generally a waste of bandwidth as, more often than not, the exact same result will be returned." - last paragraph – John Dvorak Nov 18 '14 at 12:39
  • 1
    Pure speculation: they may have tightened up the rules after recent DDOS attack on SO. I suggest registering on Stack Apps to get a key for your application. You don't need to make anything about your code public there: the software automatically generates a private page for your app and gives you a key parameter. Passing &key=... with your requests should solve the problem. – user259867 Nov 18 '14 at 13:46
  • Related: The API is being very standoffish – Tim Stone Nov 18 '14 at 20:13

It looks like you selectively read the throttle guidance

A dynamic throttle is also in place on a per-method level. If an application receives a response with the backoff field set, it must wait that many seconds before hitting the same method again. For the purposes of throttling, all /me routes are considered to be identical to their /users/{ids} equivalent. Note that backoff is set based on a combination of factors, and may not be consistently returned for the same arguments to the same method. Additionally, all methods (even seemingly trivial ones) may return backoff.

If I take your first URL on my first response I immediately get a backoff for 10 seconds:

{ ... ,"has_more":true,
       "backoff":10,
       "quota_max":300,
       "quota_remaining":299}

So don't go down the route to guesstimate how many calls per minute you're allowed. Instead always honor the backoff.

  • 2
    This is the correct answer. If I recall correctly using a key also gives you less strict throttling, so register for one in case you can't actually meet this backoff period. – Kasra Rahjerdi Nov 18 '14 at 21:34
  • Unfortunately the API doesn't always return the backoff parameter and sometime just block without any explanation. – this.lau_ Nov 20 '16 at 14:14
  • @this.lau_ yes, that is possible if you happen to be on an IP address that has many connections. You'll be blocked by the HA-proxy in that case with a 503 http error. You can recover from those by backing off for 60 seconds. – rene Nov 20 '16 at 15:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .