7

I am using the Stack Exchange API to get Stack Overflow data, but I am getting "too many request from this ip" after making only five requests. I got JSON content like this:

{u'error_id': 502, u'error_message': u'too many requests from this IP, more requests available in 68492 seconds', u'error_name': u'throttle_violation'}

I made the following requests to Stack Exchange without an access token:

  1. "https://api.stackexchange.com/2.2/posts?order=desc&sort=activity&site=stackoverflow" ---> 2 times called this API

  2. "https://api.stackexchange.com/2.2/search?order=desc&min=1416268800&sort=activity&tagged=python&site=stackoverflow" --> 3 times called this API and on the 4th call I got an exception

Improve this question
7
  • Shared IP? Memory failure? A rogue loop in your code?
    – John Dvorak
    Commented Nov 18, 2014 at 11:23
  • No loop in my code I am making one request at a time and IP is not shared
    – Yogesh
    Commented Nov 18, 2014 at 11:27
  • Also, you shouldn't be requesting the same URL twice in less than a minute. Also, have you got a backoff header?
    – John Dvorak
    Commented Nov 18, 2014 at 11:28
  • 1
    but requesting twice in less than a minute is not mentioned in rate limiting api.stackexchange.com/docs/throttle right ?
    – Yogesh
    Commented Nov 18, 2014 at 11:30
  • "While not strictly a throttle, the Stack Exchange API employs heavy caching and as such no application should make semantically identical requests more than once a minute. This is generally a waste of bandwidth as, more often than not, the exact same result will be returned." - last paragraph
    – John Dvorak
    Commented Nov 18, 2014 at 12:39
  • 2
    Pure speculation: they may have tightened up the rules after recent DDOS attack on SO. I suggest registering on Stack Apps to get a key for your application. You don't need to make anything about your code public there: the software automatically generates a private page for your app and gives you a key parameter. Passing &key=... with your requests should solve the problem.
    – user259867
    Commented Nov 18, 2014 at 13:46
  • Related: The API is being very standoffish
    – Tim Stone
    Commented Nov 18, 2014 at 20:13

1 Answer 1

Reset to default
17

It looks like you selectively read the throttle guidance

A dynamic throttle is also in place on a per-method level. If an application receives a response with the backoff field set, it must wait that many seconds before hitting the same method again. For the purposes of throttling, all /me routes are considered to be identical to their /users/{ids} equivalent. Note that backoff is set based on a combination of factors, and may not be consistently returned for the same arguments to the same method. Additionally, all methods (even seemingly trivial ones) may return backoff.

If I take your first URL on my first response I immediately get a backoff for 10 seconds:

{ ... ,"has_more":true,
       "backoff":10,
       "quota_max":300,
       "quota_remaining":299}

So don't go down the route to guesstimate how many calls per minute you're allowed. Instead always honor the backoff.

Note that the backoff can be "undefined" (as in not present in the response) meaning you don't have to honor a backoff before your next call. But don't be abusive, keep it way under 30 requests per sec.

Make sure you have included the backoff field in your field selection:

backoff field marked enabled in the field selection dialog

Improve this answer
3
  • 3
    This is the correct answer. If I recall correctly using a key also gives you less strict throttling, so register for one in case you can't actually meet this backoff period.
    – Kasra Rahjerdi StaffMod
    Commented Nov 18, 2014 at 21:34
  • 3
    Unfortunately the API doesn't always return the backoff parameter and sometime just block without any explanation.
    – laurent
    Commented Nov 20, 2016 at 14:14
  • 2
    @this.lau_ yes, that is possible if you happen to be on an IP address that has many connections. You'll be blocked by the HA-proxy in that case with a 503 http error. You can recover from those by backing off for 60 seconds.
    – rene Mod
    Commented Nov 20, 2016 at 15:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .