6

We have the following logic to make GET requests to https://api.stackexchange.com/2.2 :

  private async get(
    apiMethod: string, // can be 'users', 'tags', 'info', or 'questions'
    url: string, // prefixed with https://api.stackexchange.com/2.2/
    searchParams: undefined | { [k: string]: string | number | undefined }
  ): Promise<unknown> {
    await this.waitUntilRateLimiterAllowsCall(apiMethod);

    let response;
    try {
      response = await this.fetch(url, {
        method: 'GET',
        headers: this.headers,
        searchParams,
      });
    } catch (e) {
      Log.warn(e, {
        meta: 'Stackoverflow GET errored',
        apiMethod,
      });
      throw e;
    }

    if (response.backoff != null) {
      // If backoff is provided, we need to wait this many seconds before
      // calling the same endpoint again
      Log.info('Backing off stackoverflow method', {
        apiMethod,
        backoffSeconds: response.backoff,
      });

      // backoff for an extra 2 seconds to give stackoverflow some room to breathe, backing off for response.backoff or response.backoff + 1 kept getting us throttled
      await this.rateLimiterBackoffStackOverflow(
        apiMethod,
        response.backoff + 2
      );
    }

    return response;
  }

The implementation of waitUntilRateLimiterAllowsCall looks like this:

  async waitUntilRateLimiterAllowsCall(
    apiMethod: string | undefined, // can be 'users', 'tags', 'info', or 'questions'
  ): Promise<void> {
    const callCost = 1;

    // Remove from our burst allowance first, which is 30 calls per 5 seconds to stay safely away from the stackoverflow abusive burst limit of 30 requests in 1 second
    await this.removeTokens('stackOverflowBurst', callCost, /* limitKey */ undefined);

    // Once we've passed our burst allowance we can eat away at our "main call" limit of 10k calls per day
    await this.removeTokens('stackOverflow', callCost, /* limitKey */ undefined);

    // Check if apiMethod has been blocked due to a backoff result
    await this.removeTokens('stackOverflowMethod', callCost, apiMethod);
  }

this.fetch is a call to a tiny wrapper around the got library https://www.npmjs.com/package/got

export const fetch = async (
  url: string,
  args: HttpFetcherArgs
): Promise<Record<string, unknown>> => {
  const req = got(url, args);

  return (await req.json()) as Record<string, unknown>;
};

These GET requests can be made by up to 5 processes running in parallel, and we manage the rate-limiting via redis with https://github.com/animir/node-rate-limiter-flexible

We have evidence of both the Log.warn and Log.info message being logged + queryable in Cloudwatch:

In the first screenshot here we show that we receive + log the Backing off stackoverflow method events.

Evidence of Log.info on backoff

In the second screenshot here, we see a flurry of stackoverflow GET errored warns with no Backing off stackoverflow method logs in the previous ~50 minutes, which suggests that api.stackexchange.com is throttling us due to "Violation of backoff parameter" without first warning us to back off.

Evidence of Log.warn during throttle, with no matching Log.info for backoff

Aside, we tried to be diligent and check that we're not violating other throttling parameters. We're confident we're below the 10k daily request rate limit:

Fewer than 6k requests in the 24 hours leading up to this period of throttled requests

  1. Does the code snippet look right to you?

  2. Are you aware of a possible bug that could be throttling without warning? Hypothetical: an ordering bug where we make a request that would have resulted in stackexchange replying with a backoff field, and we issue a second request in parallel => both requests 400 with this "Violation of backoff parameter" error without us ever receiving the backoff field.

  3. Is there correct behaviour that we can build here without being told to backoff? While searching for previous answers I found Include backoff value (or remaining time) in API backoff violation response, would you reconsider that request and include the backoff in the throttle response so we know how long to back off for when we get throttled?

19
  • 1
    Looking at the logs I wonder if you're not violating the request limit: Every application is subject to an IP based concurrent request throttle. If a single IP is making more than 30 requests a second, new requests will be dropped. The exact ban period is subject to change, but will be on the order of 30 seconds to a few minutes typically. Note that exactly what response an application gets (in terms of HTTP code, text, and so on) is undefined when subject to this ban; we consider > 30 request/sec per IP to be very abusive and thus cut the requests off very harshly.
    – rene
    Apr 2 at 14:12
  • 1
    If you hit that limit, you'll not get a backoff because the application never served your call so it didn't had to do any bookkeeping for a backoff now or in the future. api.stackexchange.com/docs/throttle
    – rene
    Apr 2 at 14:13
  • 2
    Are you sure that this application is the only thing making requests from that public IP address? Unless you have this application as the only thing from that IP address, then the backoff could easily have been sent in response to some request by a different application/browser/browser tab/userscript/whatever.
    – Makyen
    Apr 2 at 14:47
  • 1
    On an off-note, last time I checked, backoff is either present or not and is never equal to null. Am I missing something? Unless this.fetch is a custom implementation, of course Apr 2 at 18:17
  • 1
    I also do not see anything in the snippet that sets the result.backoff property - can this be the typo why you are getting rate-limited? Apr 2 at 18:22
  • 2
    @OlegValteriswithUkraine I assumed it was JavaScript, with response set to await this.fetch(), which presumably results in response set to the decoded JSON Object. So, response.backoff will be undefined most of the time, which would make response.backoff != null false, as it uses != rather than !==.
    – Makyen
    Apr 2 at 19:48
  • @rene > I wonder if you're not violating the request limit If we are, I'd expect to see a different error message from "Violation of backoff parameter" which implies specifically that we were told to backoff, not that we're over the request limit. Still, our rate limiter should prevent this. I've updated my original post to show how waitUntilRateLimiterAllowsCall is implemented. Apr 2 at 20:07
  • @Makyen > Are you sure that this application is the only thing making requests from that public IP address? I grep'd our codebase, I can't see anything speaking to api.stackexchange.com except this method. Apr 2 at 20:07
  • @OlegValteriswithUkraine, Makyen is correct about the absence of backoff resulting in the response.backoff access returning undefined and failing the != null check. Also, the first screenshot provides evidence that when backoff is present, we log the message that we expect to see. Apr 2 at 20:08
  • 2
    The short answer is that, yes, this can happen, particularly when you have more than one request in flight at a time (e.g. one may or may not get a backoff and the others may just get an error). You won't have an accurate view of the conditions under which it's happening without logging every request (both upon start and end). Once it happens, you should stop making requests for a period of time (10, 15 or 30 seconds is usually sufficient, but it's sometimes minutes). I'd note that SE doesn't consider this behavior to be a bug. IIRC, there are other questions asking about similar behavior.
    – Makyen
    Apr 2 at 20:28
  • 1
    @RafBlecher the part about !== null was just an off-note in case it might help - so I assume the response object is set to the result of await fetch_response.json()? I am more worried about the result.backoff - can you show where this property is set? Apr 2 at 20:38
  • 1
    If this is not considered a bug, can we revisit putting the backoff field in the 502 "Violation of backoff parameter" response so we don't have to guess how long we have to back off for? "30 seconds is usually sufficient, but it's sometimes minutes" is a little vague to build on top of. For now, I'll put in a 30 second rate limiter block when we get this 502 Apr 2 at 20:46
  • 1
    @RafBlecher I see, O.K., that part is clear to me now. Can you clarify what exactly is result.backoff and how is it set? I do not see neither where it is declared nor where the backoff property is set on it. Is it a typo? If so, should it not be response.backoff + 2? Apr 2 at 20:50
  • 2
    Ohhh I see, I'm sorry, I modified this code snippet to remove a call to some JSON validation that we're doing hence the response/result name mismatch -- I'll update the code example, result.backoff should be response.backoff Apr 2 at 20:53
  • 2
    It is worth mentioning that the API end point sits behind the HA-proxy as well. That error response might come directly from HA-proxy throttling settings and for obvious reasons the HA-proxy has no knowledge about the backoff parameter or how long it expects you have to wait as those throttles are dynamic.
    – rene
    Apr 2 at 21:16

0

You must log in to answer this question.

Browse other questions tagged .