GET requests to https://api.stackexchange.com/2.2 methods return HTTP 400 "Violation of backoff parameter" without previously warning to back off

Question

We have the following logic to make GET requests to https://api.stackexchange.com/2.2:

  private async get(
    apiMethod: string, // can be 'users', 'tags', 'info', or 'questions'
    url: string, // prefixed with https://api.stackexchange.com/2.2/
    searchParams: undefined | { [k: string]: string | number | undefined }
  ): Promise<unknown> {
    await this.waitUntilRateLimiterAllowsCall(apiMethod);

    let response;
    try {
      response = await this.fetch(url, {
        method: 'GET',
        headers: this.headers,
        searchParams,
      });
    } catch (e) {
      Log.warn(e, {
        meta: 'Stack Overflow GET errored',
        apiMethod,
      });
      throw e;
    }

    if (response.backoff != null) {
      // If backoff is provided, we need to wait this many seconds before
      // calling the same endpoint again
      Log.info('Backing off Stack Overflow method', {
        apiMethod,
        backoffSeconds: response.backoff,
      });

      // backoff for an extra 2 seconds to give Stack Overflow some room to breathe, backing off for response.backoff or response.backoff + 1 kept getting us throttled
      await this.rateLimiterBackoffStackOverflow(
        apiMethod,
        response.backoff + 2
      );
    }

    return response;
  }

The implementation of waitUntilRateLimiterAllowsCall looks like this:

  async waitUntilRateLimiterAllowsCall(
    apiMethod: string | undefined, // can be 'users', 'tags', 'info', or 'questions'
  ): Promise<void> {
    const callCost = 1;

    // Remove from our burst allowance first, which is 30 calls per 5 seconds to stay safely away from the Stack Overflow abusive burst limit of 30 requests in 1 second
    await this.removeTokens('stackOverflowBurst', callCost, /* limitKey */ undefined);

    // Once we've passed our burst allowance we can eat away at our "main call" limit of 10k calls per day
    await this.removeTokens('stackOverflow', callCost, /* limitKey */ undefined);

    // Check if apiMethod has been blocked due to a backoff result
    await this.removeTokens('stackOverflowMethod', callCost, apiMethod);
  }

this.fetch is a call to a tiny wrapper around the got library https://www.npmjs.com/package/got

export const fetch = async (
  url: string,
  args: HttpFetcherArgs
): Promise<Record<string, unknown>> => {
  const req = got(url, args);

  return (await req.json()) as Record<string, unknown>;
};

These GET requests can be made by up to five processes running in parallel, and we manage the rate-limiting via Redis with node-rate-limiter-flexible.

We have evidence of both the Log.warn and Log.info message being logged + queryable in Cloudwatch:

In the first screenshot here we show that we receive + log the Backing off Stack Overflow method events.

In the second screenshot here, we see a flurry of Stack Overflow GET errored warns with no Backing off Stack Overflow method logs in the previous ~50 minutes, which suggests that api.stackexchange.com is throttling us due to "Violation of backoff parameter" without first warning us to back off.

Aside, we tried to be diligent and check that we're not violating other throttling parameters. We're confident we're below the 10k daily request rate limit:

1. Does the code snippet look right to you?

2. Are you aware of a possible bug that could be throttling without warning? Hypothetical: an ordering bug where we make a request that would have resulted in Stack Exchange replying with a backoff field, and we issue a second request in parallel => both requests 400 with this "Violation of backoff parameter" error without us ever receiving the backoff field.

3. Is there correct behaviour that we can build here without being told to backoff? While searching for previous answers I found Include backoff value (or remaining time) in API backoff violation response, would you reconsider that request and include the backoff in the throttle response so we know how long to back off for when we get throttled?