SmokeDetector is seeing SE API errors 10,704 times more often than normal (2022-12-09)

Question

For a bit over the last hour, SmokeDetector (SD) has, once again, been seeing a flood of SE API errors. This is well outside what's "normal" to see for SmokeDetector. The only recent change (yesterday) to SmokeDetector in the area of the SE API was the switch to using the 2.4 version of the SE API in order to support scanning questions in the Staging Ground. In the past, this sort of dramatic increase in problems has been resolved by fixing issues on SE's servers.

You can see a list of the reported errors with this search in chat.

The errors seem to be on the /questions endpoint, which SD, currently, only uses for requests in its primary automatic post scanning process and only for Stack Overflow. All other SE sites, currently, use the /questions/{ids} endpoint. The reported error is "Violation of backoff parameter". SD can't have more than one SE API request in flight at a time and always obeys any backoff which it receives. In fact, it waits 2 seconds more between requests than the backoff parameter tells it to. Currently, what appears to be happening is that SD is getting one error, then waiting 12 + 2 seconds and trying again, a few times, until it works. Then, after a short time the issue happens again.

Note: Seeing such issues, very rarely, is something that's "normal" for the SE API. Historically, discounting the times there have been major sequences like this, SD reports such errors 47.5 per year since 2016, when SD started 24/7/365 operation. As of May, such issues increased by more of an order of magnitude. As of writing this, there are 58 such error reports in the last hour, a rate that is 10,704 times higher than normal.

For context, various similar SE API issues have been reported by myself and others from time to time. None of the following are duplicates, as this is a new, somewhat similar, occurrence:

• SE API errors have dramatically increased to 2,768 times the "normal" average error rate
• SE API quota_remaining property is periodically missing from the response Reporting a > 10x increase in errors from May 2022, and increasing (image from cocomac in chat), but marked as status-bydesign

Answer 1

We had a cached execution plan that was performing poorly, using a non-ideal index. We've (I say we - I mean Andy) evicted the cache so SQL Server can build a new plan and we're no longer seeing any exceptions come in.

API calls have uneven cardinality, and allow just about any set of inputs to be run, so we have a high likelihood of caching a plan that works well for one set of parameters, and poorly for another, so we're going to have a follow-up to add OPTION (OPTIMIZE FOR UNKNOWN) to address potential bad caches in the future.

This is a common issue with the API because of wildly varying input parameters.