Over the last several days while running SmokeDetector, we've periodically seen multiple responses from the SE API with the quota_remaining missing, with the most recent group a few hours ago. The quota_remaining property being missing from the SE API response happens from time to time, so SmokeDetector normally just reports it into chat, auto-recovers, and we typically ignore it. However, it's reported into chat so people can watch for patterns or major outages. At this point, it's clear that these errors are happening much more commonly than "normal" and that there's a pattern when it's happening during the day.
If you do a search for quota_remaining in the Charcoal HQ room, you'll see that SmokeDetector has intermittently reported such errors over its existence. However, recently, a pattern has developed. Starting in mid to late 2022 May, there are a much larger number of such reports and the reports are very closely clustered around two times of day 18:18 UTC (starting 2022-05-16) and 02:31 UTC (starting 2022-05-26), with some other anomalous ones. There are also indications that the issues may predate that, as there's a set at 2022-05-14 at 03:56 UTC and sets at similar times going back further and even into earlier years.
From SmokeDetector's point of view, this is really just something that's causing a couple/few minutes of delay in fetching question data one or two times a day. SmokeDetector automatically recovers from this happening, so it's really not a big deal from our point of view. However, that it's happening and gotten much more common indicates there's a problem with the SE API which has recently developed or gotten worse. This question is just to bring the issue to SE's attention and let you choose to investigate, or not.
Recent message from SD reporting the issue (from search link above)
| User | Day | UTC time | Message |
|---|---|---|---|
| SmokeDetector | 2022-05-31 | 18:19 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:19:05 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-31 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:51 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:57 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:12 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:26 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:40 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:43 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-30 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:58 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:20 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:01 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:20 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:15 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:04 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:18 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:33 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:47 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:03 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:17 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:37 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:48 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-28 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:50 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-26 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:07 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-26 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:21 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-26 | 02:32 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:35 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-26 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:39 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-26 | 02:31 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:53 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-24 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:13 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-23 | 18:19 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:19:06 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-23 | 02:31 | The quota_remaining property was not in the API response. Error: no method found with this name at 02:31:00 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-21 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:27 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-20 | 18:20 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-20 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-20 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:24 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-20 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:40 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-20 | 18:19 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:54 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-18 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:45 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-16 | 18:18 | The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions |
| SmokeDetector | 2022-05-14 | 03:57 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:57:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449832 |
| SmokeDetector | 2022-05-14 | 03:56 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:06 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449899 |
| SmokeDetector | 2022-05-14 | 03:56 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:27 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/702458 |
| SmokeDetector | 2022-05-14 | 03:56 | The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:42 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/80215 |
SE API Servers are out of sync
Semi-related: Another issue which we've seen with the SE API recently is that a couple of times in the last few days SmokeDetector's SE API quota has "rolled over" more than once per day. In the past, this tended to indicate that at least one of the SE API servers has gotten substantially out of sync with the rest. You can do a search for "API quota rolled over" in Charcoal HQ and see that the quota has "rolled over" more than once in a 24 hour period a couple of times recently, on 2022-05-28 and 2022-05-31. I mention it here, because it's something which might be easy to look at at the same time. If it happens another time or two, I'll probably create a separate bug report. [Note: the SE API servers being slightly out of sync is common, so SmokeDetector filters smaller differences out. It only reports a rollover more than once per day when the SE API servers get out of sync substantially by more than 5,000 of SmokeDetector's requests.]
