22

Over the last several days while running SmokeDetector, we've periodically seen multiple responses from the SE API with the quota_remaining missing, with the most recent group a few hours ago. The quota_remaining property being missing from the SE API response happens from time to time, so SmokeDetector normally just reports it into chat, auto-recovers, and we typically ignore it. However, it's reported into chat so people can watch for patterns or major outages. At this point, it's clear that these errors are happening much more commonly than "normal" and that there's a pattern when it's happening during the day.

If you do a search for quota_remaining in the Charcoal HQ room, you'll see that SmokeDetector has intermittently reported such errors over its existence. However, recently, a pattern has developed. Starting in mid to late 2022 May, there are a much larger number of such reports and the reports are very closely clustered around two times of day 18:18 UTC (starting 2022-05-16) and 02:31 UTC (starting 2022-05-26), with some other anomalous ones. There are also indications that the issues may predate that, as there's a set at 2022-05-14 at 03:56 UTC and sets at similar times going back further and even into earlier years.

From SmokeDetector's point of view, this is really just something that's causing a couple/few minutes of delay in fetching question data one or two times a day. SmokeDetector automatically recovers from this happening, so it's really not a big deal from our point of view. However, that it's happening and gotten much more common indicates there's a problem with the SE API which has recently developed or gotten worse. This question is just to bring the issue to SE's attention and let you choose to investigate, or not.

Recent message from SD reporting the issue (from search link above)

User Day UTC time Message
SmokeDetector 2022-05-31 18:19 The quota_remaining property was not in the API response. Error: no method found with this name at 18:19:05 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-31 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:51 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:57 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:12 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:26 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:40 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:43 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-30 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:58 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:20 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:01 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:20 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:15 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:04 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:18 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:33 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:47 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:03 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:17 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:37 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:48 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-28 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:50 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-26 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:07 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-26 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:21 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-26 02:32 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:32:35 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-26 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:39 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-26 02:31 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 02:31:53 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-24 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:13 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-23 18:19 The quota_remaining property was not in the API response. Error: no method found with this name at 18:19:06 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-23 02:31 The quota_remaining property was not in the API response. Error: no method found with this name at 02:31:00 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-21 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:27 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-20 18:20 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:20:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-20 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-20 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:24 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-20 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:40 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-20 18:19 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 18:19:54 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-18 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:45 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-16 18:18 The quota_remaining property was not in the API response. Error: no method found with this name at 18:18:29 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions
SmokeDetector 2022-05-14 03:57 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:57:09 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449832
SmokeDetector 2022-05-14 03:56 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:06 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/4449899
SmokeDetector 2022-05-14 03:56 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:27 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/702458
SmokeDetector 2022-05-14 03:56 The quota_remaining property was not in the API response. Error: Violation of backoff parameter at 03:56:42 UTC. Backing off on requests for the next 12 seconds. Previous URL: https://api.stackexchange.com/2.2/questions/80215
SE API Servers are out of sync

Semi-related: Another issue which we've seen with the SE API recently is that a couple of times in the last few days SmokeDetector's SE API quota has "rolled over" more than once per day. In the past, this tended to indicate that at least one of the SE API servers has gotten substantially out of sync with the rest. You can do a search for "API quota rolled over" in Charcoal HQ and see that the quota has "rolled over" more than once in a 24 hour period a couple of times recently, on 2022-05-28 and 2022-05-31. I mention it here, because it's something which might be easy to look at at the same time. If it happens another time or two, I'll probably create a separate bug report. [Note: the SE API servers being slightly out of sync is common, so SmokeDetector filters smaller differences out. It only reports a rollover more than once per day when the SE API servers get out of sync substantially by more than 5,000 of SmokeDetector's requests.]

2
  • 1
    Commenting here to notify you of the tagging change to status-bydesign, since the tag edit and edit to the answer wouldn't have notified you (unless you followed the answer). Commented Sep 19, 2022 at 12:07
  • @SonictheAnonymousHedgehog Thanks for the ping. I've left some comments on the answer.
    – Makyen
    Commented Sep 19, 2022 at 18:55

1 Answer 1

8

So, the official answer on this is that the SE API doesn't include quota information when it detects any errors while processing a request, because it cannot guarantee that the quota information got updated and will be accurate.

With that said, if you're seeing the API fail more frequently than it used to, we'd be interested in looking into what might be happening there, either that or if the API isn't failing and the quota information still isn't there.

EDIT: Marking this as status-bydesign for now. Feel free to reach out to us if any of the described above seems to be happening.

7
  • SmokeDetector runs 24/7/365. It made between 5k to 30k requests per day, every day, over the last 7 years. The chat search which I linked to in the question gives you a list of the 674 times we've received a response from the SE API without a quota_remaining property since the beginning of 2016, including a date/time stamp. 210 of those times have been since May 2022, which would be an average of about 563/year, which is 7.6 times the average rate of 74.4/year which had been seen in the prior 6.33 years.
    – Makyen
    Commented Sep 19, 2022 at 18:54
  • However, that preceding 6 years 4 months contains a few times where there were other known and resolved issues with the SE API. The largest of those was on 2019-11-13 and resulted in 163 chat messages reporting the lack of the quota_remaining property. Discounting that single day, leaves 301 other times in 6 1/3 years when SD hasn't seen a quota_remaining property in the response, for an average of 47.5 times/year. So, the rate since May 2022 is at least 11.78 times higher than "normal".
    – Makyen
    Commented Sep 19, 2022 at 18:54
  • Note: I wouldn't have been reporting anything if it didn't appear to be notably abnormal behavior. I could investigate this a bit further on our end, but definitively demonstrating it's an issue on SE's side would cost me actual money to set up yet another independent server to bang away at the SE API with a request that's a static string and record data as to when it gets these errors. From our point of view, this getting these is annoying, but it's at a level which we can easily ignore.
    – Makyen
    Commented Sep 19, 2022 at 18:54
  • 1
    Thank you for the further insight @Makyen, appreciate the help. I've selected 10 errors in your list and was able to associate each to a timeout on our end. It makes sense that we'd be seeing more of those for 2022, we are suffering more backend pressure than ever (including more DDoS attacks). We are working to improve the performance and reliability of various pieces of our backend, which will eventually translate into less errors on your end as well. At this point we just request patience until things improve on our end.
    – Felippe Rangel StaffMod
    Commented Sep 21, 2022 at 13:30
  • With that said, this is still status-bydesign, since an error will cause the quota_remaining to be missing in any API call.
    – Felippe Rangel StaffMod
    Commented Sep 21, 2022 at 13:31
  • 2
    Just "there's more load now" doesn't seem consistent with the issues being, primarily, at almost exactly the same times of day, to the minute, over a period of months. If it was just that there's more load or the result of DDoS attacks, I'd expect the pattern of errors to be significantly different, much more spread out throughout the day or concentrated into larger blocks of time. For DDoS attacks, I'd expect a substantially higher number during the periods when you've been under attack, which has appeared to be for hours or days, not just for a few minutes at the same time on many days.
    – Makyen
    Commented Sep 21, 2022 at 14:48
  • With the recent migration to Cloudflare, does it affect this issue/the API in general? Commented Oct 18, 2023 at 3:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .