Update: see bottom for new information, this isn't a fix for anything but the API is actually broke again on further investigation.
Full disclosure: I scrape a bunch of data (non-protected, things that don't require user authentication for) out of the API every night.
I found another pattern recently in the data-of-my-data or the meta-data of the meta-data (since my data is already meta-data).
I log how long it takes to complete API requests (because it's important to know, I have things that use a specific timing and I need to know when it's time to adjust my timing) and I found this interesting characteristic of the API requests:
Start End Time Taken Total Requests Requests / Second Milliseconds / Request Sites Loaded Milliseconds / Site Backoff Count Backoff Seconds End Quota 2017-01-16 00:00:24 2017-01-16 00:05:30 0:05:05 2269 7.42 134.69 324 943.23 8 80 8875 2017-01-15 00:00:11 2017-01-15 00:05:02 0:04:51 2269 7.79 128.42 324 899.35 7 70 9182 2017-01-14 00:00:48 2017-01-14 00:05:48 0:04:59 2269 7.57 132.18 324 925.68 8 80 8792 2017-01-13 00:00:57 2017-01-13 00:08:26 0:07:28 2269 5.06 197.52 324 1383.25 9 90 8141 2017-01-12 00:00:35 2017-01-12 00:07:35 0:06:59 2269 5.41 184.99 324 1295.52 7 70 8358 2017-01-11 00:00:40 2017-01-11 00:08:12 0:07:32 2269 5.02 199.21 324 1395.11 8 80 8149
If you pay close enough attention (data is tab-separated, you can drop it in Excel / whatever if you like) you see that the times change somewhat drastically between the 13th and the 14th. They go from a consistent 7 minutes (maybe add 30 seconds sometimes) to a consistent 5 minutes. What caused the drop of two minutes off my request time?
You can also see that my backoff counts and whatnot haven't changed a bit, I'm getting 7-8 backoffs a night on average, and have been for months. (At most I've gotten 11 backoffs, the fewest has been 6.)
I've verified the data (twice now, because this bothered me) and everything is good, so something caused my 2269 requests (always the same number unless
/badges/name goes down again).
The only other information I see that might be pertinent is that on Jan 12, 2017, the API revision changed so I assume it's something to do with a change on the SE side:
Gathered ApiRevision 2017-01-16 00:02:39.1231102 2017.1.11.24452 2017-01-15 00:02:15.3911888 2017.1.11.24452 2017-01-14 00:02:52.5475485 2017.1.11.24452 2017-01-13 00:03:55.9381213 2017.1.11.24452 2017-01-12 00:03:14.5476025 2017.1.11.24452 2017-01-11 00:03:44.5318892 2017.1.3.24329
Any information would be greatly welcome, it doesn't affect me much (just decreases my compute time, which is nice) but it does not give me warm fuzzies inside to see such a drastic change when I changed absolutely nothing about what I'm doing. I've also verified with the provider of the server I reside this information on that they didn't change anything either, and they confirmed that my stack is the exact same as it has been.
As it turns out, further investigation revealed that the API is currently throwing a
500 Internal Server Error on the
/badges/name endpoint again. (Apparently catching exceptions is way faster than getting the actual data from that endpoint.) I didn't notice because I don't currently use that data for anything. Seems to only happen when I set
pagesize=100 (a completely valid parameter and value).
Previous bug report: Bug with `badges/name` endpoint