13

Update: see bottom for new information, this isn't a fix for anything but the API is actually broke again on further investigation.

Full disclosure: I scrape a bunch of data (non-protected, things that don't require user authentication for) out of the API every night.

Often times I find patterns in this data and report them here if they are significant or my comrade finds issues with the documentation and we report them as well.

I found another pattern recently in the data-of-my-data or the meta-data of the meta-data (since my data is already meta-data).

I log how long it takes to complete API requests (because it's important to know, I have things that use a specific timing and I need to know when it's time to adjust my timing) and I found this interesting characteristic of the API requests:

Start   End Time Taken  Total Requests  Requests / Second   Milliseconds / Request  Sites Loaded    Milliseconds / Site Backoff Count   Backoff Seconds End Quota
2017-01-16 00:00:24 2017-01-16 00:05:30 0:05:05 2269    7.42    134.69  324 943.23  8   80  8875
2017-01-15 00:00:11 2017-01-15 00:05:02 0:04:51 2269    7.79    128.42  324 899.35  7   70  9182
2017-01-14 00:00:48 2017-01-14 00:05:48 0:04:59 2269    7.57    132.18  324 925.68  8   80  8792
2017-01-13 00:00:57 2017-01-13 00:08:26 0:07:28 2269    5.06    197.52  324 1383.25 9   90  8141
2017-01-12 00:00:35 2017-01-12 00:07:35 0:06:59 2269    5.41    184.99  324 1295.52 7   70  8358
2017-01-11 00:00:40 2017-01-11 00:08:12 0:07:32 2269    5.02    199.21  324 1395.11 8   80  8149

If you pay close enough attention (data is tab-separated, you can drop it in Excel / whatever if you like) you see that the times change somewhat drastically between the 13th and the 14th. They go from a consistent 7 minutes (maybe add 30 seconds sometimes) to a consistent 5 minutes. What caused the drop of two minutes off my request time?

You can also see that my backoff counts and whatnot haven't changed a bit, I'm getting 7-8 backoffs a night on average, and have been for months. (At most I've gotten 11 backoffs, the fewest has been 6.)

I've verified the data (twice now, because this bothered me) and everything is good, so something caused my 2269 requests (always the same number unless /badges/name goes down again).

The only other information I see that might be pertinent is that on Jan 12, 2017, the API revision changed so I assume it's something to do with a change on the SE side:

Gathered    ApiRevision
2017-01-16 00:02:39.1231102 2017.1.11.24452
2017-01-15 00:02:15.3911888 2017.1.11.24452
2017-01-14 00:02:52.5475485 2017.1.11.24452
2017-01-13 00:03:55.9381213 2017.1.11.24452
2017-01-12 00:03:14.5476025 2017.1.11.24452
2017-01-11 00:03:44.5318892 2017.1.3.24329

Any information would be greatly welcome, it doesn't affect me much (just decreases my compute time, which is nice) but it does not give me warm fuzzies inside to see such a drastic change when I changed absolutely nothing about what I'm doing. I've also verified with the provider of the server I reside this information on that they didn't change anything either, and they confirmed that my stack is the exact same as it has been.

As it turns out, further investigation revealed that the API is currently throwing a 500 Internal Server Error on the /badges/name endpoint again. (Apparently catching exceptions is way faster than getting the actual data from that endpoint.) I didn't notice because I don't currently use that data for anything. Seems to only happen when I set pagesize=100 (a completely valid parameter and value).

Previous bug report: Bug with `badges/name` endpoint

Improve this question
4
  • Is the API involved with Microsoft's ASP.NET in some way?
    – Anthony Pham
    Commented Jan 16, 2017 at 16:51
  • @AnthonyPham Which portion, mine or the SE side?
    – Der Kommissar
    Commented Jan 16, 2017 at 16:51
  • Both if possible. The SE side preferbably
    – Anthony Pham
    Commented Jan 16, 2017 at 16:52
  • @AnthonyPham From what I understand SE is an entirely Microsoft shop, the version number is consistent with the Microsoft numbering scheme as well, so the API should be ASP.NET. My side is gathered by a .NET Service on a server that runs each night at 00:00:30UTC+00:00 +/- 30 seconds. This server has no ASP.NET infrastructure on it, and dumps the data directly to a separate SQL server, which provides the data to a third server which is my ASP.NET server.
    – Der Kommissar
    Commented Jan 16, 2017 at 16:53

2 Answers 2

Reset to default
8

So, this finally got solved.

Apparently, it's heavily related to the Census badge being awarded recently and that broke the API for some reason.

I confirmed a few minutes ago that this endpoint is fixed, we'll see how the time updates stack up tonight when the automatic run occurs.

Improve this answer
2
  • It might be due to the tons of badges given...
    – Anthony Pham
    Commented Jan 21, 2017 at 1:02
  • @AnthonyPham SE told me it required an API rebuild to fix it. I'd bet that it wasn't prepared to handle a new badge.
    – Der Kommissar
    Commented Jan 21, 2017 at 1:17
2

This is probably a shot to the Moon, but I'll try even though I have probably no idea about your side or SE's API.

Stack Exchange's API is ASP.NET as you have mentioned in your comment. So while scanning around, zaq posted a comment in chat linking to Nick Craver's Twitter account. Conveniently, he is a SE developer and says miscellaneous things related to programming and some HTML stuff. While stalking him , I mean looking at his tweets on January 14, I found a miracle (I hope):

enter image description here

He first tweets something about the "MiniProfiler" then cleaning the stacks and the lack of StackTrace() which was a API oversight fixed in 2.0. I'm not quite sure how the stacks and "MiniProfiler" are related to the API but maybe there was some minor new update where the fixture of the StackTrace() was reversed. The rendering of the "MiniProfiler" could have caused Nick Craver to clean the stacks and complain about StackTrace() though.

Or simply blame SQL and not using enough jQuery...

enter image description here

Improve this answer
7
  • Reviewing his tweets, I found: twitter.com/Nick_Craver/status/819537688890056704 Perhaps there was something really ugly in the SQL that the API was using?
    – Der Kommissar
    Commented Jan 16, 2017 at 17:12
  • @EBrown It is SQL after all
    – Anthony Pham
    Commented Jan 16, 2017 at 17:15
  • I mean, I have had my fair share of not-fun times with SQL server. It really can be a d...pain sometimes.
    – Der Kommissar
    Commented Jan 16, 2017 at 17:16
  • @EBrown See my edit :)
    – Anthony Pham
    Commented Jan 16, 2017 at 17:20
  • What do you mean by "complains a lot of coding"?
    – This_is_NOT_a_forum
    Commented Jan 16, 2017 at 18:25
  • @PeterMortensen I'll fix that
    – Anthony Pham
    Commented Jan 16, 2017 at 19:03
  • @AnthonyPham As it turns out, the API is currently throwing an error for /badges/name again, I don't use the data collected from that endpoint so I didn't even notice. Grrr.
    – Der Kommissar
    Commented Jan 18, 2017 at 6:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .