173

TL;DR: On June 15, 2023, starting at 13:52 UTC, an integer overflow in our database caused the SE Network sites to become unavailable for all users.

Timeline:

On June 15, 2023, at 13:52 UTC, we started noticing elevated errors related to changes made to our main database. By 13:57 UTC, SE Network sites began returning 500 errors for logged-in users, and by 13:58 UTC, all SE Network sites became unavailable for all users.

Our DBRE team quickly identified the root cause and started working on a solution. By 14:00 UTC, they applied a fix to the database, which corrected the original errors. However, at 14:05 UTC, we observed that SQL CPU utilization was still over 90% due to the initial error, and it did not return to normal operation parameters.

To further investigate the cause of the SQL CPU contention, we decided at 14:39 UTC to direct all traffic to our secondary data center in read-only mode. At 14:42 UTC, we turned off one of our debugging tools that frequently polls SQL to help narrow down the list of potential causes. At 15:02 UTC, we decided to stop the main application in our primary data center to halt all SQL transactions. This led to SQL CPU utilization dropping to idle levels in our primary data center by 15:05 UTC.

We then proceeded to start the application again one by one, carefully monitoring SQL utilization and looking for signs of contention. By 15:20 UTC, the application was fully running in our primary data center, and all metrics were within normal limits. Finally, at 15:24 UTC, with the primary data center now healthy, we directed all traffic back and the website came back up.

StackStatus:

During the incident, we noticed https://www.stackstatus.net was unavailable. We use a 3rd-party incident management tool to help us work through incidents, and they host our status page. It seems they were not quite equipped to handle the burst of traffic from this outage. We’ve notified the vendor, and they’ve since resolved the issue and have a post-mortem to run themselves.

Some technical details:

The maximum value an integer can have is a little over 2 billion. If a table fills at about half a million rows per day, divided out, that means it takes roughly 12 years to reach the maximum.

The database that caused the issues today has a table that has been steadily incrementing for that long until we ran out of integers and overflowed. We previously implemented a fix to mitigate this issue and added additional monitoring for all the indices. Today’s incident occurred because we were restoring data that inadvertently caused the index to revert to the previous overflowed state. Yup, we totally fixed, un-fixed, then re-fixed the problem. We needed to go to the store and buy more integers… again. Our bad, y’all.

Technical bits aside, you know what's odd? Any integer not wholly divisible by 2!

19
  • 42
    Tell me you were testing in production without telling me you were testing in production: Today’s incident occurred because we were restoring data that inadvertently caused the index to revert to the previous overflowed state.
    – Hack5
    Jun 15, 2023 at 19:42
  • 91
    @Hack5 Well, you know what they say... everybody has a testing environment, but only some are lucky enough to have a dedicated place for production.
    – Adam Lear StaffMod
    Jun 15, 2023 at 19:48
  • 70
    That was enlightening to read in these bleak days, and you have my sincere thanks for that. I wish everything would be so transparent, full with spicy details, and above all - admitting mistakes. Well done!!! Jun 15, 2023 at 19:58
  • 19
    My company also ran out of integers a few years ago. So much for Cantor.
    – Barmar
    Jun 15, 2023 at 20:16
  • 8
    The cascading failure of stackstatus.net is interesting, and a good argument against distribution of work to third parties for the sake of reliability. Jun 15, 2023 at 20:47
  • 1
    Don't unfix your fix with the planned maintenance now! (Though I guess that would be hard since it's unrelated.) Jun 15, 2023 at 21:02
  • 33
    No audience will go directly to the status page as quickly as this audience. That's quite a burst of traffic for them, and difficult to prepare for. It's interesting to see that it is struggling again now during the scheduled maintenance. It does include 2MB of JavaScript though...
    – Liam
    Jun 15, 2023 at 21:25
  • 4
    Thanks @Liam, that's a good data point I can give them.
    – Josh Zhang StaffMod
    Jun 15, 2023 at 21:26
  • 9
    Just curious ... Was this a DB row index? So, what was the [initial] fix? To start over at zero (i.e. problem will reoccur in 2035)? Change field to unsigned to get another 2 billion? Was the field/number width increased to 64 bits? Jun 16, 2023 at 0:31
  • 10
    Thank you for sharing this. It is heartwarming to read this report on the network status. Hope to see this type of communications become an example for every team, every policy, every incident...
    – M--
    Jun 16, 2023 at 0:55
  • 5
    "We needed to go to the store and buy more integers… again" Ah yes, the good old freestore. Make sure to try them on first, so that they are of the right size and signedness. Don't buy them online - last time I did that I got some big endian 16 bit ones in one's complement.
    – Lundin
    Jun 16, 2023 at 6:29
  • 6
    @Barmar : your company should turn to integerv6, or use NAT Jun 16, 2023 at 7:31
  • 7
    Please tell me your fix was to use bigint, and not to roll it back round to -2billion. Why didn't you realize this was a heavily used table and use bigint in the first place? Hindsight I suppose. Jun 16, 2023 at 10:32
  • 2
    @Charlieface SEDE only gets updated on Sundays, IIRC
    – Lino
    Jun 16, 2023 at 11:20
  • 10
    @Charlieface their job is to minimize downtime, not to have infinite foresight or to correct issues in a way that won't possibly draw any scrutiny from a random commenter on meta. Just changing a PK column to bigint has other far-reaching implications, especially for a company who famously jams all of their database queries into C# and dapper (so imagine what happens to the applications, API, downstream sources like SEDE, etc. when Int32 can no longer fit the output). Jun 16, 2023 at 16:18

3 Answers 3

115

Stack Overflow... overflowed
#irony

1
  • 71
    time to rename the company to Integer Overflow, huh
    – Lino
    Jun 15, 2023 at 20:26
25

One question bugs me though, is this related to the planned maintenance that is scheduled to start soon?

Even if not, will it affect that maintenance?

4
  • 44
    Completely unrelated and will not affect it. We talked about this internally and our confidence is high. In fact I'm heading to the data center right now.
    – Josh Zhang StaffMod
    Jun 15, 2023 at 20:05
  • 6
    @Josh beware dark shadows lurking in the data center, and call me if you need help banishing them. ;-) Jun 15, 2023 at 20:10
  • 1
    @JoshZhang I'm usually not happy when hearing "our confidence is high" immediately followed by "I'm heading to the data center right now" ... :) Jun 20, 2023 at 3:10
  • @HagenvonEitzen my presence at the data center was always part of the maintenance plan. Also high confidence doesn't mean we don't plan for contingencies :)
    – Josh Zhang StaffMod
    Jun 20, 2023 at 12:25
10

So while you're doing this, do you think it might be a good time to evaluate other potential overflow errors in the future? What I mean is that this reminds me of the 2038 Problem in the sense is that it is at least somewhat predictable when these issues will start to show themselves. Perhaps now is a good time to try and figure that out.

I'm sure that is a lot easier said than done, but still...


Edit: Some people took this as condescending and I want to clarify that I certainly didn't mean it like that.

My point is just that in my experience, if it happens in one table in the database it could potentially happen in another. I'm sure everyone would prefer finding them ahead of time and fixing them before becoming a problem instead of having further incidents.

What is missing from this response for me is the fact that SE didn't say anything about making sure similar things didn't happen again. It's more of just an "this happened, oops :("

7
  • 12
    I mean, he said they were aware of this and mitigated it. They just inadvertently undid the fix. This isn't really something you plan for as much as you learn from Jun 16, 2023 at 6:32
  • @MissSkooter Sure you can plan for it: you put in place monitoring that tells you when an int IDENTITY column is increasing fast enough that it's going to need a bigint Jun 16, 2023 at 10:33
  • 1
    Why do you think they haven't already evaluated that? Jun 16, 2023 at 11:37
  • 3
    @Stuckat1337 Mainly because there was clearly an issue that seemed to take them by surprise.
    – Chipster
    Jun 16, 2023 at 15:37
  • 6
    "do you think it might be a good time" is incredibly condescending. From the technical description it sounds like they reseeded the table to -2 billion, but then restored some row(s) in the positive range, which wouldn't actually reseed back to the 2 billion mark, but rather lead to PK violations. Instead of attacking the team for not being as smart as you, and trying to teach them a lesson you think they're incapable of learning without your guidance, you might actually want to acknowledge them for addressing the issue as quickly as they did. Jun 16, 2023 at 16:15
  • 7
    ...and being honest about it. They didn't have to give any technical details at all, and nit-picking them to death and talking down to them is unlikely to have any impact other than reverting back to descriptions we used to get, like "there was a database problem, we fixed it, have a good one." Jun 16, 2023 at 16:20
  • 6
    Stack Exchange will not exist in 2038, so why worry? Jun 16, 2023 at 19:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .