Since this answer doesn’t seem to be drawing any attention from anyone who can do something about it, I’m posting it as a question. The Mathematics Stack Exchange has been in read-only mode for 7 hours now; it seems that the other Stack Exchange sites are back to normal, so perhaps Mathematics was somehow forgotten in the process of switching back?
-
3I've now been automatically logged out and can't log back in – "Login is currently unavailable".– jorikiMar 18, 2022 at 9:05
-
4Well it's in the ticketing system and it's a few hours to dawn in New York. Best I can do on my end ._.– Journeyman Geek ModMar 18, 2022 at 9:14
-
7Great ! Only one site still down, and this must be the site I almost only used since a year. Typical for my bad luck !– PeterMar 18, 2022 at 9:55
-
8Sorry for this oversight, we’re on it.– Aaron Bertrand StaffMar 18, 2022 at 11:36
-
1Things are back to normal at MSE, thank you @AaronBertrand.– Sarvesh Ravichandran IyerMar 18, 2022 at 11:52
-
5The site is back. We'll post an answer when we have a full explanation of what happened.– Aaron Bertrand StaffMar 18, 2022 at 11:52
-
2Mathematics is restored– ArulkumarMar 18, 2022 at 12:18
-
1[status-unstuck]– Robert ColumbiaMar 18, 2022 at 12:43
1 Answer
{ cross-posted from Why is MSE still down (read-only)? }
What happened?
During our planned maintenance window on March 17th/18th, a sequence of events in our Colorado data center left Mathematics Stack Exchange in read-only mode for several hours after the maintenance was finished.
TL;DR
The site is back up. The maintenance window was officially closed on March 18th at 01:28 UTC, at which point all other sites were fully operational, but Mathematics remained in read-only mode until it was manually corrected at roughly 11:46 UTC.
What caused the issue?
A number of factors contributed to the issue itself and our failure to recognize it. The long version:
- As part of our manual failover process, we move all sites to read-only mode due to a performance issue we’ve been experiencing for months but have yet to solve.
- This involves shifting site traffic to our web servers in Colorado, where they can read data from the secondaries there without any impact from operations happening in New York.
- Fundamentally, the application should not try to write to a read-only secondary; if successive attempts fail, the source site is automatically switched to read-only mode for safety, because it’s a clear sign that something is awry.
- In this case, the Mathematics site did try to write to a read-only secondary following a very specific (non-nefarious) sequence of events from an end user.
- Because this happened while all of the other sites were in read-only mode, any notifications about this site specifically were silenced for maintenance, as we were intentionally setting the entire network read-only and also doing maintenance that would cause significant alert noise anyway.
What contributed to the duration?
At the end of our maintenance window, we spot check a handful of things, including a few specific web sites (but certainly not all of them). We don’t manually open every single web site in a browser to ensure the banner is gone, or test that we can successfully post a comment or vote on a post.
In this case, all of the places we checked at 01:30 UTC looked good, including our open-source monitoring tool, OpServer - which provides a fantastic pulse on the state of our databases. If a specific site’s primary database couldn’t be written to, we certainly would have seen it there. But the site itself does not surface its status there - that is shown elsewhere, on an internal dashboard. So we went to bed.
How can we improve?
We can improve internal notifications and time to resolution for this scenario:
- We can manually detect when a site is automatically shifted to read-only in response to a scenario like the above {done}
- We can automate and alert on that monitor
- We can further improve the way the application attempts to write to the database during maintenance
- We can add to our maintenance window runbooks:
- Checking the internal dashboard in addition to OpServer and other checks {done}
- Increasing the number of sites we spot check
- Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too)
-
1Regarding Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too) - that's certainly doable. When I had the time for it, I had all meta sites' websockets for new questions linked to a notification system and the influx of new questions was (usually) manageable. A cross-site SEDE query (run directly on the real meta DBs, of course) would work too, and I know you're capable of writing one :)– Glorfindel ModMar 18, 2022 at 20:24
-
5@Glorfindel That's probably fair, but that's kind of like a last-resort catch-all. I'd much rather know about an issue before any user does. :-) (Also, some of the first bits of meta activity were comments on the original maintenance notification and not new posts.)– Aaron Bertrand StaffMar 18, 2022 at 20:28