44

Since this answer doesn’t seem to be drawing any attention from anyone who can do something about it, I’m posting it as a question. The Mathematics Stack Exchange has been in read-only mode for 7 hours now; it seems that the other Stack Exchange sites are back to normal, so perhaps Mathematics was somehow forgotten in the process of switching back?

8
  • 3
    I've now been automatically logged out and can't log back in – "Login is currently unavailable".
    – joriki
    Mar 18 at 9:05
  • 4
    Well it's in the ticketing system and it's a few hours to dawn in New York. Best I can do on my end ._.
    – Journeyman Geek Mod
    Mar 18 at 9:14
  • 7
    Great ! Only one site still down, and this must be the site I almost only used since a year. Typical for my bad luck !
    – Peter
    Mar 18 at 9:55
  • 8
    Sorry for this oversight, we’re on it.
    – Aaron Bertrand Staff
    Mar 18 at 11:36
  • 1
    Things are back to normal at MSE, thank you @AaronBertrand. Mar 18 at 11:52
  • 5
    The site is back. We'll post an answer when we have a full explanation of what happened.
    – Aaron Bertrand Staff
    Mar 18 at 11:52
  • 2
    Mathematics is restored
    – Arulkumar
    Mar 18 at 12:18
  • 1
    [status-unstuck] Mar 18 at 12:43

1 Answer 1

17

{ cross-posted from Why is MSE still down (read-only)? }

What happened?

During our planned maintenance window on March 17th/18th, a sequence of events in our Colorado data center left Mathematics Stack Exchange in read-only mode for several hours after the maintenance was finished.

TL;DR

The site is back up. The maintenance window was officially closed on March 18th at 01:28 UTC, at which point all other sites were fully operational, but Mathematics remained in read-only mode until it was manually corrected at roughly 11:46 UTC.

What caused the issue?

A number of factors contributed to the issue itself and our failure to recognize it. The long version:

  • As part of our manual failover process, we move all sites to read-only mode due to a performance issue we’ve been experiencing for months but have yet to solve.
  • This involves shifting site traffic to our web servers in Colorado, where they can read data from the secondaries there without any impact from operations happening in New York.
  • Fundamentally, the application should not try to write to a read-only secondary; if successive attempts fail, the source site is automatically switched to read-only mode for safety, because it’s a clear sign that something is awry.
  • In this case, the Mathematics site did try to write to a read-only secondary following a very specific (non-nefarious) sequence of events from an end user.
  • Because this happened while all of the other sites were in read-only mode, any notifications about this site specifically were silenced for maintenance, as we were intentionally setting the entire network read-only and also doing maintenance that would cause significant alert noise anyway.

What contributed to the duration?

At the end of our maintenance window, we spot check a handful of things, including a few specific web sites (but certainly not all of them). We don’t manually open every single web site in a browser to ensure the banner is gone, or test that we can successfully post a comment or vote on a post.

In this case, all of the places we checked at 01:30 UTC looked good, including our open-source monitoring tool, OpServer - which provides a fantastic pulse on the state of our databases. If a specific site’s primary database couldn’t be written to, we certainly would have seen it there. But the site itself does not surface its status there - that is shown elsewhere, on an internal dashboard. So we went to bed.

How can we improve?

We can improve internal notifications and time to resolution for this scenario:

  • We can manually detect when a site is automatically shifted to read-only in response to a scenario like the above {done}
  • We can automate and alert on that monitor
  • We can further improve the way the application attempts to write to the database during maintenance
  • We can add to our maintenance window runbooks:
    • Checking the internal dashboard in addition to OpServer and other checks {done}
    • Increasing the number of sites we spot check
    • Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too)
2
  • 1
    Regarding Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too) - that's certainly doable. When I had the time for it, I had all meta sites' websockets for new questions linked to a notification system and the influx of new questions was (usually) manageable. A cross-site SEDE query (run directly on the real meta DBs, of course) would work too, and I know you're capable of writing one :)
    – Glorfindel Mod
    Mar 18 at 20:24
  • 5
    @Glorfindel That's probably fair, but that's kind of like a last-resort catch-all. I'd much rather know about an issue before any user does. :-) (Also, some of the first bits of meta activity were comments on the original maintenance notification and not new posts.)
    – Aaron Bertrand Staff
    Mar 18 at 20:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .