math.SE has been stuck in read-only mode for 7 hours

Question

Since this answer doesn’t seem to be drawing any attention from anyone who can do something about it, I’m posting it as a question. The Mathematics Stack Exchange has been in read-only mode for 7 hours now; it seems that the other Stack Exchange sites are back to normal, so perhaps Mathematics was somehow forgotten in the process of switching back?

Answer 1

{ cross-posted from Why is MSE still down (read-only)? }

What happened?

During our planned maintenance window on March 17th/18th, a sequence of events in our Colorado data center left Mathematics Stack Exchange in read-only mode for several hours after the maintenance was finished.

TL;DR

The site is back up. The maintenance window was officially closed on March 18th at 01:28 UTC, at which point all other sites were fully operational, but Mathematics remained in read-only mode until it was manually corrected at roughly 11:46 UTC.

What caused the issue?

A number of factors contributed to the issue itself and our failure to recognize it. The long version:

• As part of our manual failover process, we move all sites to read-only mode due to a performance issue we’ve been experiencing for months but have yet to solve.
• This involves shifting site traffic to our web servers in Colorado, where they can read data from the secondaries there without any impact from operations happening in New York.
• Fundamentally, the application should not try to write to a read-only secondary; if successive attempts fail, the source site is automatically switched to read-only mode for safety, because it’s a clear sign that something is awry.
• In this case, the Mathematics site did try to write to a read-only secondary following a very specific (non-nefarious) sequence of events from an end user.
• Because this happened while all of the other sites were in read-only mode, any notifications about this site specifically were silenced for maintenance, as we were intentionally setting the entire network read-only and also doing maintenance that would cause significant alert noise anyway.

What contributed to the duration?

At the end of our maintenance window, we spot check a handful of things, including a few specific web sites (but certainly not all of them). We don’t manually open every single web site in a browser to ensure the banner is gone, or test that we can successfully post a comment or vote on a post.

In this case, all of the places we checked at 01:30 UTC looked good, including our open-source monitoring tool, OpServer - which provides a fantastic pulse on the state of our databases. If a specific site’s primary database couldn’t be written to, we certainly would have seen it there. But the site itself does not surface its status there - that is shown elsewhere, on an internal dashboard. So we went to bed.

How can we improve?

We can improve internal notifications and time to resolution for this scenario:

• We can manually detect when a site is automatically shifted to read-only in response to a scenario like the above {done}
• We can automate and alert on that monitor
• We can further improve the way the application attempts to write to the database during maintenance
• We can add to our maintenance window runbooks:
  • Checking the internal dashboard in addition to OpServer and other checks {done}
  • Increasing the number of sites we spot check
  • Waiting an hour and checking meta (though there are a lot of individual meta sites to check, too)