Update: The maintenance went as planned.
tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites, Jobs, Chat, and Teams. All sites will be read-only for up to an hour on Saturday, March 28, 2020. Enterprise cloud-hosted instances will not be impacted.
There will be a service degradation for up to an hour at 13:00 UTC (9 AM US/Eastern) on Saturday, March 28th, 2020. During that time, questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read-only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're "read-only" for maintenance.
Intermediate Version of What's Taking Place?
Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.
What we'll be doing
During the service interruption, we will be applying Windows updates, and more importantly we'll be upgrading from SQL Server 2017 to SQL Server 2019. In order for us to do the work, we'll do a failover from the primary location to a secondary, then apply Windows patches and upgrade; in between each step, we'll reboot the affected server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.
Technical Version of What's Happening?
As mentioned we have two datacenters, one in NY and one in CO. In total, we have about 33 database servers, with the Stack Overflow/Stack Exchange sites databases on 6 of them. Each cluster (one for SO, one for SE) has a primary in NY, a local secondary in NY, and a remote secondary in CO.
In order to make our databases available in both NY and CO, we use Always On Availability Groups (AG), as well as Distributed Availability Groups (DAG). A SQL Server DAG allows us to distribute data from SO/SE to some of our internal SQL servers for reporting and other consumption.
There is a very specific order in which SQL Servers are to be updated when inside of an availability group:
- Remote secondaries - in our case CO
- Local secondaries - our backup servers in NY
- Primary Replica - the primary servers in NY
I have been slowly upgrading all of our SQL Servers to 2019, and we're now at the point where we need to do the main SQL clusters running the entire network. Most of our internal servers have been done, but now we need to patch/upgrade the ones in the AGs/DAGs...and there are 11 of them. They all need to be patched/upgraded in a very short amount of time.
More specifically, I will be patching/upgrading some of the secondaries on March 27th. Before the failover on the morning of the 28th, I'll patch and upgrade the remaining secondaries. All of the upgraded servers will be unavailable until we initiate the failover. This is because they become inaccessible since we're changing to a new SQL Server version.
Once we're ready on the 28th, we will flip the sites to read-only, perform the failover, begin patching Windows, and upgrading SQL Server on the remaining servers. When we execute the code to failover, there might be a brief period where we are offline while the SQL Servers are moving things around.
We expect that the site will be in a read-only state for less than an hour.
Questions or concerns?
Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.