Update: The maintenance went as planned. We were read-only for less than 10 minutes and everything is back online and working as expected.
tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites. All sites will be read-only for up to an hour on March 17, 2018. Enterprise cloud hosted instances will not be impacted.
Short Version:
There will be a service degradation for up to an hour at 13:00 UTC (9AM US/Eastern) on March 17th, 2018. During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won’t change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we’re ‘read only’ for maintenance.
You can watch us stream it on YouTube live.
Intermediate Version of What’s Taking Place?
Background
Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (well, actually New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another location so we can patch, upgrade, and reboot them.
What we’ll be doing
During the service interruption, the servers will be getting some much needed love and attention. This means that we’ll be applying Windows updates, SQL Server patches, and finally we’ll be upgrading from SQL Server 2016 to SQL Server 2017. In order for us to do the work, we’ll do a failover from the primary location to a secondary, then apply windows patches and upgrade; in between each step, we’ll reboot the affected server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.
Longer More Technical Version of What’s Happening?
As mentioned we have two datacenters, one in NY and one in CO. In total we have about 30 database servers, with the Stack Overflow/Stack Exchange sites databases on 6 of them. Each cluster (one for SO, one for SE) has a primary in NY, a local secondary in NY, and a remote secondary in CO.
In order to make our databases available in both NY and CO, we use Always On Availability Groups (AG), as well as Distributed Availability Groups (DAG). A SQL Server DAG allows us to distribute data from SO/SE to some of our internal SQL servers for reporting and other consumption.
There is a very specific order in which SQL Servers are to be updated when inside of an availability group:
- Remote secondaries - in our case CO
- Local secondaries - our backup servers in NY
- Primary Replica - the primary servers in NY
Over the past month, I have been slowly patching and upgrading all of our SQL Servers. During this time, we discovered that if a server is a member of an AG/DAG and is downstream, meaning it’s a secondary, and it gets upgraded to a higher version of SQL Server (i.e. 2017) than the primary/upstream server (i.e. 2016) … it becomes unreadable by the rest of the group. This means we can no longer offload anything from the primary server (like read-only queries) or otherwise utilize these servers, until we failover to them. Needless to say, the initial patching plan needed to be reevaluated because we couldn’t spread out the patching of the AG/DAG servers as we’d initially planned.
Most of our internal servers have been done, but now we need to patch/upgrade the ones in the AGs/DAGs … and there are 11 of them. They all need to be patched/upgraded in a very short turnaround time.
More specifically, we will be patching/upgrading the remote and local secondaries on March 16th. This will make them unavailable until we perform the failover on the 17th. They will remain online and will continue to receive data from the primary databases though, and thus will be ready for the failover the next day.
On the 17th, we will flip the sites to read-only, perform the failover and begin patching Windows and upgrading SQL Server.
We expect that the site will be in a read-only state for less than an hour. Once we’re done with the maintenance, we will perform another failover to put things back in place as they were at the start, and take the site out of a read-only state.
Questions or concerns?
Please post a comment or answer below; I’ll do my best to address any concerns between now and the maintenance window.
