Update: The maintenance went as planned. We were read-only for less than 10 minutes and everything is back online and working as expected.

tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites. All sites will be read-only for up to an hour on March 17, 2018. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to an hour at 13:00 UTC (9AM US/Eastern) on March 17th, 2018. During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won’t change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we’re ‘read only’ for maintenance.

You can watch us stream it on YouTube live.

Intermediate Version of What’s Taking Place?


Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (well, actually New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another location so we can patch, upgrade, and reboot them.

What we’ll be doing

During the service interruption, the servers will be getting some much needed love and attention. This means that we’ll be applying Windows updates, SQL Server patches, and finally we’ll be upgrading from SQL Server 2016 to SQL Server 2017. In order for us to do the work, we’ll do a failover from the primary location to a secondary, then apply windows patches and upgrade; in between each step, we’ll reboot the affected server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.

Longer More Technical Version of What’s Happening?

As mentioned we have two datacenters, one in NY and one in CO. In total we have about 30 database servers, with the Stack Overflow/Stack Exchange sites databases on 6 of them. Each cluster (one for SO, one for SE) has a primary in NY, a local secondary in NY, and a remote secondary in CO.

In order to make our databases available in both NY and CO, we use Always On Availability Groups (AG), as well as Distributed Availability Groups (DAG). A SQL Server DAG allows us to distribute data from SO/SE to some of our internal SQL servers for reporting and other consumption.

There is a very specific order in which SQL Servers are to be updated when inside of an availability group:

  • Remote secondaries - in our case CO
  • Local secondaries - our backup servers in NY
  • Primary Replica - the primary servers in NY

Over the past month, I have been slowly patching and upgrading all of our SQL Servers. During this time, we discovered that if a server is a member of an AG/DAG and is downstream, meaning it’s a secondary, and it gets upgraded to a higher version of SQL Server (i.e. 2017) than the primary/upstream server (i.e. 2016) … it becomes unreadable by the rest of the group. This means we can no longer offload anything from the primary server (like read-only queries) or otherwise utilize these servers, until we failover to them. Needless to say, the initial patching plan needed to be reevaluated because we couldn’t spread out the patching of the AG/DAG servers as we’d initially planned.

Most of our internal servers have been done, but now we need to patch/upgrade the ones in the AGs/DAGs … and there are 11 of them. They all need to be patched/upgraded in a very short turnaround time.

More specifically, we will be patching/upgrading the remote and local secondaries on March 16th. This will make them unavailable until we perform the failover on the 17th. They will remain online and will continue to receive data from the primary databases though, and thus will be ready for the failover the next day.

On the 17th, we will flip the sites to read-only, perform the failover and begin patching Windows and upgrading SQL Server.

We expect that the site will be in a read-only state for less than an hour. Once we’re done with the maintenance, we will perform another failover to put things back in place as they were at the start, and take the site out of a read-only state.

Questions or concerns?

Please post a comment or answer below; I’ll do my best to address any concerns between now and the maintenance window.

  • 3
    Are chat and SEDE unaffected?
    – rene
    Mar 8, 2018 at 20:44
  • 6
    @rene I'd say chat will be impacted as it's running on the same server as the SE cluster, but I'll confirm. SEDE is on a separate server that's already been upgraded so it should be ok.
    – Taryn
    Mar 8, 2018 at 20:47
  • 8
    Looks like you'll be having a fun St. Patty's day! Mar 8, 2018 at 21:40
  • 85
    But........................... What am I going to do for a whole hour? Mar 8, 2018 at 22:46
  • 13
  • 9
    @Yaroslav good news for you, we’re most likely going to livestream what we are going to do. I’m still writing up the full playbook but I’ve tested in a lab the ‘anticipated’ plan.
    – Taryn
    Mar 8, 2018 at 23:45
  • 36
    @K.Dᴀᴠɪs take a long walk and get some fresh air.
    – Taryn
    Mar 8, 2018 at 23:46
  • 26
    @K.Dᴀᴠɪs when I was at Bell Labs and the system was down, we said "this was a good time to consider the theoretical aspects of our research"
    – Tom Limoncelli StaffMod
    Mar 9, 2018 at 14:31
  • 8
    @Taryn have you considered running SQL Server on Linux instead? Mar 9, 2018 at 14:43
  • 11
    @NickCraver hmmmmmm
    – Taryn
    Mar 9, 2018 at 14:44
  • 29
    @Taryn ,...but, if you guys run into a problem during the maintenance, how are you going to post a question for us to solve?
    – ashleedawg
    Mar 9, 2018 at 18:30
  • 6
    During the outage please post your questions on your local Craigslist site. (This listing must be from the last outage.)
    – ashleedawg
    Mar 9, 2018 at 18:38
  • 13
    @Mast Rollback plan is don't screw up.
    – Taryn
    Mar 9, 2018 at 18:44
  • 6
    I'm debating whether I should flag @MaxVernon's comment as offensive.
    – TRiG
    Mar 10, 2018 at 16:52
  • 11
    We salute you for getting such a wonderful site working with Microsoft Windows and SQL. Who would think it was possible, but you did it. Mar 12, 2018 at 23:40


Browse other questions tagged .