Update: The maintenance went as planned. We were read-only for less than 10 minutes and everything is back online and working as expected.


tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites. All sites will be read-only for up to an hour on March 17, 2018. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to an hour at 13:00 UTC (9AM US/Eastern) on March 17th, 2018. During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won’t change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we’re ‘read only’ for maintenance.

You can watch us stream it on YouTube live.

Intermediate Version of What’s Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (well, actually New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another location so we can patch, upgrade, and reboot them.

What we’ll be doing

During the service interruption, the servers will be getting some much needed love and attention. This means that we’ll be applying Windows updates, SQL Server patches, and finally we’ll be upgrading from SQL Server 2016 to SQL Server 2017. In order for us to do the work, we’ll do a failover from the primary location to a secondary, then apply windows patches and upgrade; in between each step, we’ll reboot the affected server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.

Longer More Technical Version of What’s Happening?

As mentioned we have two datacenters, one in NY and one in CO. In total we have about 30 database servers, with the Stack Overflow/Stack Exchange sites databases on 6 of them. Each cluster (one for SO, one for SE) has a primary in NY, a local secondary in NY, and a remote secondary in CO.

In order to make our databases available in both NY and CO, we use Always On Availability Groups (AG), as well as Distributed Availability Groups (DAG). A SQL Server DAG allows us to distribute data from SO/SE to some of our internal SQL servers for reporting and other consumption.

There is a very specific order in which SQL Servers are to be updated when inside of an availability group:

  • Remote secondaries - in our case CO
  • Local secondaries - our backup servers in NY
  • Primary Replica - the primary servers in NY

Over the past month, I have been slowly patching and upgrading all of our SQL Servers. During this time, we discovered that if a server is a member of an AG/DAG and is downstream, meaning it’s a secondary, and it gets upgraded to a higher version of SQL Server (i.e. 2017) than the primary/upstream server (i.e. 2016) … it becomes unreadable by the rest of the group. This means we can no longer offload anything from the primary server (like read-only queries) or otherwise utilize these servers, until we failover to them. Needless to say, the initial patching plan needed to be reevaluated because we couldn’t spread out the patching of the AG/DAG servers as we’d initially planned.

Most of our internal servers have been done, but now we need to patch/upgrade the ones in the AGs/DAGs … and there are 11 of them. They all need to be patched/upgraded in a very short turnaround time.

More specifically, we will be patching/upgrading the remote and local secondaries on March 16th. This will make them unavailable until we perform the failover on the 17th. They will remain online and will continue to receive data from the primary databases though, and thus will be ready for the failover the next day.

On the 17th, we will flip the sites to read-only, perform the failover and begin patching Windows and upgrading SQL Server.

We expect that the site will be in a read-only state for less than an hour. Once we’re done with the maintenance, we will perform another failover to put things back in place as they were at the start, and take the site out of a read-only state.

Questions or concerns?

Please post a comment or answer below; I’ll do my best to address any concerns between now and the maintenance window.

  • 3
    Are chat and SEDE unaffected? – rene Mar 8 at 20:44
  • 6
    @rene I'd say chat will be impacted as it's running on the same server as the SE cluster, but I'll confirm. SEDE is on a separate server that's already been upgraded so it should be ok. – Taryn Mar 8 at 20:47
  • 8
    Looks like you'll be having a fun St. Patty's day! – Max Vernon Mar 8 at 21:40
  • 85
    But........................... What am I going to do for a whole hour? – K.Dᴀᴠɪs Mar 8 at 22:46
  • 13
  • 9
    @Yaroslav good news for you, we’re most likely going to livestream what we are going to do. I’m still writing up the full playbook but I’ve tested in a lab the ‘anticipated’ plan. – Taryn Mar 8 at 23:45
  • 36
    @K.Dᴀᴠɪs take a long walk and get some fresh air. – Taryn Mar 8 at 23:46
  • 26
    @K.Dᴀᴠɪs when I was at Bell Labs and the system was down, we said "this was a good time to consider the theoretical aspects of our research" – Tom Limoncelli Mar 9 at 14:31
  • 8
    @Taryn have you considered running SQL Server on Linux instead? – Nick Craver Mar 9 at 14:43
  • 11
    @NickCraver hmmmmmm – Taryn Mar 9 at 14:44
  • 29
    @Taryn ,...but, if you guys run into a problem during the maintenance, how are you going to post a question for us to solve? – ashleedawg Mar 9 at 18:30
  • 5
    During the outage please post your questions on your local Craigslist site. (This listing must be from the last outage.) – ashleedawg Mar 9 at 18:38
  • 13
    @Mast Rollback plan is don't screw up. – Taryn Mar 9 at 18:44
  • 6
    I'm debating whether I should flag @MaxVernon's comment as offensive. – TRiG is Timothy Richard Green Mar 10 at 16:52
  • 11
    We salute you for getting such a wonderful site working with Microsoft Windows and SQL. Who would think it was possible, but you did it. – ctrl-alt-delor Mar 12 at 23:40

You must log in to answer this question.

Browse other questions tagged .