81

Update: The maintenance went as planned.

tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites, Jobs, Chat, and Teams. All sites will be read-only for up to an hour on Saturday, March 28, 2020. Enterprise cloud-hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to an hour at 13:00 UTC (9 AM US/Eastern) on Saturday, March 28th, 2020. During that time, questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read-only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're "read-only" for maintenance.

Intermediate Version of What's Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.

What we'll be doing

During the service interruption, we will be applying Windows updates, and more importantly we'll be upgrading from SQL Server 2017 to SQL Server 2019. In order for us to do the work, we'll do a failover from the primary location to a secondary, then apply Windows patches and upgrade; in between each step, we'll reboot the affected server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.

Technical Version of What's Happening?

As mentioned we have two datacenters, one in NY and one in CO. In total, we have about 33 database servers, with the Stack Overflow/Stack Exchange sites databases on 6 of them. Each cluster (one for SO, one for SE) has a primary in NY, a local secondary in NY, and a remote secondary in CO.

In order to make our databases available in both NY and CO, we use Always On Availability Groups (AG), as well as Distributed Availability Groups (DAG). A SQL Server DAG allows us to distribute data from SO/SE to some of our internal SQL servers for reporting and other consumption.

There is a very specific order in which SQL Servers are to be updated when inside of an availability group:

  • Remote secondaries - in our case CO
  • Local secondaries - our backup servers in NY
  • Primary Replica - the primary servers in NY

I have been slowly upgrading all of our SQL Servers to 2019, and we're now at the point where we need to do the main SQL clusters running the entire network. Most of our internal servers have been done, but now we need to patch/upgrade the ones in the AGs/DAGs...and there are 11 of them. They all need to be patched/upgraded in a very short amount of time.

More specifically, I will be patching/upgrading some of the secondaries on March 27th. Before the failover on the morning of the 28th, I'll patch and upgrade the remaining secondaries. All of the upgraded servers will be unavailable until we initiate the failover. This is because they become inaccessible since we're changing to a new SQL Server version.

Once we're ready on the 28th, we will flip the sites to read-only, perform the failover, begin patching Windows, and upgrading SQL Server on the remaining servers. When we execute the code to failover, there might be a brief period where we are offline while the SQL Servers are moving things around.

We expect that the site will be in a read-only state for less than an hour.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

Improve this question
20
  • 1
    Will there be status updates provided should things take longer than the expected window and where?
    – JFoxx64
    Feb 25, 2020 at 20:10
  • 19
    @RageFoxx We tend to post to @StackStatus during the maintenance.
    – Taryn
    Feb 25, 2020 at 20:14
  • 7
    Thanks for the heads up. I'm available for mental support, if there is a need.
    – rene
    Feb 25, 2020 at 20:25
  • 45
    Who's terrible idea was it to run on SQL Server?!?
    – Nick Craver
    Feb 25, 2020 at 20:38
  • 32
    @NickCraver Especially on Windows...geez.
    – Taryn
    Feb 25, 2020 at 20:46
  • 4
    @Nick Weren't you part at this decision?? :D
    – πάντα ῥεῖ
    Feb 25, 2020 at 21:49
  • 3
    Shoulda started with PL/SQL 😀
    – Rob
    Feb 25, 2020 at 22:27
  • 17
    Are you upgrading Windows Server to Linux? 🤣
    – Phil
    Feb 26, 2020 at 10:59
  • 4
    Does this really need to be featured?
    – pppery
    Feb 27, 2020 at 2:54
  • 30
    @pppery Yes, we always provide notice when we are performing maintenance that could lead to a potential outage or read-only, especially if it could be for an extended period of time. Not everyone visits Meta to see these announcements, so featuring it gives the notice more visibility.
    – Taryn
    Feb 27, 2020 at 2:56
  • 1
    Good luck. Hopefully this won’t as stressful as it was in the past!
    – dalearn
    Mar 23, 2020 at 18:20
  • 2
    @pppery Yes, absolutely. It's great that they do this!
    – Asteroids With Wings
    Mar 24, 2020 at 11:33
  • 4
    "we will be applying Windows updates". You are not using Linux?
    – user1271772
    Mar 24, 2020 at 19:05
  • 5
    @user1271772 Yes, they are using Linux but old habits do not die easily ... I have a cron job that runs apt-get install windows-updates every other Tuesday ...
    – rene
    Mar 25, 2020 at 7:42
  • 6
    @user1271772 We use both Windows and Linux on various servers, but all of our SQL Servers are Windows servers, so, yes, we need to install Windows Updates.
    – Taryn
    Mar 25, 2020 at 16:14

3 Answers 3

Reset to default
52

A distinct "meta" answer on the how, not the what.

Another great example how interaction between the company and the community can look like: obvious, Taryn put in quite some effort to let us know what is going to happen and why! And on top of that, the promise to be available and answer.

And not surprisingly, the community reacts kindly.

Now let's just keep our fingers crossed that the upgrade works out as planned, so that no users comes in later on bitterly complaining about random acts of incompetence. So, seriously folks: even in case something goes wrong, don't use this place for nerdish attacks on say disliked operating systems...

One thing though: unless I am mistaken, you only posted this announcement here. I appreciate posting on MSE, but I think that putting up a similar post on MSO could be beneficial.

Improve this answer
6
  • 28
    The post is linked in the 'Featured on Meta' links — that's probably sufficient. That's how I came across it.
    – Jonathan Leffler
    Feb 26, 2020 at 4:11
  • 2
    @JonathanLeffler makes sense, but I am on mobile, and you can't see that part there. Thanks for the clarification!
    – GhostCat
    Feb 26, 2020 at 4:25
  • 22
    @GhostCatsalutesMonicaC. I didn't post on MSO because this announcement is featured so it will show in the sidebar on Stack Overflow. I followed our normal operating procedure for these types of announcements.
    – Taryn
    Feb 26, 2020 at 13:59
  • 3
    @Taryn Sure, makes all sense.
    – GhostCat
    Feb 26, 2020 at 14:24
  • 7
    Note that, as far as my moderate involvement is concerned, I've always felt to have been properly informed of the various technical maintenances, tests and migrations from SE team for several years now.
    – Pac0
    Feb 27, 2020 at 12:25
  • Disliked operating systems deleted all my data long before version 10 was released that had a similar issue.
    – S.S. Anne
    Mar 28, 2020 at 20:25
7

Good luck with the upgrades, we appreciate these updates.

And I think I speak for everyone when I say that I hope that anyone who has to physically go in on Saturday stays safe and doesn't catch the dread virus.

Sending good, antiviral vibes your way!

Improve this answer
2
  • 4
    Thankfully, we don't have to go anywhere to do the work. We can do all the upgrades remotely.
    – Taryn
    Mar 26, 2020 at 16:13
  • 1
    @Taryn and what if someone shuts down a server by mistake? ;-)
    – Shadow The GPT Wizard
    Mar 26, 2020 at 22:00
1

I read the revision history, and noticed that you mentioned apparently unrelated "CO issues". What were those?

I'm glad that this was fast and fairly painless. Thanks for your work!

Improve this answer
4
  • Wild guess: Corona(virus) Outbreak
    – Jenayah
    Mar 28, 2020 at 0:09
  • @Jenayah Wild guess: Colorado, where one of their servers is located.
    – S.S. Anne
    Mar 28, 2020 at 0:10
  • Plot twist: it's both! :-)
    – Jenayah
    Mar 28, 2020 at 0:10
  • 5
    The week before this was originally scheduled, we had a severe hardware failure in our Colorado aka CO datacenter. We pushed the maintenance because we were all too exhausted from those issues to do another round of upgrades.
    – Taryn
    Mar 28, 2020 at 0:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .