29

tl;dr; Planned service interruption that will impact all Stack Exchange sites, Jobs, and Chat. All sites may be read-only for up to 60 minutes on Wednesday, July 28, 2021 at 12:00am (midnight) UTC (Tuesday July 27 8PM US/EDT). Stack Overflow might experience some instability during this time as well. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to 60 minutes 12:00am UTC (midnight) on July 28th, 2021 (Tuesday July 27th 8:00 PM US/EDT). During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're 'read only' for maintenance.

Longer Version of What's Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.

What we'll be doing

During the service interruption, we'll be performing two failovers from the primary location to a secondary server in New York for the SQL Servers that support the Stack Exchange Network, jobs, and chat. We're doing multiple failovers to get some detailed information of failover performance for Microsoft.

We expect that the site will be in a read-only state for less than 60 minutes in total. Once we've done the failover, and everything seems to be up and running, we'll take the sites out of read-only.

We'll be doing some behind the scenes work to hopefully minimize the impact to Stack Overflow during the failovers.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

8
  • 7
    Do you expect there are juicy details in this ongoing investigation that is worth sharing once it is all settled? if so, can we hope for a blog post from you on these matters? – rene Jul 20 at 16:04
  • 7
    Also: isn't planning maintenance just now a bit cheating so you could post to get up the leader of the Community-a-thon? ;) – rene Jul 20 at 16:06
  • 11
    @rene I sure hope so. Here are some basic details so far, we've done multiple captures at the SQL Server level and haven't found any smoking guns to the issue, so this time they are having use get captures using the WPR on both the primary and secondary servers that we are failing over. This get details on CPU across the entire server not just SQL, so fingers crossed we'll get more info on what is resulting in the slow failovers. – Taryn Jul 20 at 16:07
  • 3
    Good luck. Having to work weekends is never fun, I hope this run will collect the data MS needs to address the issue; and that the next time you have to do this it'll be to test how well the fix works. – Dan Is Fiddling By Firelight Jul 20 at 18:52
  • Is noon in Europe really the best time? Would (say) 6pm EDT = 11pm BST = midnight CET = 3:30am Sunday New Delhi = 6am Sunday HK = 8am Sunday in Sydney not be better? – Andrew Leach Jul 21 at 15:21
  • 1
    @AndrewLeach It's the weekend with lower traffic so the time is just fine. – Taryn Jul 21 at 15:22
  • I'm glad that changing the time to the lowest-traffic point in the weekend would be better :-) Perhaps not this time round since everything is set; but surely for other occasions. – Andrew Leach Jul 21 at 15:24
  • 2
    Y'all seem to be having a terrible week, hopefully the random wierdness like this doesn't keep up – Journeyman Geek 2 days ago

You must log in to answer this question.

Browse other questions tagged .