34

Update: The maintenance went as planned.


tl;dr; Planned service interruption that will impact all Stack Exchange sites, Jobs, and Chat. All sites may be read-only for up to 60 minutes on Thursday, July 29, 2021 at 12:00am (midnight) UTC (Wednesday July 28 8PM US/EDT). Stack Overflow might experience some instability during this time as well. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to 60 minutes 12:00am UTC (midnight) on July 29th, 2021 (Wednesday July 28th 8:00 PM US/EDT). During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're 'read only' for maintenance.

Longer Version of What's Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.

What we'll be doing

During the service interruption, we'll be performing two failovers from the primary location to a secondary server in New York for the SQL Servers that support the Stack Exchange Network, jobs, and chat. We're doing multiple failovers to get some detailed information of failover performance for Microsoft.

We expect that the site will be in a read-only state for less than 60 minutes in total. Once we've done the failover, and everything seems to be up and running, we'll take the sites out of read-only.

We'll be doing some behind the scenes work to hopefully minimize the impact to Stack Overflow during the failovers.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

14
  • 7
    Do you expect there are juicy details in this ongoing investigation that is worth sharing once it is all settled? if so, can we hope for a blog post from you on these matters?
    – rene
    Jul 20 at 16:04
  • 7
    Also: isn't planning maintenance just now a bit cheating so you could post to get up the leader of the Community-a-thon? ;)
    – rene
    Jul 20 at 16:06
  • 11
    @rene I sure hope so. Here are some basic details so far, we've done multiple captures at the SQL Server level and haven't found any smoking guns to the issue, so this time they are having use get captures using the WPR on both the primary and secondary servers that we are failing over. This get details on CPU across the entire server not just SQL, so fingers crossed we'll get more info on what is resulting in the slow failovers.
    – Taryn StaffMod
    Jul 20 at 16:07
  • 3
    Good luck. Having to work weekends is never fun, I hope this run will collect the data MS needs to address the issue; and that the next time you have to do this it'll be to test how well the fix works. Jul 20 at 18:52
  • 1
    Is noon in Europe really the best time? Would (say) 6pm EDT = 11pm BST = midnight CET = 3:30am Sunday New Delhi = 6am Sunday HK = 8am Sunday in Sydney not be better? Jul 21 at 15:21
  • 2
    @AndrewLeach It's the weekend with lower traffic so the time is just fine.
    – Taryn StaffMod
    Jul 21 at 15:22
  • I'm glad that changing the time to the lowest-traffic point in the weekend would be better :-) Perhaps not this time round since everything is set; but surely for other occasions. Jul 21 at 15:24
  • 6
    Y'all seem to be having a terrible week, hopefully the random wierdness like this doesn't keep up
    – Journeyman Geek Mod
    Jul 24 at 15:43
  • @Andrew Leach: How can you conclude that? BTW, in the original version of this question it was planned for Saturday, July 24, 2021 at 12:00 pm UTC (8:00 AM US/EDT). Jul 27 at 9:02
  • @P.Mort My first comment here was made on the basis of noon UTC, but 6pm EDT would put most of the world in twilight or darkness, with most of the US being sometime Saturday afternoon. Is 6pm EDT really not lower traffic overall, worldwide? Jul 27 at 10:24
  • 3
    @AndrewLeach there is never an ideal time to perform maintenance but we attempt to select times when there will be lower traffic and provide notice of it to hopefully minimize impact to all users
    – Taryn StaffMod
    Jul 27 at 10:49
  • Is everyday the maintenance will be scheduled for another date. It was July 24 then July 28 then July 29 then what? Jul 27 at 22:14
  • 5
    @KevinM.Mansour we’ve had some unexpected issues which resulted in it being postponed.
    – Taryn StaffMod
    Jul 27 at 22:14
  • 2
    Good luck fixing those @Taryn
    – Luuklag
    Jul 27 at 22:35

You must log in to answer this question.

Browse other questions tagged .