38

Update: The maintenance went as planned.

tl;dr; Planned service interruption that will impact all Stack Exchange sites, Jobs, and Chat. All sites may be read-only for up to 15 minutes on Friday June 4, 2021 at 12:00am UTC (Thursday June 3rd 8 PM US/EDT). Stack Overflow might experience some instability during this time as well. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to 15 minutes 12:00am UTC on June 4th, 2021 (June 3rd 8:00 PM US/EDT). During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're 'read only' for maintenance.

Longer Version of What's Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.

What we'll be doing

During the service interruption, we'll be performing two failovers from the primary location to a secondary server in New York for the SQL Servers that support the Stack Exchange Network, jobs, and chat. We're doing multiple failovers to get some detailed information of failover performance for Microsoft.

We expect that the site will be in a read-only state for less than 15 minutes in total. Once we've done the failover, and everything seems to be up and running, we'll take the sites out of read-only.

Also during this time, we will be performing some much needed patching of Redis which could make Stack Overflow unstable for short periods of time.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

Improve this question
10
  • 8
    Can you please add the time in the 24 hour clock? Those of us who do not use AM/PM have quite a puzzle now.
    – Willeke
    May 27, 2021 at 20:44
  • 4
    @Willeke there is a link to the calendar/time in the post, and you can adjust it accordingly to a 24 hr clock and as well as your time zone.
    – Taryn
    May 27, 2021 at 20:47
  • 1
    @Willeke 12am is 00:00. AM is 00:00-11:59 and PM is 12:00-23:59. For any programmers, to convert the time it's just H % 12 + (PM ? 12 : 0).
    – Pluto
    May 27, 2021 at 20:48
  • 3
    @Taryn I just tried the link and clicking the 24 hour button at the top right; it doesn't work on Chrome or Firefox...
    – Pluto
    May 27, 2021 at 20:54
  • 1
    @Taryn I can also confirm what Pluto mentioned for Firefox - on the linked page, clicking either "24" (24-hour format) or "MX" (mixed-mode, i.e. showing each location's time in its native/standard format) does nothing for me in Firefox - it seems to try to reload the page after I click, but then appears to still be in "am/pm" (12-hour format).
    – V2Blast StaffMod
    May 28, 2021 at 22:01
  • 1
    @Willeke "...quite a puzzle"? Please tell me that's a joke. People who don't use 24 hour notation sometimes have trouble with 24 hour time but not the other way around.
    – Paul
    May 29, 2021 at 13:22
  • 6
    @PaulGifford, I do not have problems with most of the AM or PM times but 00:00 and 12:00 of the 24 hour clock confuse me. And I have noticed that this is so by many people. It is quite easy when you write a message to either include both AM and 24 hour time or chose a time which is not confusing, like even 0:01 AM.
    – Willeke
    May 29, 2021 at 14:45
  • 2
    Can the planned maintance activity be updated in the stackstatus.net too?
    – Arulkumar
    May 31, 2021 at 11:45
  • 4
    @Paul How could anyone have trouble figuring out a 24 hour clock? It's trivial. On the other hand whether 12AM or 12PM is midnight is a rather confusing affair all in all and I had Latin classes and know how the terminology came to be.
    – Voo
    Jun 3, 2021 at 19:06
  • @Paul, absolutely right. Well, actually, since 12 noon is neither 12 am nor 12 pm and 12 midnight is kind of both, I guess they're sort of right, albeit horribly inelegant.
    – Deirdre Hendrick
    Jun 4, 2021 at 0:27

1 Answer 1

Reset to default
29

Now you got me interested (not that I'm not interested or don't value your awesome work otherwise):

We're doing multiple failovers to get some detailed information of failover performance for Microsoft

Two questions for you:

  • What is special about your setup that Microsoft can't set up by themselves?
  • What is typical failover performance? Is that wall clock time or are we looking at IOPS or other measures?
Improve this answer
4
  • 26
    Good questions. When we failover the Stack Exchange cluster (SE Network, Jobs, Chat) we have noticed a significant slowdown of the failover time since we upgraded to SQL Server 2019. We used to failover and be back up within a few minutes, now we're seeing CPU on the secondary (former primary) spike to 100% CPU for about 25 minutes. I suspect it's due to one Availability Group with 362 databases in it, but we're capturing some stats for Microsoft to help us debug it. The goal is to perform the failovers and send them the details for analysis.
    – Taryn
    May 26, 2021 at 15:34
  • 4
    Ouch, depending on the setup downtime could be from a few to 30 seconds; over 20 minutes is 20 minutes too long: docs.microsoft.com/en-us/sql/database-engine/…
    – Rob
    May 27, 2021 at 1:49
  • 11
    @Rob it’s shouldn’t be that long as the CPU issue appears on the secondary servers so the primary should be able to receive traffic as expected. We previously captured data on March but need more details to hopefully identify what the issue is.
    – Taryn
    May 27, 2021 at 2:36
  • 1
    Looks like as of half an hour ago it's still pinned at 100% on all cores.
    – Rob
    Jun 4, 2021 at 1:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .