40

Update: The maintenance went as planned.


tl;dr; Planned service interruption that will impact all Stack Exchange sites, Jobs, and Chat. All sites may be read-only for up to 15 minutes on Friday June 4, 2021 at 12:00am UTC (Thursday June 3rd 8 PM US/EDT). Stack Overflow might experience some instability during this time as well. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to 15 minutes 12:00am UTC on June 4th, 2021 (June 3rd 8:00 PM US/EDT). During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won't be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're 'read only' for maintenance.

Longer Version of What's Taking Place?

Background

Our primary database servers, which power Stack Overflow, Stack Exchange, Chat, and other things, run out of New York (really New Jersey) with our secondary location in Colorado. When we need to perform maintenance on the primary servers, we need to failover to another server so we can patch, upgrade, and reboot them.

What we'll be doing

During the service interruption, we'll be performing two failovers from the primary location to a secondary server in New York for the SQL Servers that support the Stack Exchange Network, jobs, and chat. We're doing multiple failovers to get some detailed information of failover performance for Microsoft.

We expect that the site will be in a read-only state for less than 15 minutes in total. Once we've done the failover, and everything seems to be up and running, we'll take the sites out of read-only.

Also during this time, we will be performing some much needed patching of Redis which could make Stack Overflow unstable for short periods of time.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

10
  • 8
    Can you please add the time in the 24 hour clock? Those of us who do not use AM/PM have quite a puzzle now. – Willeke May 27 at 20:44
  • 4
    @Willeke there is a link to the calendar/time in the post, and you can adjust it accordingly to a 24 hr clock and as well as your time zone. – Taryn ModStaff May 27 at 20:47
  • 1
    @Willeke 12am is 00:00. AM is 00:00-11:59 and PM is 12:00-23:59. For any programmers, to convert the time it's just H % 12 + (PM ? 12 : 0). – Pluto May 27 at 20:48
  • 3
    @Taryn I just tried the link and clicking the 24 hour button at the top right; it doesn't work on Chrome or Firefox... – Pluto May 27 at 20:54
  • 1
    @Taryn I can also confirm what Pluto mentioned for Firefox - on the linked page, clicking either "24" (24-hour format) or "MX" (mixed-mode, i.e. showing each location's time in its native/standard format) does nothing for me in Firefox - it seems to try to reload the page after I click, but then appears to still be in "am/pm" (12-hour format). – V2Blast May 28 at 22:01
  • 1
    @Willeke "...quite a puzzle"? Please tell me that's a joke. People who don't use 24 hour notation sometimes have trouble with 24 hour time but not the other way around. – Paul May 29 at 13:22
  • 6
    @PaulGifford, I do not have problems with most of the AM or PM times but 00:00 and 12:00 of the 24 hour clock confuse me. And I have noticed that this is so by many people. It is quite easy when you write a message to either include both AM and 24 hour time or chose a time which is not confusing, like even 0:01 AM. – Willeke May 29 at 14:45
  • 2
    Can the planned maintance activity be updated in the stackstatus.net too? – Arulkumar May 31 at 11:45
  • 4
    @Paul How could anyone have trouble figuring out a 24 hour clock? It's trivial. On the other hand whether 12AM or 12PM is midnight is a rather confusing affair all in all and I had Latin classes and know how the terminology came to be. – Voo Jun 3 at 19:06
  • @Paul, absolutely right. Well, actually, since 12 noon is neither 12 am nor 12 pm and 12 midnight is kind of both, I guess they're sort of right, albeit horribly inelegant. – Deirdre Hendrick Jun 4 at 0:27
30

Now you got me interested (not that I'm not interested or don't value your awesome work otherwise):

We're doing multiple failovers to get some detailed information of failover performance for Microsoft

Two questions for you:

  • What is special about your setup that Microsoft can't set up by themselves?
  • What is typical failover performance? Is that wall clock time or are we looking at IOPS or other measures?
4
  • 26
    Good questions. When we failover the Stack Exchange cluster (SE Network, Jobs, Chat) we have noticed a significant slowdown of the failover time since we upgraded to SQL Server 2019. We used to failover and be back up within a few minutes, now we're seeing CPU on the secondary (former primary) spike to 100% CPU for about 25 minutes. I suspect it's due to one Availability Group with 362 databases in it, but we're capturing some stats for Microsoft to help us debug it. The goal is to perform the failovers and send them the details for analysis. – Taryn ModStaff May 26 at 15:34
  • 4
    Ouch, depending on the setup downtime could be from a few to 30 seconds; over 20 minutes is 20 minutes too long: docs.microsoft.com/en-us/sql/database-engine/… – Rob May 27 at 1:49
  • 11
    @Rob it’s shouldn’t be that long as the CPU issue appears on the secondary servers so the primary should be able to receive traffic as expected. We previously captured data on March but need more details to hopefully identify what the issue is. – Taryn ModStaff May 27 at 2:36
  • 1
    Looks like as of half an hour ago it's still pinned at 100% on all cores. – Rob Jun 4 at 1:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .