66

Update: The maintenance went as planned.

tl;dr; Planned service interruption that will impact all Stack Overflow/Stack Exchange sites. All sites may be read-only for up to 15 minutes on Wednesday September 11, 2019 at 1:00 UTC. Enterprise cloud hosted instances will not be impacted.

Short Version:

There will be a service degradation for up to 15 minutes 1:00 UTC (9 PM US/Eastern) on September 11th, 2019. During that time questions and answers will still display, job listings will still work, and job ads will still display. However, the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post, edit or vote on questions/comments/answers, reputation won't change, etc. This should minimize the disruption to the majority of casual readers. We will display a banner on the sites stating we're 'read only' for maintenance.

Longer Version of What's Taking Place?

Background

We have two datacenters, with our main datacenter in New York (actually New Jersey) and the secondary in Colorado. When we need to perform maintenance on the database servers that power Stack Overflow, Stack Exchange, Chat, and other things, we need to failover to another location - either a local secondary (i.e. one in New York) or a remote secondary (i.e. in Colorado). The failover allows us to patch the server, and reboot them.

What we'll be doing

During the service interruption, we'll be performing a failover from the primary location to a secondary server in New York. This allows us to patch the former primary SQL Server and apply Windows updates to the server. By putting the sites in a read-only state, we reduce the chance of data loss and the entire process becomes safer.

We expect that the site will be in a read-only state for less than 15 minutes. Once we've done the failover, and everything seems to be up and running, we'll take the sites out of read-only.

Questions or concerns?

Please post a comment or answer below; I'll do my best to address any concerns between now and the maintenance window.

Improve this question
18
  • 2
    Chat will be blib only?
    – rene
    Sep 5, 2019 at 18:37
  • 5
    @rene Chat should blip only when we do the failover of that AG
    – Taryn
    Sep 5, 2019 at 18:38
  • And SEDE is running in CO so I assume that is unaffected?
    – rene
    Sep 5, 2019 at 18:42
  • 7
    @rene SEDE is a completely separate server and will not be impacted by this work.
    – Taryn
    Sep 5, 2019 at 18:43
  • 20
    Why are you guys still running Windows?
    – user474678
    Sep 5, 2019 at 20:14
  • 116
    @JL2210 We like to play games on the servers on the weekends and we couldn't get the video or sound drivers to work under Linux.
    – Nick Craver
    Sep 5, 2019 at 20:30
  • 2
    @Nick Craver: Will the port to .NET Core enable running on Linux?
    – This_is_NOT_a_forum
    Sep 6, 2019 at 8:14
  • 7
    We can wait for 15 mins for SO who saved us hours of coding time..
    – Krishnadas PC
    Sep 6, 2019 at 11:24
  • 1
    "We have two datacenters, with our main datacenter in New York (actually New Jersey)". So why are you saying New York? Just say New Jersey. we won't hold it against you.
    – j08691
    Sep 6, 2019 at 20:34
  • 2
    @j08691 internally we still say NY. All of our servers in that datacenter start with NY. Too much work to rename everything
    – Taryn
    Sep 6, 2019 at 20:42
  • 1
    @Olivia That was addressed in above comments. Chat will be down for 15 minutes (blip only).
    – user474678
    Sep 7, 2019 at 21:07
  • 2
    @JL2210 What is a chat "blip"?
    – Lawrence
    Sep 8, 2019 at 12:40
  • 3
    @Lawrence Chat will suffer a quick outage while we failover the availability group it’s in. Should be very quick
    – Taryn
    Sep 8, 2019 at 12:45
  • 1
    @Olivia See above comment.
    – user474678
    Sep 8, 2019 at 13:11
  • 1
    My edit was to address confusion from some users (including Americans) who only read the first few words in the Community Bulletin network-wide and wonder why the site is down on September 10 while the sidebar said September 11. Why was that rolled back?
    – Sonic the Anonymous Hedgehog
    Sep 10, 2019 at 22:34

3 Answers 3

Reset to default
43

Thanks for the notice. I'm surprised it's only 15 minutes. Can you really fail over, do Windows and SQL Server updates, and be back up and running in 15 minutes? (My Windows updates never go that smoothly...) Or are you changing which of your NY servers is primary -- failing over to the current secondary, making sure it comes up ok, and then enabling writes and making that the new primary? Or will there be a second brief outage as you fail over back to the primary?

These answers don't affect me; I'm just curious how you manage all this.

Improve this answer
3
  • 29
    Nope, we definitely can't do all that in 15 minutes. The timeframe covers the failover piece. Our setup has 3 servers in a cluster (2 in NY, 1 in CO) which allows me to patch CO and the secondary in NY before we do the failover making those up to date on patches. When we perform the failover the NY servers switch positions - the current secondary becomes primary - while the primary moves to the secondary position which allows me to finish patching the old primary without disruption. Make sense?
    – Taryn
    Sep 5, 2019 at 21:12
  • 19
    @Taryn so the end result is that the NY primary and secondary trade places, presumably to trade back the next time you have to do this? Makes sense, yes (and, given the 15-minute window, what I was guessing). Does all this mean that in your three-server cluster, only one is active at a time? (But kept in sync so if the active one fails you can switch to a secondary.) Is CO the "NY power outage" contingency? (Y'all have probably written stuff about this already; feel free to point me at it. And yeah, I'm just curious/nosy.)
    – Monica Cellio
    Sep 5, 2019 at 21:17
  • 22
    Yes, the NY servers trade places until the next time we do this. And since we use SQL Servers Always On Availability Groups - we serve data from both NY servers - the primary gets read/write and the secondary gets read-only to offload more expensive stuff that doesn't involve writes. CO is present for redundancy in the event NY goes completely offline. No worries on the questions, I know people are curious about how we do this.
    – Taryn
    Sep 5, 2019 at 21:21
7

What is the end goal of the maintenance? That is, when everything comes back up, what will be different? Will there be new features? Will the site be faster? More resistant to spam? Is there a component that is showing signs of imminent failure and needs immediate repair? Are you just going to defrag and install OS updates? In other words, what would happen if this maintenance didn't happen?

Improve this answer
4
  • 25
    We're patching SQL Server and applying Windows Updates to the current primary servers in all of our clusters. We can't do those things without performing a failover. The goal of the maintenance is to patch and apply fixes to some bugs within SQL Server. I try to apply patching quarterly to our SQL Servers and it's time for them to be done.
    – Taryn
    Sep 5, 2019 at 18:45
  • 6
    Its simple. MORE COWBELL
    – Journeyman Geek Mod
    Sep 6, 2019 at 13:03
  • 1
    @Robert Patching your OS fixes security flaws, and other bugs. You should do it often.
    – Lightness Races in Orbit
    Sep 10, 2019 at 1:06
  • @JourneymanGeek infinitely preferable to MORE COBOL
    – mbrig
    Sep 10, 2019 at 21:12
7

Thank you for letting us know. This should let people plan for the scheduled maintenance and adjust their schedules accordingly.

Thank you also for letting us know why. A lot of the confusion on the Stack Exchange network now is why things are done. Explaining the situation helps us understand the reasoning.

On a side note, why are you still running Windows? Linux allows live patching (patching a running kernel) and atomic upgrades. This allows software upgrades without the need to reboot.

Improve this answer
13
  • 23
    Regardless of whether it's Windows or Linux - the critical piece is we need to patch SQL Server. We cannot patch a primary SQL Server and have it stay up, we need to perform a failover to the secondary to patch what was the former primary. Patching of SQL Server stops the service which would take us down, so we perform a failover to prevent a full outage.
    – Taryn
    Sep 5, 2019 at 20:20
  • 6
    Well SE runs on .net and mssql. From what I've caught from Twitter, job postings and petty dabbling, while works ongoing to move stuff to newer, cross platform .net and mssql... I've not heard of any big sites running .net and mssql on Linux. The current setup has worked reliably over the past decade or so. At some point in the future it might be an option but apparently not yet
    – Journeyman Geek Mod
    Sep 8, 2019 at 1:27
  • 4
    This question is pretty much brought up everytime there is a planned maintenance announcement. See Which tools and technologies are used to build the Stack Exchange Network?.
    – Marc.2377
    Sep 8, 2019 at 7:17
  • @JourneymanGeek SQL Server and .net core might work, if I'm not misunderstanding your comment. But I have an aversion to Windows because it destroyed all my photos in an update, so...
    – user474678
    Sep 8, 2019 at 13:15
  • 7
    @JL2210 might work isn't good enough. This has worked reliably and consistently. Its what Jeff and Joel and Sam and all the rest of the early SO folks used. Porting stuff to .net core is currently in progress from what I can tell from public sources but that has to get done, components tested and so on. There seem to be eventual plans to run some products like SO enterprise on linux (on kubernetes?) but that's somewhat in the future.
    – Journeyman Geek Mod
    Sep 8, 2019 at 13:47
  • 2
    Also, as a community mod, and mock-chicken entrail reader, please take anything I say as a pinch of salt. All this is derived from many publicly available sources - like twitter and job ads
    – Journeyman Geek Mod
    Sep 8, 2019 at 13:48
  • @JourneymanGeek Maybe someone could pull up enough resources sometime to see if this could work. I'd really be happy if there could be a Stack Exchange network running on Linux.
    – user474678
    Sep 8, 2019 at 13:50
  • 3
    @JL2210 FTR patching a live kernel is still risky and concern only security patches (as far as I know Kpatch and livepatch, I may be outdated). Upgrading a kernel still require to reboot, as upgrading a Postgres or MySQL instance would require to restart the service and a DB failover somewhere in the path.
    – Tensibai
    Sep 9, 2019 at 15:30
  • @Tensibai Atomic upgrades mainly concern userspace.
    – user474678
    Sep 9, 2019 at 16:03
  • 4
    Anyway, patching (delivering a new binary) needs reloading this binary, in memory patching works for small changes only and is not a reliable method. I did it for a few systems where we had only a yearly window for complete patch, and I've never been confident and never saw a full year where all gone well 100%. 15 mins fail over in read only (not shut down) is fairly good whatever the system are in my experience, pushing Linux for that point is not really relevant IMHO.
    – Tensibai
    Sep 9, 2019 at 20:44
  • @Tensibai Yeah. However, it was just a side note. I even tried to remove it after the fact, but it was added back.
    – user474678
    Sep 9, 2019 at 21:10
  • 2
    @JourneymanGeek "This has worked reliably and consistently" Well... ;)
    – Lightness Races in Orbit
    Sep 10, 2019 at 1:08
  • 1
    @JL2210 Yep, sorry, I couldn't resist to say explicitly using Linux wouldn't fix everything :)
    – Tensibai
    Sep 10, 2019 at 7:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .