121

Update: (2020-12-09 UTC) Window C is complete. No outages or degradations.

Update: (2020-12-04 UTC) Window B is complete. No outages or degradations.

Update: (2020-12-02 UTC) Window A is complete. No outages or degradations.

TL;DR: The SRE Team at Stack Overflow is announcing 3 maintenance windows. No downtime is planned but the work is classified as “risky” therefore this announcement. The maintenance is expected to take 10-20 minutes and will happen sometime during the 2-hour window beginning at 8:30PM ET (01:30 UTC the next day).

Window UTC ET (New York) PT (US/West) Potentially Affected Platforms
Window A Dec 2, 0130 Tue, Dec 1, 8:30PM Tue, Dec 1, 5:30PM Public SEDE, possibly others
Window B Dec 4, 0130 Thu, Dec 3, 8:30PM Thu, Dec 3, 5:30PM Public Q&A, Chat, Talent, Teams, possibly others
Window C Dec 9, 0130 Tue, Dec 8, 8:30PM Tue, Dec 8, 5:30PM Public SEDE, possibly others

The details: We will be reconfiguring some network hardware in attempt to diagnose and fix a performance issue. This should not cause a disruption, but lab tests have found evidence contrary to vendor pronouncements. In the lab we were able to remediate any problems in 10-30 minutes.

If there is an outage, we appreciate your patience.

Questions or concerns? Please post a comment or answer below.

17
  • 3
    Window A and C is for maintenance in Denver, CO and Window B in New Jersey (the NY datacenter)?
    – rene
    Nov 24 '20 at 12:21
  • 16
    Very close! A=CO, B=NJ, C=the link between them.
    – Tom Limoncelli StaffMod
    Nov 24 '20 at 17:15
  • 6
    Which vendor are you referring to?
    – Travis J
    Nov 24 '20 at 18:52
  • 69
    How did we use to live without tables? Nov 25 '20 at 0:19
  • 26
    It is so very refreshing and reassuring to see this evidence that the SRE team at Stack Overflow conducts lab testing instead of just poking at production. And furthermore, that you even ran simulations in the lab environment to make sure that you could address any problems in 10-30 minutes. Keep up the amazing work!
    – Cody Gray
    Nov 25 '20 at 0:43
  • 47
    The moment you realize that SE engineers can't just lookup common solutions to issues like everyone else when their live deployment goes downhill
    – dube
    Nov 25 '20 at 8:24
  • 17
    @FranckDernoncourt We sat on the floor to eat dinner. Nov 25 '20 at 18:25
  • 2
    @TravisJ Which vendor? I'll never reveal :-)
    – Tom Limoncelli StaffMod
    Nov 25 '20 at 19:12
  • 4
    @fez Yep, it's a new feature: meta.stackexchange.com/questions/356997/… Nov 25 '20 at 23:33
  • 5
    In plain English, what's a "Public SEDE"?
    – martineau
    Nov 25 '20 at 23:55
  • 6
    @martineau SEDE stands for "Stack Exchange Data Explorer". It's a service hosted at data.stackexchange.com that allows anyone to write queries against sanitized versions of our Q&A databases.
    – Adam Lear StaffMod
    Nov 26 '20 at 0:49
  • 48
    When I read the title I thought "Dec 2/4/9" was a new crazy way for Americans to format a datestamp.
    – pipe
    Nov 26 '20 at 3:45
  • 5
    @pipe: Haha, I'm American and it still didn't jump out at me as a list of 3 separate dates rather than a singular (weird) M/D/Y format until I looked at the dates on the table and then back at the title again.
    – V2Blast StaffMod
    Nov 27 '20 at 8:16
  • 4
    As a site used worldwide, thank you very much for describing the downtime window in UTC time first. All of us can easily (almost automatically) work out the offset between our own local time and UTC, whereas when times are parochially given only in a US timezone (helping only 300M of the world's population), or indeed any other local timezone, as some organisations do, it takes rather more thought to work out how that relates to our own local time. (Could you maybe change "early morning" to "01:30" as the sidebar truncates the title so it ends with "8:30PM" which is therefore very confusing!)
    – dave559
    Nov 27 '20 at 16:38
  • 6
    For transparency on the work progress, I recommend having a live graph of the coffee consumption on your team. ;-)
    – Brad
    Nov 27 '20 at 22:23
57

Will there be a maintenance banner during this period in case users experience some difficulty? This heads up is great, but it may not be broadcast to all users across the network before the window and having a reminder does not hurt.

4
  • 13
    Good thinking! We hadn't planned on it. We'll learn a lot in Window A and that will inform our decisions about Window B.
    – Tom Limoncelli StaffMod
    Nov 25 '20 at 15:01
  • 8
    @TomLimoncelli: Thanks. A simple banner message warning of possible downtime shortly before and during the window of time (at least for the time when Public Q&A might be affected) and directing people to this Q&A would probably be a good idea.
    – V2Blast StaffMod
    Nov 26 '20 at 1:24
  • 1
    Agreed. From 5 minutes up front till the end of the window a small banner would probably save us a lot of meta questions and concerned users if something goes wrong.
    – Mast
    Nov 27 '20 at 10:53
  • @Dom The banner is reserved for podcasts news only! :)
    – dwirony
    Dec 7 '20 at 21:38
24

Thanks for the warning. It's certainly not necessary for you to tell us. But thanks for doing so anyway.

Let's all hope it goes well and that there are no problems!

3
  • 39
    SE usually warns us in advance for maintenance, as far as I noticed, for several years now, so nothing extraordinary. However, I would rather say it is, to some extent, necessary, in order to keep trust / respect from the community, on which all is built upon.
    – Pac0
    Nov 24 '20 at 22:02
  • 18
    It is necessary. The world has come to expect large organisations like SE to keep their Internet services available all the time. Downtime, without warning, comes across as evidence of incompetence or an attack. On a more general note, announcements for any major changes are necessary for community trust. Nov 25 '20 at 10:19
  • 21
    I try to over-communicate about this kind of thing.
    – Tom Limoncelli StaffMod
    Nov 25 '20 at 14:55
14

Along the lines of the other questions:

Will there be some sort of live notification? Not everyone is going to keep track of meta, which I suppose is their own fault.

In other words, when you're typing a comment or an answer, the system will warn you that someone has posted something new and give you the chance to expand it and read the comment/answer.

Will there be something along those lines, just in case we're mid-answer? Maybe something saying, "The live site is currently unavailable due to maintenance. Submitting your question/comment now will not work."?

My main concern is the potential people who have started typing out a response before the site goes down, if the site goes down, may end up having their bits lost to the ether if they press the enter button or submit button while the site is down.

2
  • 8
    Good question! When the site goes into read-only mode (which happens as a precaution automatically for some errors, manually for others) it does have many warnings like that. We could do more, but it is a matter of prioritizing.
    – Tom Limoncelli StaffMod
    Nov 25 '20 at 15:00
  • 1
    @TomLimoncelli Thanks for the response. I know that I really dislike replicating work, so the warnings will be appreciated, certainly by more folks than just myself.
    – KGIII
    Nov 25 '20 at 16:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .