119

Update: (2020-12-09 UTC) Window C is complete. No outages or degradations.

Update: (2020-12-04 UTC) Window B is complete. No outages or degradations.

Update: (2020-12-02 UTC) Window A is complete. No outages or degradations.

TL;DR: The SRE Team at Stack Overflow is announcing 3 maintenance windows. No downtime is planned but the work is classified as “risky” therefore this announcement. The maintenance is expected to take 10-20 minutes and will happen sometime during the 2-hour window beginning at 8:30PM ET (01:30 UTC the next day).

Window UTC ET (New York) PT (US/West) Potentially Affected Platforms
Window A Dec 2, 0130 Tue, Dec 1, 8:30PM Tue, Dec 1, 5:30PM Public SEDE, possibly others
Window B Dec 4, 0130 Thu, Dec 3, 8:30PM Thu, Dec 3, 5:30PM Public Q&A, Chat, Talent, Teams, possibly others
Window C Dec 9, 0130 Tue, Dec 8, 8:30PM Tue, Dec 8, 5:30PM Public SEDE, possibly others

The details: We will be reconfiguring some network hardware in attempt to diagnose and fix a performance issue. This should not cause a disruption, but lab tests have found evidence contrary to vendor pronouncements. In the lab we were able to remediate any problems in 10-30 minutes.

If there is an outage, we appreciate your patience.

Questions or concerns? Please post a comment or answer below.

Improve this question
17
  • 3
    Window A and C is for maintenance in Denver, CO and Window B in New Jersey (the NY datacenter)?
    – rene Mod
    Commented Nov 24, 2020 at 12:21
  • 16
    Very close! A=CO, B=NJ, C=the link between them.
    – Tom Limoncelli StaffMod
    Commented Nov 24, 2020 at 17:15
  • 6
    Which vendor are you referring to?
    – Travis J
    Commented Nov 24, 2020 at 18:52
  • 69
    How did we use to live without tables?
    – Franck Dernoncourt
    Commented Nov 25, 2020 at 0:19
  • 26
    It is so very refreshing and reassuring to see this evidence that the SRE team at Stack Overflow conducts lab testing instead of just poking at production. And furthermore, that you even ran simulations in the lab environment to make sure that you could address any problems in 10-30 minutes. Keep up the amazing work!
    – Cody Gray
    Commented Nov 25, 2020 at 0:43
  • 47
    The moment you realize that SE engineers can't just lookup common solutions to issues like everyone else when their live deployment goes downhill
    – dube
    Commented Nov 25, 2020 at 8:24
  • 17
    @FranckDernoncourt We sat on the floor to eat dinner.
    – Dawood ibn Kareem
    Commented Nov 25, 2020 at 18:25
  • 2
    @TravisJ Which vendor? I'll never reveal :-)
    – Tom Limoncelli StaffMod
    Commented Nov 25, 2020 at 19:12
  • 4
    @fez Yep, it's a new feature: meta.stackexchange.com/questions/356997/…
    – Sonic the Anonymous Hedgehog
    Commented Nov 25, 2020 at 23:33
  • 5
    In plain English, what's a "Public SEDE"?
    – martineau
    Commented Nov 25, 2020 at 23:55
  • 6
    @martineau SEDE stands for "Stack Exchange Data Explorer". It's a service hosted at data.stackexchange.com that allows anyone to write queries against sanitized versions of our Q&A databases.
    – Adam Lear StaffMod
    Commented Nov 26, 2020 at 0:49
  • 48
    When I read the title I thought "Dec 2/4/9" was a new crazy way for Americans to format a datestamp.
    – pipe
    Commented Nov 26, 2020 at 3:45
  • 5
    @pipe: Haha, I'm American and it still didn't jump out at me as a list of 3 separate dates rather than a singular (weird) M/D/Y format until I looked at the dates on the table and then back at the title again.
    – V2Blast
    Commented Nov 27, 2020 at 8:16
  • 4
    As a site used worldwide, thank you very much for describing the downtime window in UTC time first. All of us can easily (almost automatically) work out the offset between our own local time and UTC, whereas when times are parochially given only in a US timezone (helping only 300M of the world's population), or indeed any other local timezone, as some organisations do, it takes rather more thought to work out how that relates to our own local time. (Could you maybe change "early morning" to "01:30" as the sidebar truncates the title so it ends with "8:30PM" which is therefore very confusing!)
    – dave559
    Commented Nov 27, 2020 at 16:38
  • 6
    For transparency on the work progress, I recommend having a live graph of the coffee consumption on your team. ;-)
    – Brad
    Commented Nov 27, 2020 at 22:23

3 Answers 3

Reset to default
57

Will there be a maintenance banner during this period in case users experience some difficulty? This heads up is great, but it may not be broadcast to all users across the network before the window and having a reminder does not hurt.

Improve this answer
4
  • 13
    Good thinking! We hadn't planned on it. We'll learn a lot in Window A and that will inform our decisions about Window B.
    – Tom Limoncelli StaffMod
    Commented Nov 25, 2020 at 15:01
  • 8
    @TomLimoncelli: Thanks. A simple banner message warning of possible downtime shortly before and during the window of time (at least for the time when Public Q&A might be affected) and directing people to this Q&A would probably be a good idea.
    – V2Blast
    Commented Nov 26, 2020 at 1:24
  • 1
    Agreed. From 5 minutes up front till the end of the window a small banner would probably save us a lot of meta questions and concerned users if something goes wrong.
    – Mast
    Commented Nov 27, 2020 at 10:53
  • @Dom The banner is reserved for podcasts news only! :)
    – dwirony
    Commented Dec 7, 2020 at 21:38
24

Thanks for the warning. It's certainly not necessary for you to tell us. But thanks for doing so anyway.

Let's all hope it goes well and that there are no problems!

Improve this answer
3
  • 39
    SE usually warns us in advance for maintenance, as far as I noticed, for several years now, so nothing extraordinary. However, I would rather say it is, to some extent, necessary, in order to keep trust / respect from the community, on which all is built upon.
    – Pac0
    Commented Nov 24, 2020 at 22:02
  • 18
    It is necessary. The world has come to expect large organisations like SE to keep their Internet services available all the time. Downtime, without warning, comes across as evidence of incompetence or an attack. On a more general note, announcements for any major changes are necessary for community trust.
    – Brian Drake
    Commented Nov 25, 2020 at 10:19
  • 21
    I try to over-communicate about this kind of thing.
    – Tom Limoncelli StaffMod
    Commented Nov 25, 2020 at 14:55
14

Along the lines of the other questions:

Will there be some sort of live notification? Not everyone is going to keep track of meta, which I suppose is their own fault.

In other words, when you're typing a comment or an answer, the system will warn you that someone has posted something new and give you the chance to expand it and read the comment/answer.

Will there be something along those lines, just in case we're mid-answer? Maybe something saying, "The live site is currently unavailable due to maintenance. Submitting your question/comment now will not work."?

My main concern is the potential people who have started typing out a response before the site goes down, if the site goes down, may end up having their bits lost to the ether if they press the enter button or submit button while the site is down.

Improve this answer
2
  • 8
    Good question! When the site goes into read-only mode (which happens as a precaution automatically for some errors, manually for others) it does have many warnings like that. We could do more, but it is a matter of prioritizing.
    – Tom Limoncelli StaffMod
    Commented Nov 25, 2020 at 15:00
  • 1
    @TomLimoncelli Thanks for the response. I know that I really dislike replicating work, so the warnings will be appreciated, certainly by more folks than just myself.
    – KGIII
    Commented Nov 25, 2020 at 16:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .