Update: The July 8 maintenance is complete. The actual read-only duration was about 8 minutes. All systems are back to normal now.


MicroVersion:

Planned service degradation: All Stack Overflow/Stack Exchange sites will be read-only for 20 minutes on Sat, July 8, 2017 shortly after 10am US/Eastern (14:00 UTC) and again on Sat, July 22. If you blink, you'll miss it.

NOTE: The July 22 date may change (it could be earlier).

Announcement:

There will be a service degradation for up to 20 minutes shortly after 14:00 UTC (10AM US/Eastern) on Sat July 8, 2017. During that time job adverts will still display, job listings will still work, questions and answers will still display. However the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post/edit questions/comments/answers, upvotes/reputations won’t change, etc. We don't love having to do this, but it is unavoidable and will prevent larger problems in the future. This should not be very disruptive since the site will still work for the majority of our users. Users that are affected will see a warning banner.

Questions or Concerns?

Please post a comment or answer below. (You might want to review the questions we got the last time we did this.)

WHAT’S HAPPENING?

Background:

  • Our websites run on computers in a datacenter called NY (located in Jersey City, New Jersey...it used to be New York).
  • We have a secondary datacenter called CO (in Denver, Colorado).
  • CO is our “failover” location… only used if NY is having a problem (or tests, or maintenance).

So what's happening?

On July 8 we will start running all web services from CO. We will run out of CO for 2 weeks while disruptive maintenance is done in NY. On July 22 the process will be reversed and we will run out of NY as we normally do.

By disabling new content (or editing old content) during the change-over and change-back, the entire process becomes safer and quicker. We did a pre-test last May which went very smoothly.

WHY?

While the company “runs in CO”, we will do maintenance in NY. This includes upgrading the software on some network equipment and other hardware.

The more we do this, the better we get at it. Therefore, we try to do either a test or an actual failover once a year. We did a small test in May 2017, but the last big test (equivalent to what’s happening on July 8 and 22) hasn’t been done since August 2015.

Even if we didn’t have maintenance in NY, we’d be running out of CO anyway. The only way we know CO is ready to serve during an emergency is to run on CO when there isn’t an emergency. Compare it to a fire drill at a school. In some places fire drills are required monthly to make sure everyone knows what to do. If it is good enough for school children, it is good enough for data centers.

Each test/failover teaches us something. Stack is a company that is all about learning, so we actually look forward to this kind of thing. (Shameless self promotion: I mention learning from outages in a talk I gave). Just preparing for the this event teaches us a lot. For example, this time we forgot to post to Meta (which is why this post is just happening now). As a result, we now have a “pre-event communication checklist” which will help us do better next time. We are big fans of checklists as a way of improving operations.

WHAT’S NEXT?

Assuming everything goes will on Saturday, July 8, we’ll reverse the process on July 22, 2017. A similar service degradation will happen at that time, then we’ll be running out of NY as usual.

  • 4
    Obvious question: what happens when something happens in CO and you cannot failover to NY? – uɐɯsO uɐɥʇɐN Jul 7 '17 at 16:49
  • 17
    @uɐɯsOuɐɥʇɐN NY will be in a warm-standby state to fail back in the case of an emergency for almost the entire maintenance period. If something catastrophic goes wrong in those few hours when we're not in a standby state and we need to fail back, then I guess we're all just going to have a very bad day. – Mark Henderson Jul 7 '17 at 16:52
  • 3
    I like the optimism @MarkHenderson. :) – Andy Jul 7 '17 at 16:54
  • 4
    "If you blink, you'll miss it." I assume "blink" in this context means "20-minute power nap"? ;-P – ETHproductions Jul 7 '17 at 17:30
  • 6
    I got a strong sense of déjà vu from this one, even though the previous run is linked twice. Since other people with the same reaction will likely look in comments first (or at least, I did): yes, there was a very similar post back in May of this year. (No bashing intended, Tom, it's just that the links are nowhere where a skimming eye can find them.) – E.P. Jul 7 '17 at 17:33
  • 2
    Is chat affected as well? As in, read-only in the same time-frame? Will chat.meta.stackexchange.com be available all time? – rene Jul 7 '17 at 17:33
  • 1
    @Joe I believe it was in Spring 2014. We've had non-emergency needs, including 2 datacenter moves. – Tom Limoncelli Jul 7 '17 at 18:46
  • 1
    @rene unknown meta.stackexchange.com/a/295302/213575 – Braiam Jul 7 '17 at 20:06
  • 4
    I have an idea: you could rename the datacenter called "NY" to "NJ" while SO is running from CO. What could possibly go wrong? – Andrew Morton Jul 7 '17 at 20:57
  • 9
    @AndrewMorton I've got a list of things that'd break, it's quite impressive and almost certainly incomplete. – Nick Craver Jul 7 '17 at 23:53
  • 15
    Do you have any suggestion for what we should do during those 20 minutes? – m69 Jul 8 '17 at 3:32
  • 6
    Possible duplicate brief maintenances planned. ;) – Mithlesh Upadhyay Jul 8 '17 at 9:26
  • 1
    What about my Fanatic badge progress?? – Unitato Jul 8 '17 at 11:14
  • 14
    @m69 You should focus on questions that will take at least 20 minutes just to research the answer. – Tom Limoncelli Jul 8 '17 at 12:08
  • 5
    @uɐɯsOuɐɥʇɐN you need to print all your favorite questions and have screenshots of your reputation and top answers. Take them to the office of Stackoverflow. Have them verify/audit your points. Likely an auditing company like Deloitte will be contracted to overtake auditing. Wait till new site is up and have your points restored. Poor Jon Skeet would have to print a lot of pages. I hope someone creates a screenshot taking script so it would save us from our misery. :D – Honey Jul 8 '17 at 18:59

Can we have a message in chat when SE is read-only?

There was a "SO / SE is in read-only mode..." message in Stack Overflow, Documentation, other SE sites, Area 51, but not chat. You could still type in the box, but when you pressed enter, the text would disappear but no message would be shown. What about replacing the chat box with the message that SE is read-only then? Example:

We're sorry, but chat is in read-only mode right now. Follow @StackStatus or stackstatus.net for more details.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .