Update: The July 8 maintenance is complete. The actual read-only duration was about 8 minutes. All systems are back to normal now.
Planned service degradation: All Stack Overflow/Stack Exchange sites will be read-only for 20 minutes on Sat, July 8, 2017 shortly after 10am US/Eastern (14:00 UTC) and again on Sat, July 22. If you blink, you'll miss it.
NOTE: The July 22 date may change (it could be earlier).
There will be a service degradation for up to 20 minutes shortly after 14:00 UTC (10AM US/Eastern) on Sat July 8, 2017. During that time job adverts will still display, job listings will still work, questions and answers will still display. However the site will be "read only," i.e. people won’t be able to add/edit new job listings, apply for jobs, post/edit questions/comments/answers, upvotes/reputations won’t change, etc. We don't love having to do this, but it is unavoidable and will prevent larger problems in the future. This should not be very disruptive since the site will still work for the majority of our users. Users that are affected will see a warning banner.
Questions or Concerns?
Please post a comment or answer below. (You might want to review the questions we got the last time we did this.)
- Our websites run on computers in a datacenter called NY (located in Jersey City, New Jersey...it used to be New York).
- We have a secondary datacenter called CO (in Denver, Colorado).
- CO is our “failover” location… only used if NY is having a problem (or tests, or maintenance).
So what's happening?
On July 8 we will start running all web services from CO. We will run out of CO for 2 weeks while disruptive maintenance is done in NY. On July 22 the process will be reversed and we will run out of NY as we normally do.
By disabling new content (or editing old content) during the change-over and change-back, the entire process becomes safer and quicker. We did a pre-test last May which went very smoothly.
While the company “runs in CO”, we will do maintenance in NY. This includes upgrading the software on some network equipment and other hardware.
The more we do this, the better we get at it. Therefore, we try to do either a test or an actual failover once a year. We did a small test in May 2017, but the last big test (equivalent to what’s happening on July 8 and 22) hasn’t been done since August 2015.
Even if we didn’t have maintenance in NY, we’d be running out of CO anyway. The only way we know CO is ready to serve during an emergency is to run on CO when there isn’t an emergency. Compare it to a fire drill at a school. In some places fire drills are required monthly to make sure everyone knows what to do. If it is good enough for school children, it is good enough for data centers.
Each test/failover teaches us something. Stack is a company that is all about learning, so we actually look forward to this kind of thing. (Shameless self promotion: I mention learning from outages in a talk I gave). Just preparing for the this event teaches us a lot. For example, this time we forgot to post to Meta (which is why this post is just happening now). As a result, we now have a “pre-event communication checklist” which will help us do better next time. We are big fans of checklists as a way of improving operations.
Assuming everything goes will on Saturday, July 8, we’ll reverse the process on July 22, 2017. A similar service degradation will happen at that time, then we’ll be running out of NY as usual.