UPDATE 2017-05-04: FYI: The planned outage last night went as planned. The user-facing "read only" period lasted about 12 minutes (from 00:01 UTC to 00:13 UTC). Thanks to everyone negatively affected. We appreciate your patience.
The good news is that the process had the desired result. It is better to find bugs in a controlled test than, say, during an outage. That's exactly what happened. We found some bugs (configuration issues and a few actual software issues) that would have been painful to fix during a real emergency. Instead, we can fix these issues as part of our normal software rollout process. In fact, many of the bugs have been fixed already.
It was also interesting to see so many interesting (and hilarious) comments. People raised interesting edge cases about reputation and other issues. Thanks to everyone that gave feedback! I learned a lot!
UPDATE: The test is complete. It started at 00:01 UTC and concluded at 00:13 UTC. Thanks!
MicroVersion: Planned service degradation: All Stack Overflow/Stack Exchange sites read-only for 20 minutes on Wed, May 3, 2017 shortly after 8PM US/Eastern (midnight UTC). If you blink, you'll miss it.
Short version:
There will be a service degradation for up to 20 minutes shortly after 8PM US/Eastern (midnight UTC) on Wed, May 3, 2017. During that time adverts will still display, listings will still work, questions and answers will still display. However the site will be "read only", i.e. people won’t be able to add/edit new listings, post/edit questions/comments/answers, etc. We don't love having to do this, but it is unavoidable and will prevent larger problems in the future. This test should not be very disruptive since the site will still work for the majority of our users. Users that are affected will see a warning banner.
Questions or concerns? Please post a comment or answer below.
IN LAYPERSON’S TERMS:
Background:
- Our websites run on computers in a datacenter called NY (located in Jersey City).
- We have a duplicate datacenter called CO (in Denver, CO).
- CO is our “failover” location… only used if NY is having a problem (and tests like this).
So what's happening?
During the service degradation, the websites will run from CO briefly and then switch back to NY. We will do this to discover any inconsistencies between NY and CO. We want to discover these issues in a safe and controlled setting, rather than during an emergency. By disabling new content (or editing old content) the entire process becomes safer and quicker.
Why?
We do these tests to make sure that we’re prepared for big emergencies.
We’d rather find problems now, when it is easy to fix things, than in the middle of a big emergency (like when Hurricane Sandy took down all of NY). So, like school kids doing fire drills to stay in practice, we do these tests occasionally to make sure we’re prepared for real emergencies. Sadly we haven't done a test in a year, which is bad.
Therefore we are doing a minor test on May 3 in preparation for a larger test this summer. Ironically the larger test will be nearly invisible to our users because of what we learn during this test.
Questions or concerns? Please post a comment or answer below.
IN TECHNICAL TERMS:
This is a "read only" failover to CO, followed by a switch back. In other words, CO is running in standby mode currently. Activate CO to make sure that it is properly configured.
The details:
- Start: Wed, May 3, 2017 shortly after 8pm EDT.
- End: 20 minutes of user-visible read-only status.
- Users affected: All services (Q&A, Talent, etc.) will be read-only for 20 minutes.
What we’ll be doing:
- Activating the CO datacenter as our read-only primary location.
- Testing services.
- Reverting back to NY.
What won’t be happening:
We won’t be shutting down anything in NY. It will stay running. This makes switching back easy.
What's happening this summer?
In June or July we'll operate out of CO for a week (maybe 2). There is some maintenance in NY we need to do that requires actual downtime. Once the maintenance is complete, we will switch back to NY. Any customer-visible outages will be extremely brief and will happen on the weekend so-as to minimize impact. Look for another post when we've set a date.
Questions or concerns? Please post a comment or answer below.