197

UPDATE 2017-05-04: FYI: The planned outage last night went as planned. The user-facing "read only" period lasted about 12 minutes (from 00:01 UTC to 00:13 UTC). Thanks to everyone negatively affected. We appreciate your patience.

The good news is that the process had the desired result. It is better to find bugs in a controlled test than, say, during an outage. That's exactly what happened. We found some bugs (configuration issues and a few actual software issues) that would have been painful to fix during a real emergency. Instead, we can fix these issues as part of our normal software rollout process. In fact, many of the bugs have been fixed already.

It was also interesting to see so many interesting (and hilarious) comments. People raised interesting edge cases about reputation and other issues. Thanks to everyone that gave feedback! I learned a lot!


UPDATE: The test is complete. It started at 00:01 UTC and concluded at 00:13 UTC. Thanks!


MicroVersion: Planned service degradation: All Stack Overflow/Stack Exchange sites read-only for 20 minutes on Wed, May 3, 2017 shortly after 8PM US/Eastern (midnight UTC). If you blink, you'll miss it.

Short version:

There will be a service degradation for up to 20 minutes shortly after 8PM US/Eastern (midnight UTC) on Wed, May 3, 2017. During that time adverts will still display, listings will still work, questions and answers will still display. However the site will be "read only", i.e. people won’t be able to add/edit new listings, post/edit questions/comments/answers, etc. We don't love having to do this, but it is unavoidable and will prevent larger problems in the future. This test should not be very disruptive since the site will still work for the majority of our users. Users that are affected will see a warning banner.

Questions or concerns? Please post a comment or answer below.

IN LAYPERSON’S TERMS:

Background:

  • Our websites run on computers in a datacenter called NY (located in Jersey City).
  • We have a duplicate datacenter called CO (in Denver, CO).
  • CO is our “failover” location… only used if NY is having a problem (and tests like this).

So what's happening?

During the service degradation, the websites will run from CO briefly and then switch back to NY. We will do this to discover any inconsistencies between NY and CO. We want to discover these issues in a safe and controlled setting, rather than during an emergency. By disabling new content (or editing old content) the entire process becomes safer and quicker.

Why?

We do these tests to make sure that we’re prepared for big emergencies.

We’d rather find problems now, when it is easy to fix things, than in the middle of a big emergency (like when Hurricane Sandy took down all of NY). So, like school kids doing fire drills to stay in practice, we do these tests occasionally to make sure we’re prepared for real emergencies. Sadly we haven't done a test in a year, which is bad.

Therefore we are doing a minor test on May 3 in preparation for a larger test this summer. Ironically the larger test will be nearly invisible to our users because of what we learn during this test.

Questions or concerns? Please post a comment or answer below.

IN TECHNICAL TERMS:

This is a "read only" failover to CO, followed by a switch back. In other words, CO is running in standby mode currently. Activate CO to make sure that it is properly configured.

The details:

What we’ll be doing:

  • Activating the CO datacenter as our read-only primary location.
  • Testing services.
  • Reverting back to NY.

What won’t be happening:

We won’t be shutting down anything in NY. It will stay running. This makes switching back easy.

What's happening this summer?

In June or July we'll operate out of CO for a week (maybe 2). There is some maintenance in NY we need to do that requires actual downtime. Once the maintenance is complete, we will switch back to NY. Any customer-visible outages will be extremely brief and will happen on the weekend so-as to minimize impact. Look for another post when we've set a date.

Questions or concerns? Please post a comment or answer below.

  • 62
    Editing the signature out – Marshmallow Apr 27 '17 at 20:33
  • 68
    The layman's explanation seems more complicated than the technical explanation, lol. – Jason C Apr 27 '17 at 20:49
  • 106
    Found it humorous that in the list of things that will still be displayed, you thought we'd like to see "adverts" up top. – Lightness Races in Orbit Apr 27 '17 at 23:41
  • 27
    Also, 20 minutes is a long blink ;) – Lightness Races in Orbit Apr 27 '17 at 23:41
  • 66
    Jokes aside, thank you for the advance notice. Seriously. :) – Lightness Races in Orbit Apr 27 '17 at 23:42
  • 18
    Will chat be affected, or just Q&A? – Mego Apr 27 '17 at 23:43
  • 31
    What happens in the summer when NY is down for maintenance and something goes wrong in CO? Everything dies? A magical unicorn explosion? Half Life 3 is released? :P – uɐɯsO uɐɥʇɐN Apr 28 '17 at 0:05
  • 6
    @uɐɯsOuɐɥʇɐN: Half Exchange 3 confirmed. But seriously, that's presumably part of what they're testing: making sure CO can remain stable for significant lengths of time in primary mode. And there's only a week of vulnerability for this… ("only"). – Nathan Tuggy Apr 28 '17 at 3:21
  • 4
    Should have been planned for a day later: May the fourth be with you – TripeHound Apr 28 '17 at 9:37
  • 19
    @uɐɯsOuɐɥʇɐN "What happens in the summer when NY is down for maintenance and something goes wrong in CO?" We would be down. The technical term for that is a "double fault". It is rare but scary. The SRE team would rush to do everything they could to bring NY back up. The reality is that the work we have planned only has a few brief full outages, and we would put them on hold if CO seemed unstable, had bad weather, etc. If we were a bank or other highly critical service we'd have a 3rd or 4th datacenter. (continued) – Tom Limoncelli Apr 28 '17 at 12:18
  • 14
    @uɐɯsOuɐɥʇɐN (continued) Having 1 failover location is called N+1 (N capacity to run the show, 1 spare). N+2 means 2 spares (N capacity to run the show, 1 spare for maintenance, 1 spare in case there is a failure during maintenance). Excuse the self-promotion, but Chapter 6 of the-cloud-book.com has a lot of info about that. – Tom Limoncelli Apr 28 '17 at 12:20
  • 9
    Love the transparency! – Daniel A. White Apr 28 '17 at 12:53
  • 18
    Finally, I will have time for my Facebook, Instagram, Twitter and WhatsApp profiles. – user343622 Apr 28 '17 at 14:33
  • 6
    Just to be clear, when you say "Wednesday after midnight UTC", you actually mean "Wednesday EDT / Thursday UTC", correct? – mmyers Apr 28 '17 at 19:23
  • 4
    That was the wildest 20 minutes of my life. So worth the hype. A++++, would read-only again! – Jason C May 4 '17 at 0:22
121

Will close and delete votes, and more importantly, flags still work? If inappropriate content gets posted right before 0:00 UTC, it might stay there for the 20 minute process if flags and deleting options are disabled. How will this be handled?

  • 59
    Nope, read-only really means read-only. No changes will be allowed. The database will be frozen in time. If a real problem occurs, we can directly access the master web tier/data center and take action. – Nick Craver Apr 27 '17 at 22:59
  • 25
    Excellent question, BTW! I'll add "votes" to the template for next time. – Tom Limoncelli Apr 27 '17 at 23:00
  • 67
    This does not provide an answer to the question. If you have a new question, please ask it by clicking the Ask Question button. Include a link to this question if it helps provide context. Just kidding :-P – Rand al'Thor Apr 27 '17 at 23:18
  • 126
    Good to know. I'll schedule my spam bots to do some extra posts at 23:59 UTC. – Suragch Apr 28 '17 at 2:02
  • 2
    Just to confirm, @NickCraver; pending close/delete votes etc that haven't taken effect will still be there when the 20 minutes are over? – Shokhet Apr 28 '17 at 4:28
  • 1
    @Suragch :-D Anyway, I'd be curious to know whether anyone will do it for real! – Fabio Turati Apr 28 '17 at 9:41
  • 3
    @Suragch Smokey's average time to deletion is something like 30 seconds atm ;) – CalvT Apr 28 '17 at 11:14
  • 1
    @CalvT 30 seconds of exposure on StackOverflow is like a year of exposure anywhere else. – Dan May 3 '17 at 20:09
  • Actually @Dan no, because what spammers don't realise is SO adds rel="nofollow noreferrer" to all links so Google doesn't index them anyway - so any spam on SO is only visible to SO users, and that's only for a few seconds – CalvT May 3 '17 at 23:03
  • @CalvT I won't disagree with you solely for the purpose of dissuading spammers. :) – Dan May 4 '17 at 14:22
38

Will X be read-only'd?

  • Main sites - Entirely
  • Meta sites - Entirely
  • Meta Stack Exchange - Entirely
  • Area 51 - Unknown (but probably)
  • Stack Overflow Chat - Unknown
  • Meta Stack Exchange Chat - Unknown
  • Stack Exchange Chat - Unknown
  • Documentation - Entirely
  • Jobs - Entirely
  • Developer Story - Entirely
  • 6
    They are talking about "All Stack Overflow/Stack Exchange sites", and Nick Craver said they'll freeze the database, so I think you can replace your list with "Everything - Entirely". – Fabio Turati Apr 28 '17 at 9:33
  • 35
    Chat is a good question, we'll discuss. Docs, Jobs, Story: yes, readonly - they're all in the Stack Overflow and Careers databases. – Nick Craver Apr 28 '17 at 10:28
  • 1
    Note that Mego originally asked about chat in this comment. – wizzwizz4 Apr 28 '17 at 17:06
  • 15
    Not being able to chat for twenty minutes would be too painful for me to bear. Please don't do this! – John Dvorak May 1 '17 at 11:15
  • 9
    You mean for 20 minutes I have to get a life??? – Y12K May 2 '17 at 5:37
7

As databases are frozen at the time, how will the 5-minute grace period for some stuffs work (for posts posted right before the freezing operation)?
Thanks in advance.

  • Let me clarify: the databases aren't frozen in time. The copy of the databases this read-only test is targeting effectively are. They are read-only replica via SQL Availability Groups. Anything successfully posted was posted to the master and will be replicated in a few seconds to the data center in question. – Nick Craver Apr 29 '17 at 11:54
  • 1
    @NickCraver But my question is about the grace period. – iBug Apr 29 '17 at 12:23
  • 3
    @NickCraver Is there going to be a race condition for someone who submits content of some kind that doesn't get recorded? Or is it, at worst, it gets recorded to NY but is invisible for 20 minutes until the test is over. – corsiKa Apr 29 '17 at 22:30
  • @iBug Can you point me to the grace period you're referring to? I honestly don't know what you're talking about... – Nick Craver Apr 30 '17 at 15:49
  • @corsiKa There's no race in play, the replication from master to replicas will be active the entire time (always the case), any chance submitted should be reflected in the other data center usually within a second. – Nick Craver Apr 30 '17 at 15:50
  • 7
    @NickCraver For example, a post edited within 5min after last edit will not be shown as another edition, a comment can only be edited within 5min after the original post, and so on. – iBug May 1 '17 at 0:00
  • 3
    Well, personally I expect we will simply lose it. Postponing the grace period during the read-only period would be useful only for people monitoring the read-only precise end. Most others will probably missed it and lose it anyway. So I would be surprised Stack Exchange staff would have deemed useful to implement such a postponing. – Frédéric May 1 '17 at 11:41
  • 1
    If the read-only mode kicks in during that grace period, you won't be able to make any additional edits until we're out of read-only. As @Frédéric said, the grace period will not be extended. – Adam Lear May 3 '17 at 8:06
  • 2
    You overthink this. The grace period for posts is about not having too many revisions in general, having couple more revisions for 5 minutes is insignificant. The grace period for comments shall be about correcting typos, not about changing content; if you can't live with some typos, just delete your comment. – yo' May 4 '17 at 13:40
6

Users that are affected will see a warning banner.

It'll be good if the banner has a link to this meta post. Otherwise there may be chances of bug reporting by people who didn't aware of this, by thinking that this happens only to them.

For a network with 100+ sites, 20 minutes is a large time span.

  • 12
    Instead of linking to the post, our read-only banner points to our Twitter account. Because unlike this post frozen in read-only mode (or worse: offline for some reason), we can provide real-time updates. Our Twitter account is @StackStatus. – Nick Craver Apr 30 '17 at 19:01
  • The intention of the above post is to Inform people this is not only for them and happening to all and it is a status bydesign and not a bug. Otherwise some may raise bugs who didn't aware of this. And Yes I agree with you. twitter is good. @NickCraver – Optimus Prime May 1 '17 at 7:47
  • 3
    "Otherwise there may be chances of bug reporting by people" ... ehh, where would they do that if the site is read-only? Also, it is stated quite clearly when read-only mode is switched on. – wythagoras May 1 '17 at 16:23
  • 1
    After the read only. Some may say just becore when I visited, I can't perform any action, I saw this message and etc. – Optimus Prime May 2 '17 at 14:06
5

Will it affect the last seen?

As I am busy with my academics, I am not able to perform any action other than visiting the site every day for the fanatic badge. So, if I visited in this 20 min's, will it be counted and the last seen be updated?

One more

Area51 act as a standalone one in case of login and also the purpose is different. The post didnt mentioned about it. It says SO, all SE sites and talent. Will Area51 come under all SE sites?

  • Area 51 is treated the same in reputation counting etc. Discuss Area 51 is similar to Meta Stack Exchange; it's a meta site that has its own reputation counter; as Meta Stack Exchange I see no reason that it shouldn't be read-only. – wizzwizz4 Apr 28 '17 at 17:03
  • But the purpose is entirely different and I think there are lots of difference in table structure of Area51 and other SE sites. – Optimus Prime Apr 28 '17 at 17:24
  • 19
    Are you really limited to a 20 minute window in the day during which you are only visiting the site? Surely you can slip such a 10 second task in at any of several points during your day? – TylerH Apr 28 '17 at 17:26
  • @SagarV It might be. I've added it to the "will x be read-only'd" table. – wizzwizz4 Apr 28 '17 at 17:27
  • 46
    A true fanatic visits the site more than once a day. Preferably once per hour (even while sleeping). – Twisty Impersonator Apr 28 '17 at 17:37
  • 12
    @Twisty Wait it's possible to close the tab? – Mego Apr 28 '17 at 18:38
  • 13
    Unfortunately, yes. Last seen will not be tracked during this window. – Nick Craver Apr 28 '17 at 21:07
  • @wizzwizz4 sounds to me like he's got a bot doing that at 12:01 each day :D – Blaine Apr 29 '17 at 8:00
  • set up a cron job which runs curl with appropriate args? (p.s. this is not how I got that badge on SO) – Sarge Borsch Apr 29 '17 at 8:15
  • 6
    @Twisty to further complete what you wrote ... and this on not just a single SE site ...! Moreover, if you're just visiting an SE site to get the "fanatic" badge, then you're missing the point of what using SE sites is all about – Pierre.Vriens Apr 29 '17 at 9:18
  • 3
    Pierre, till before 20 days, I were active on SO for a minimum of 9 hours and I reached 60+ on the badge count. Now I have my exams and some academic works and I simply don't want to loose the day count. That is the only purpose of my visit. – Optimus Prime Apr 30 '17 at 15:47
  • 1
    Yes, "last seen" storage will be affected during that time. Sort of. If I recall the code right (don't have it in front of me at the moment), we're also going to ignore your login cookie during this time, so we wouldn't even attempt to record a visit for your account. – Adam Lear May 3 '17 at 8:08
2

Apart from this challenging task described in your question, you may want to also consider these related topics:

  • using Monkeys and Gorillas.
  • rules about where to put the SE Eggs.

For way more details on this, refer to the answer to my DevOps.SE question "How to improve DRP-testing of SE sites?".

PS: if you did already consider these topics, it might be interesting to publish a similar question about them too.

-5

Method of killing spam

For 30 minutes, 23:30 to 00:00 UTC, we should have a no-creation rule. That means no:

  • Posting questions and answers
  • Editing and suggesting edits
  • Chatting
  • Other changes that non-logged in users would see
  • Voting??

But these would still work:

  • Delete/close votes
  • Duphammers
  • Tag burnination
  • sudo rm -rf badPost/
  • Flags (all kinds)

That means we can deal with spam created at 23:29 for 30 minutes. And because a large amount of flaggers and close/delete-voters will be looking for new spam in that 30 min window, not much is likely to appear, so we have no spam!

What I mean is that we have an extra 30 minutes before 00:00 to remove spam while no new spam is created.

  • Not sure what you try to say here.... it was made 10000% clear the sites will be read only during the outage. No posts, no flags, no close votes, no spam. – Shadow The Curly Braced Wizard May 3 '17 at 10:20
  • 3
    @ShadowWizard this is an attempt to solve meta.stackexchange.com/a/295290/205264. – Keelan May 3 '17 at 10:21
  • 1
    @Keelan solve what? you mean this was meant to be a comment to that answer? Nick gave official answer to that question in a comment, really can't see anything that is missing. – Shadow The Curly Braced Wizard May 3 '17 at 10:22
  • @ShadowWizard solve the problem that spam can be added right before midnight, and stays there for 20 minutes. Nick's comment does not solve it, but acknowledges that it's there. I don't find it a big problem, just explaining why this answer is here. – Keelan May 3 '17 at 10:24
  • @ShadowWizard What I mean is that we have an extra 30 minutes before 00:00 to remove spam while no new spam is created. – programmer5000 May 3 '17 at 10:29
  • 2
    So this deserves a separate discussion, I don't think this is fitting answer to the outage announcement. – Shadow The Curly Braced Wizard May 3 '17 at 10:38
  • @ShadowWizard should I make it a separate feature-request? – programmer5000 May 3 '17 at 10:39
  • Not sure feature request, more of a discussion, but yes, think better have it on its own – Shadow The Curly Braced Wizard May 3 '17 at 10:40
  • 1
    I don't think this is a good idea. You don't want to limit such big sites for a long period. SE team wants this to be opt out quickly. Since they can quickly delete spams internally, you won't see them on the site for long time (which I think will have a similar time period with delete/close votes, etc.). – alpakyol May 3 '17 at 10:40
  • @user3280126 Maybe a 15-minute window instead? – programmer5000 May 3 '17 at 10:45
  • @programmer5000 I believe, they considered 20 mins as a best case so, other than that interval, more close to 0 minute, it's better. Spams can be deletable during the maintenance so, if I were them, I don't increase the degradation time. – alpakyol May 3 '17 at 11:27
  • How would you burninate tags without editing? For non spam/abusive posts it's best to explain what's wrong with the post as you close/flag it and this wouldn't allow it. Also, voting to close as a duplicate always leaves a comment. – Laurel May 3 '17 at 21:27
  • 1
    I can't even see most spam questions, they're deleted within a minute or 2, so maybe a 2 or 3 minute freeze on new Q's is all that's needed. Spammers are usually new users so can't edit without approval, so no need to stop edits IMO. As for answers... they'd be at the bottom of the list for established Q's anyway – Xen2050 May 3 '17 at 21:56
  • 3
    -1 because this invents a problem (supposed rush of spam just before the outage) that doesn't exist (historically this hasn't been true and even if a few posts were visible through the RO period it wouldn't be a big deal) and in the process creates a much bigger problem (longer outage) that effects more people. The less time in RO mode the better. If you're that worried about spam pay extra attention to add manual flags to anything smoke detector comes up with in the minutes leading up to the outage. – Caleb May 4 '17 at 9:05
  • 1
    Even if some spam was not caught just before the outage, I think we can live with couple porn links on the site for 20 minutes. – yo' May 4 '17 at 13:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .