201

UPDATE 2017-05-04: FYI: The planned outage last night went as planned. The user-facing "read only" period lasted about 12 minutes (from 00:01 UTC to 00:13 UTC). Thanks to everyone negatively affected. We appreciate your patience.

The good news is that the process had the desired result. It is better to find bugs in a controlled test than, say, during an outage. That's exactly what happened. We found some bugs (configuration issues and a few actual software issues) that would have been painful to fix during a real emergency. Instead, we can fix these issues as part of our normal software rollout process. In fact, many of the bugs have been fixed already.

It was also interesting to see so many interesting (and hilarious) comments. People raised interesting edge cases about reputation and other issues. Thanks to everyone that gave feedback! I learned a lot!

UPDATE: The test is complete. It started at 00:01 UTC and concluded at 00:13 UTC. Thanks!

MicroVersion: Planned service degradation: All Stack Overflow/Stack Exchange sites read-only for 20 minutes on Wed, May 3, 2017 shortly after 8PM US/Eastern (midnight UTC). If you blink, you'll miss it.

Short version:

There will be a service degradation for up to 20 minutes shortly after 8PM US/Eastern (midnight UTC) on Wed, May 3, 2017. During that time adverts will still display, listings will still work, questions and answers will still display. However the site will be "read only", i.e. people won’t be able to add/edit new listings, post/edit questions/comments/answers, etc. We don't love having to do this, but it is unavoidable and will prevent larger problems in the future. This test should not be very disruptive since the site will still work for the majority of our users. Users that are affected will see a warning banner.

Questions or concerns? Please post a comment or answer below.

IN LAYPERSON’S TERMS:

Background:

  • Our websites run on computers in a datacenter called NY (located in Jersey City).
  • We have a duplicate datacenter called CO (in Denver, CO).
  • CO is our “failover” location… only used if NY is having a problem (and tests like this).

So what's happening?

During the service degradation, the websites will run from CO briefly and then switch back to NY. We will do this to discover any inconsistencies between NY and CO. We want to discover these issues in a safe and controlled setting, rather than during an emergency. By disabling new content (or editing old content) the entire process becomes safer and quicker.

Why?

We do these tests to make sure that we’re prepared for big emergencies.

We’d rather find problems now, when it is easy to fix things, than in the middle of a big emergency (like when Hurricane Sandy took down all of NY). So, like school kids doing fire drills to stay in practice, we do these tests occasionally to make sure we’re prepared for real emergencies. Sadly we haven't done a test in a year, which is bad.

Therefore we are doing a minor test on May 3 in preparation for a larger test this summer. Ironically the larger test will be nearly invisible to our users because of what we learn during this test.

Questions or concerns? Please post a comment or answer below.

IN TECHNICAL TERMS:

This is a "read only" failover to CO, followed by a switch back. In other words, CO is running in standby mode currently. Activate CO to make sure that it is properly configured.

The details:

What we’ll be doing:

  • Activating the CO datacenter as our read-only primary location.
  • Testing services.
  • Reverting back to NY.

What won’t be happening:

We won’t be shutting down anything in NY. It will stay running. This makes switching back easy.

What's happening this summer?

In June or July we'll operate out of CO for a week (maybe 2). There is some maintenance in NY we need to do that requires actual downtime. Once the maintenance is complete, we will switch back to NY. Any customer-visible outages will be extremely brief and will happen on the weekend so-as to minimize impact. Look for another post when we've set a date.

Questions or concerns? Please post a comment or answer below.

Improve this question
54
  • 64
    Editing the signature out
    – M.A.R.
    Apr 27, 2017 at 20:33
  • 70
    The layman's explanation seems more complicated than the technical explanation, lol.
    – Jason C
    Apr 27, 2017 at 20:49
  • 107
    Found it humorous that in the list of things that will still be displayed, you thought we'd like to see "adverts" up top.
    – Lightness Races in Orbit
    Apr 27, 2017 at 23:41
  • 27
    Also, 20 minutes is a long blink ;)
    – Lightness Races in Orbit
    Apr 27, 2017 at 23:41
  • 67
    Jokes aside, thank you for the advance notice. Seriously. :)
    – Lightness Races in Orbit
    Apr 27, 2017 at 23:42
  • 19
    Will chat be affected, or just Q&A?
    – user307833
    Apr 27, 2017 at 23:43
  • 32
    What happens in the summer when NY is down for maintenance and something goes wrong in CO? Everything dies? A magical unicorn explosion? Half Life 3 is released? :P
    – uɐɯsO uɐɥʇɐN
    Apr 28, 2017 at 0:05
  • 6
    @uɐɯsOuɐɥʇɐN: Half Exchange 3 confirmed. But seriously, that's presumably part of what they're testing: making sure CO can remain stable for significant lengths of time in primary mode. And there's only a week of vulnerability for this… ("only").
    – Nathan Tuggy
    Apr 28, 2017 at 3:21
  • 4
    Should have been planned for a day later: May the fourth be with you
    – TripeHound
    Apr 28, 2017 at 9:37
  • 19
    @uɐɯsOuɐɥʇɐN "What happens in the summer when NY is down for maintenance and something goes wrong in CO?" We would be down. The technical term for that is a "double fault". It is rare but scary. The SRE team would rush to do everything they could to bring NY back up. The reality is that the work we have planned only has a few brief full outages, and we would put them on hold if CO seemed unstable, had bad weather, etc. If we were a bank or other highly critical service we'd have a 3rd or 4th datacenter. (continued)
    – Tom Limoncelli StaffMod
    Apr 28, 2017 at 12:18
  • 14
    @uɐɯsOuɐɥʇɐN (continued) Having 1 failover location is called N+1 (N capacity to run the show, 1 spare). N+2 means 2 spares (N capacity to run the show, 1 spare for maintenance, 1 spare in case there is a failure during maintenance). Excuse the self-promotion, but Chapter 6 of the-cloud-book.com has a lot of info about that.
    – Tom Limoncelli StaffMod
    Apr 28, 2017 at 12:20
  • 9
    Love the transparency!
    – Daniel A. White
    Apr 28, 2017 at 12:53
  • 18
    Finally, I will have time for my Facebook, Instagram, Twitter and WhatsApp profiles.
    – user343622
    Apr 28, 2017 at 14:33
  • 6
    Just to be clear, when you say "Wednesday after midnight UTC", you actually mean "Wednesday EDT / Thursday UTC", correct?
    – mmyers
    Apr 28, 2017 at 19:23
  • 4
    That was the wildest 20 minutes of my life. So worth the hype. A++++, would read-only again!
    – Jason C
    May 4, 2017 at 0:22

7 Answers 7

Reset to default
121

Will close and delete votes, and more importantly, flags still work? If inappropriate content gets posted right before 0:00 UTC, it might stay there for the 20 minute process if flags and deleting options are disabled. How will this be handled?

Improve this answer
10
  • 60
    Nope, read-only really means read-only. No changes will be allowed. The database will be frozen in time. If a real problem occurs, we can directly access the master web tier/data center and take action.
    – Nick Craver
    Apr 27, 2017 at 22:59
  • 25
    Excellent question, BTW! I'll add "votes" to the template for next time.
    – Tom Limoncelli StaffMod
    Apr 27, 2017 at 23:00
  • 67
    This does not provide an answer to the question. If you have a new question, please ask it by clicking the Ask Question button. Include a link to this question if it helps provide context. Just kidding :-P
    – Rand al'Thor
    Apr 27, 2017 at 23:18
  • 129
    Good to know. I'll schedule my spam bots to do some extra posts at 23:59 UTC.
    – Suragch
    Apr 28, 2017 at 2:02
  • 2
    Just to confirm, @NickCraver; pending close/delete votes etc that haven't taken effect will still be there when the 20 minutes are over?
    – MTL
    Apr 28, 2017 at 4:28
  • 1
    @Suragch :-D Anyway, I'd be curious to know whether anyone will do it for real!
    – Fabio says Reinstate Monica
    Apr 28, 2017 at 9:41
  • 4
    @Suragch Smokey's average time to deletion is something like 30 seconds atm ;)
    – CalvT
    Apr 28, 2017 at 11:14
  • 1
    @CalvT 30 seconds of exposure on StackOverflow is like a year of exposure anywhere else.
    – Dan
    May 3, 2017 at 20:09
  • Actually @Dan no, because what spammers don't realise is SO adds rel="nofollow noreferrer" to all links so Google doesn't index them anyway - so any spam on SO is only visible to SO users, and that's only for a few seconds
    – CalvT
    May 3, 2017 at 23:03
  • @CalvT I won't disagree with you solely for the purpose of dissuading spammers. :)
    – Dan
    May 4, 2017 at 14:22
39

Will X be read-only'd?

  • Main sites - Entirely
  • Meta sites - Entirely
  • Meta Stack Exchange - Entirely
  • Area 51 - Unknown (but probably)
  • Stack Overflow Chat - Unknown
  • Meta Stack Exchange Chat - Unknown
  • Stack Exchange Chat - Unknown
  • Documentation - Entirely
  • Jobs - Entirely
  • Developer Story - Entirely
Improve this answer
5
  • 6
    They are talking about "All Stack Overflow/Stack Exchange sites", and Nick Craver said they'll freeze the database, so I think you can replace your list with "Everything - Entirely".
    – Fabio says Reinstate Monica
    Apr 28, 2017 at 9:33
  • 36
    Chat is a good question, we'll discuss. Docs, Jobs, Story: yes, readonly - they're all in the Stack Overflow and Careers databases.
    – Nick Craver
    Apr 28, 2017 at 10:28
  • 1
    Note that Mego originally asked about chat in this comment.
    – wizzwizz4
    Apr 28, 2017 at 17:06
  • 16
    Not being able to chat for twenty minutes would be too painful for me to bear. Please don't do this!
    – John Dvorak
    May 1, 2017 at 11:15
  • 10
    You mean for 20 minutes I have to get a life???
    – Y12K
    May 2, 2017 at 5:37
9

As databases are frozen at the time, how will the 5-minute grace period for some stuffs work (for posts posted right before the freezing operation)?
Thanks in advance.

Improve this answer
9
  • Let me clarify: the databases aren't frozen in time. The copy of the databases this read-only test is targeting effectively are. They are read-only replica via SQL Availability Groups. Anything successfully posted was posted to the master and will be replicated in a few seconds to the data center in question.
    – Nick Craver
    Apr 29, 2017 at 11:54
  • 1
    @NickCraver But my question is about the grace period.
    – iBug says Reinstate Monica
    Apr 29, 2017 at 12:23
  • 3
    @NickCraver Is there going to be a race condition for someone who submits content of some kind that doesn't get recorded? Or is it, at worst, it gets recorded to NY but is invisible for 20 minutes until the test is over.
    – corsiKa
    Apr 29, 2017 at 22:30
  • @iBug Can you point me to the grace period you're referring to? I honestly don't know what you're talking about...
    – Nick Craver
    Apr 30, 2017 at 15:49
  • @corsiKa There's no race in play, the replication from master to replicas will be active the entire time (always the case), any chance submitted should be reflected in the other data center usually within a second.
    – Nick Craver
    Apr 30, 2017 at 15:50
  • 8
    @NickCraver For example, a post edited within 5min after last edit will not be shown as another edition, a comment can only be edited within 5min after the original post, and so on.
    – iBug says Reinstate Monica
    May 1, 2017 at 0:00
  • 3
    Well, personally I expect we will simply lose it. Postponing the grace period during the read-only period would be useful only for people monitoring the read-only precise end. Most others will probably missed it and lose it anyway. So I would be surprised Stack Exchange staff would have deemed useful to implement such a postponing.
    – Frédéric
    May 1, 2017 at 11:41
  • 2
    If the read-only mode kicks in during that grace period, you won't be able to make any additional edits until we're out of read-only. As @Frédéric said, the grace period will not be extended.
    – Adam Lear StaffMod
    May 3, 2017 at 8:06
  • 3
    You overthink this. The grace period for posts is about not having too many revisions in general, having couple more revisions for 5 minutes is insignificant. The grace period for comments shall be about correcting typos, not about changing content; if you can't live with some typos, just delete your comment.
    – yo'
    May 4, 2017 at 13:40
7

Users that are affected will see a warning banner.

It'll be good if the banner has a link to this meta post. Otherwise there may be chances of bug reporting by people who didn't aware of this, by thinking that this happens only to them.

For a network with 100+ sites, 20 minutes is a large time span.

Improve this answer
4
  • 13
    Instead of linking to the post, our read-only banner points to our Twitter account. Because unlike this post frozen in read-only mode (or worse: offline for some reason), we can provide real-time updates. Our Twitter account is @StackStatus.
    – Nick Craver
    Apr 30, 2017 at 19:01
  • The intention of the above post is to Inform people this is not only for them and happening to all and it is a status bydesign and not a bug. Otherwise some may raise bugs who didn't aware of this. And Yes I agree with you. twitter is good. @NickCraver
    – Optimus Prime
    May 1, 2017 at 7:47
  • 3
    "Otherwise there may be chances of bug reporting by people" ... ehh, where would they do that if the site is read-only? Also, it is stated quite clearly when read-only mode is switched on.
    – wythagoras
    May 1, 2017 at 16:23
  • 1
    After the read only. Some may say just becore when I visited, I can't perform any action, I saw this message and etc.
    – Optimus Prime
    May 2, 2017 at 14:06
5

Will it affect the last seen?

As I am busy with my academics, I am not able to perform any action other than visiting the site every day for the fanatic badge. So, if I visited in this 20 min's, will it be counted and the last seen be updated?

One more

Area51 act as a standalone one in case of login and also the purpose is different. The post didnt mentioned about it. It says SO, all SE sites and talent. Will Area51 come under all SE sites?

Improve this answer
12
  • Area 51 is treated the same in reputation counting etc. Discuss Area 51 is similar to Meta Stack Exchange; it's a meta site that has its own reputation counter; as Meta Stack Exchange I see no reason that it shouldn't be read-only.
    – wizzwizz4
    Apr 28, 2017 at 17:03
  • But the purpose is entirely different and I think there are lots of difference in table structure of Area51 and other SE sites.
    – Optimus Prime
    Apr 28, 2017 at 17:24
  • 20
    Are you really limited to a 20 minute window in the day during which you are only visiting the site? Surely you can slip such a 10 second task in at any of several points during your day?
    – TylerH
    Apr 28, 2017 at 17:26
  • @SagarV It might be. I've added it to the "will x be read-only'd" table.
    – wizzwizz4
    Apr 28, 2017 at 17:27
  • 46
    A true fanatic visits the site more than once a day. Preferably once per hour (even while sleeping).
    – I say Reinstate Monica
    Apr 28, 2017 at 17:37
  • 12
    @Twisty Wait it's possible to close the tab?
    – user307833
    Apr 28, 2017 at 18:38
  • 13
    Unfortunately, yes. Last seen will not be tracked during this window.
    – Nick Craver
    Apr 28, 2017 at 21:07
  • @wizzwizz4 sounds to me like he's got a bot doing that at 12:01 each day :D
    – Blaine
    Apr 29, 2017 at 8:00
  • set up a cron job which runs curl with appropriate args? (p.s. this is not how I got that badge on SO)
    – Display Name
    Apr 29, 2017 at 8:15
  • 6
    @Twisty to further complete what you wrote ... and this on not just a single SE site ...! Moreover, if you're just visiting an SE site to get the "fanatic" badge, then you're missing the point of what using SE sites is all about
    – Pierre.Vriens
    Apr 29, 2017 at 9:18
  • 3
    Pierre, till before 20 days, I were active on SO for a minimum of 9 hours and I reached 60+ on the badge count. Now I have my exams and some academic works and I simply don't want to loose the day count. That is the only purpose of my visit.
    – Optimus Prime
    Apr 30, 2017 at 15:47
  • 1
    Yes, "last seen" storage will be affected during that time. Sort of. If I recall the code right (don't have it in front of me at the moment), we're also going to ignore your login cookie during this time, so we wouldn't even attempt to record a visit for your account.
    – Adam Lear StaffMod
    May 3, 2017 at 8:08
3

Apart from this challenging task described in your question, you may want to also consider these related topics:

  • using Monkeys and Gorillas.
  • rules about where to put the SE Eggs.

For way more details on this, refer to the answer to my DevOps.SE question "How to improve DRP-testing of SE sites?".

PS: if you did already consider these topics, it might be interesting to publish a similar question about them too.

Improve this answer
-4

Method of killing spam

For 30 minutes, 23:30 to 00:00 UTC, we should have a no-creation rule. That means no:

  • Posting questions and answers
  • Editing and suggesting edits
  • Chatting
  • Other changes that non-logged in users would see
  • Voting??

But these would still work:

  • Delete/close votes
  • Duphammers
  • Tag burnination
  • sudo rm -rf badPost/
  • Flags (all kinds)

That means we can deal with spam created at 23:29 for 30 minutes. And because a large amount of flaggers and close/delete-voters will be looking for new spam in that 30 min window, not much is likely to appear, so we have no spam!

What I mean is that we have an extra 30 minutes before 00:00 to remove spam while no new spam is created.

Improve this answer
15
  • 1
    Not sure what you try to say here.... it was made 10000% clear the sites will be read only during the outage. No posts, no flags, no close votes, no spam.
    – Shadow Wizard Love Zelda
    May 3, 2017 at 10:20
  • 3
    @ShadowWizard this is an attempt to solve meta.stackexchange.com/a/295290/205264.
    – Keelan
    May 3, 2017 at 10:21
  • 1
    @Keelan solve what? you mean this was meant to be a comment to that answer? Nick gave official answer to that question in a comment, really can't see anything that is missing.
    – Shadow Wizard Love Zelda
    May 3, 2017 at 10:22
  • @ShadowWizard solve the problem that spam can be added right before midnight, and stays there for 20 minutes. Nick's comment does not solve it, but acknowledges that it's there. I don't find it a big problem, just explaining why this answer is here.
    – Keelan
    May 3, 2017 at 10:24
  • @ShadowWizard What I mean is that we have an extra 30 minutes before 00:00 to remove spam while no new spam is created.
    – user338745
    May 3, 2017 at 10:29
  • 2
    So this deserves a separate discussion, I don't think this is fitting answer to the outage announcement.
    – Shadow Wizard Love Zelda
    May 3, 2017 at 10:38
  • @ShadowWizard should I make it a separate feature-request?
    – user338745
    May 3, 2017 at 10:39
  • Not sure feature request, more of a discussion, but yes, think better have it on its own
    – Shadow Wizard Love Zelda
    May 3, 2017 at 10:40
  • 1
    I don't think this is a good idea. You don't want to limit such big sites for a long period. SE team wants this to be opt out quickly. Since they can quickly delete spams internally, you won't see them on the site for long time (which I think will have a similar time period with delete/close votes, etc.).
    – alpakyol
    May 3, 2017 at 10:40
  • @user3280126 Maybe a 15-minute window instead?
    – user338745
    May 3, 2017 at 10:45
  • 1
    @programmer5000 I believe, they considered 20 mins as a best case so, other than that interval, more close to 0 minute, it's better. Spams can be deletable during the maintenance so, if I were them, I don't increase the degradation time.
    – alpakyol
    May 3, 2017 at 11:27
  • How would you burninate tags without editing? For non spam/abusive posts it's best to explain what's wrong with the post as you close/flag it and this wouldn't allow it. Also, voting to close as a duplicate always leaves a comment.
    – Laurel
    May 3, 2017 at 21:27
  • 2
    I can't even see most spam questions, they're deleted within a minute or 2, so maybe a 2 or 3 minute freeze on new Q's is all that's needed. Spammers are usually new users so can't edit without approval, so no need to stop edits IMO. As for answers... they'd be at the bottom of the list for established Q's anyway
    – Xen2050
    May 3, 2017 at 21:56
  • 4
    -1 because this invents a problem (supposed rush of spam just before the outage) that doesn't exist (historically this hasn't been true and even if a few posts were visible through the RO period it wouldn't be a big deal) and in the process creates a much bigger problem (longer outage) that effects more people. The less time in RO mode the better. If you're that worried about spam pay extra attention to add manual flags to anything smoke detector comes up with in the minutes leading up to the outage.
    – Caleb
    May 4, 2017 at 9:05
  • 2
    Even if some spam was not caught just before the outage, I think we can live with couple porn links on the site for 20 minutes.
    – yo'
    May 4, 2017 at 13:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .