9

Today, the entire stack exchange network was in read-only mode for several hours right in the middle of normal office hours for EU. This was pretty annoying, because it meant that I couldn't ask for advice on certain subjects.

Was this just regular maintenance? if so, I can't remember it being announced in advance.

  • 3
    read this - stackstatus.net/post/85902805739/… – Taryn May 16 '14 at 13:59
  • 18
    Re downvotes; I note that the stackstatus isn't linked to from the site when in downtime so its not unreasonable to be curious – Richard Tingle May 16 '14 at 14:01
  • 3
    What @RichardTingle said. This is a perfectly legitimate support question, and the answer is only obvious if you already know it. – yannis May 16 '14 at 14:03
  • 5
    Right, but a simple search would have given information about the StackStatus Twitter link. So this question shows "no research effort". – bluemax May 16 '14 at 14:19
  • 2
    Someone done an UPSie at the hosting provider. twitter.com/nick_craver/status/467240946372202496 – Martijn Pieters May 16 '14 at 14:23
  • 8
    I think that stackstatus.net should be linked in the "the page is read only" text, explaining that more information might appear there later. – yo' May 16 '14 at 14:35
  • 2
    Also note that Stackexchange websites are owned and hosted in the US and this issue came in the middle of the night there. They acted quickly IMHO. – Ninsuo May 16 '14 at 15:13
  • 1
    As people correctly deduced, I was unaware of the existence of the stackstatus.net website. If I knew about this website, I wouldn't have asked this question. I agree that it should be publicized more clearly when this happens again. – Nzall May 17 '14 at 20:45
13

Stack Exchange Primary Data Center Offline - May 16th, 2014:

Our data center racks in New York unexpectedly lost connectivity. There was a failure in the UPS feeding our equipment that went undetected by our provider. Here is a timeline of major events so far:

  • 06:43 UTC - The UPS feeding our racks in the New York data center stopped feeding utility power and failed to provider backup power
  • 07:04 UTC - We learned the nature of the UPS issue and began failover to our Oregon facility
  • 07:07 UTC - Oregon came online in read-only mode
  • 07:10-09:00 UTC - We experienced issues with stability in Oregon due to the expected failover with caching and database connections. We will detail these issues in the postmortem.
  • 09:50 UTC - New York data center regains power
  • 09:54 UTC - Load balancers were reactivated and New York is again reachable.
  • 09:56 UTC - SQL Databases are online and restoring databases
  • 10:00 UTC - SQL Databases came back online without any apparent damage (phew)
  • Sounds like a hectic morning for the SE Staff. – Amicable May 16 '14 at 15:46
  • @Amicable for some staff members, it was the middle of the night. – Steven V May 16 '14 at 15:56
  • 06:43 UTC is like 01:43 in New York. – ypercubeᵀᴹ May 17 '14 at 0:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .