50

Many of the Stack Exchange sites have been going off and on such as Worldbuilding, Medical Sciences, Psychology & Neuroscience and Writing over the last couple of hours for me. The maintenance page shows, referring to the Twitter account and Status blogs.

Neither the Twitter account nor the Status blog show any indication of a current update.

Fortunately however, meta is fine ATM... Could we have an update as soon as convenient?

2 hours after posting, the sites are up, but slow to load. (Southern UK).

5 hours after posting, all sites are going up and down including meta.

Tweet by SO an hour before this edit:

We’re aware something is happening, and we’re working on it. We’ve tried unplugging and plugging back in, but that didn’t fix it.

Latest Update
16 hours after the post a (hopefully) final Tweet:

I think we discovered what was wrong. Things should be back up!

0

1 Answer 1

29

We were hit by a DDoS attack. While the attack itself was mitigated by our systems, the conditions set off a series of errors that managed to uncover an edge case in one of our backend systems.

It took several hours for us to identify the issue because the errors only manifested themselves once an hour. While we were investigating, the site was put into read only mode and ran out of our secondary data center. The irony is that the edge case that was discovered was only exposed because we have been improving the resilience of our DDoS mitigation.

1
  • 6
    Problem with edges is that there will always be more edges. But hey, at least this keep you busy, giving you challenges to solve, and sharpening your skills! ;) Mar 15 at 7:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .