33

I've been getting this on and off for what must be hours:

We are currently offline for maintenance

Routine maintenance usually takes less than an hour. If this turns into an extended outage, we will tweet updates from @StackStatus or post details on the status blog.

The @StackStatus twitter account eventually tweeted:

We were having an issue. It's much better now. We're monitoring closely. We will provide additional details on meta.

My friend also has the same problem, but I haven't seen anyone here mention it yet. What's happening? When can we expect it to stop doing this?

13
  • Could only reproduce on meta sites, a few hours ago Jan 29 at 3:54
  • 1
    Beat me to it, but I couldn't load meta to post. Incidentally the site was offline at 03.15 GMT ish yesterday also. Repro on worldbuilding, skeptics, biology, scifi-fantasy, medicalsciences, psychology and main meta. Jan 29 at 3:56
  • 1
    People have reported problems on Twitter. It seems to be working fine again. Jan 29 at 3:58
  • "We had another issue just now, but everything is up again. We're still monitoring. 4:20 am · 29 Jan 2022" - it's still ongoing. Jan 29 at 4:28
  • 33
    We are being hit by a large evolving DDoS attack, official answer coming Monday.
    – Josh Zhang StaffMod
    Jan 29 at 4:34
  • @SebastianSimon people mention midwest usa in the tweets, I am from Asia, no problem till now. Jan 29 at 6:10
  • @VScode_fanboy I'm in west-coast USA, and a few hours ago, the website was down and would sometimes load a little bit. It's back now, but I can confirm that there were issues in west USA.
    – cocomac
    Jan 29 at 6:18
  • 2
    @cocomac hmm, it seems it is globally spread, but affects people randomly. (Perfect example of a DDoS) Jan 29 at 6:24
  • @VScode_fanboy They also mention Australia and several other places in the world. Jan 29 at 6:28
  • @SebastianSimon Yes, that's what I meant, let's wait till the official answer comes? should we make a chat room for discussing our ideas and theories? because I sense that the comments like that will be here. Jan 29 at 6:30
  • @VScode_fanboy It'd best be on a separate server. Jan 29 at 7:37
  • 1
    @ARogueAnt. stin.to/o3k51 or us21.chatzy.com/62949396861134, what looks better? Jan 29 at 7:58
  • 3

1 Answer 1

40

On Friday the 28th around 23:30 UTC, Stack Overflow was hit by a DDoS attack. Unlike previous attacks this month, this one ramped up very quickly and brought SE Network sites down. Once alerted, the SRE Team began analyzing the nature of the attack and took steps to mitigate it.

We were able to stabilize site traffic the first time around 00:50 UTC. The attacker discovered the mitigations we put in place, adapted, and resumed the attack around 02:50 UTC. Due to the active and evolving nature of this attack, we took a more targeted approach to mitigating the attack and were able to stabilize site traffic again around 04:00 UTC.

You can see by the graph below, the attacker attempted to adapt again but was unsuccessful for the remainder of the weekend.

Bandwidth graph showing DDoS attack, with the bytes transferred showing as two blocks of activity and a final sharp peak as described previously.

11
  • 4
    Thank you and the team who responded, for all the hard work in keeping the site running. Do you have any details to share later down the line or further steps planned? Jan 31 at 15:32
  • 2
  • 4
    @AncientSwordRage, we are going through normal post mortem rituals after the incident but I'm not sure if there will be any more information to share in the future.
    – Josh Zhang StaffMod
    Jan 31 at 15:39
  • 2
    @Luuklag, we are unable to positively identify the attacker but Friday's attack and the one you linked are different in nature.
    – Josh Zhang StaffMod
    Jan 31 at 15:40
  • 30
    True story: responding to this DDoS interrupted my Friday DnD 5e campaign w/ friends. My level 5 Tiefling Warlock got put into NPC / follower mode for 90 mins while I jumped in and helped the team. DDoSers: so freaking inconsiderate of others' time :P.
    – Haney StaffMod
    Jan 31 at 15:43
  • What software do you use to get that graph? Jan 31 at 15:44
  • Good job, I only noticed a couple of 2-3 minute downtimes.
    – bad_coder
    Jan 31 at 15:46
  • 1
    @HenryWHHackv3.0b I'd guess grafana, though that's from messing with it for a future project that's on hold so I could be wrong
    – Journeyman Geek Mod
    Jan 31 at 15:48
  • 1
    @HenryWHHackv3.0b that specific graph is from our Fastly (CDN) dashboard. We also use SignalFX (Splunk Observability).
    – Josh Zhang StaffMod
    Jan 31 at 15:52
  • 1
    Are the DDoS attacks back again right now? I get “This service is unavailable.” a few times and if the pages load, then only slowly. Feb 4 at 13:11
  • 1
    @SebastianSimon based on the similar comment, I'd say that yes, looks like same attacker. Feb 7 at 7:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .