33

I've been getting this on and off for what must be hours:

We are currently offline for maintenance

Routine maintenance usually takes less than an hour. If this turns into an extended outage, we will tweet updates from @StackStatus or post details on the status blog.

The @StackStatus twitter account eventually tweeted:

We were having an issue. It's much better now. We're monitoring closely. We will provide additional details on meta.

My friend also has the same problem, but I haven't seen anyone here mention it yet. What's happening? When can we expect it to stop doing this?

Improve this question
13
  • Could only reproduce on meta sites, a few hours ago
    – Sonic the Anonymous Hedgehog
    Commented Jan 29, 2022 at 3:54
  • 1
    Beat me to it, but I couldn't load meta to post. Incidentally the site was offline at 03.15 GMT ish yesterday also. Repro on worldbuilding, skeptics, biology, scifi-fantasy, medicalsciences, psychology and main meta.
    – W.O.
    Commented Jan 29, 2022 at 3:56
  • 1
    People have reported problems on Twitter. It seems to be working fine again.
    – Sebastian Simon
    Commented Jan 29, 2022 at 3:58
  • "We had another issue just now, but everything is up again. We're still monitoring. 4:20 am · 29 Jan 2022" - it's still ongoing.
    – W.O.
    Commented Jan 29, 2022 at 4:28
  • 33
    We are being hit by a large evolving DDoS attack, official answer coming Monday.
    – Josh Zhang StaffMod
    Commented Jan 29, 2022 at 4:34
  • @SebastianSimon people mention midwest usa in the tweets, I am from Asia, no problem till now.
    – FriendlyFire
    Commented Jan 29, 2022 at 6:10
  • @VScode_fanboy I'm in west-coast USA, and a few hours ago, the website was down and would sometimes load a little bit. It's back now, but I can confirm that there were issues in west USA.
    – cocomac
    Commented Jan 29, 2022 at 6:18
  • 2
    @cocomac hmm, it seems it is globally spread, but affects people randomly. (Perfect example of a DDoS)
    – FriendlyFire
    Commented Jan 29, 2022 at 6:24
  • @VScode_fanboy They also mention Australia and several other places in the world.
    – Sebastian Simon
    Commented Jan 29, 2022 at 6:28
  • @SebastianSimon Yes, that's what I meant, let's wait till the official answer comes? should we make a chat room for discussing our ideas and theories? because I sense that the comments like that will be here.
    – FriendlyFire
    Commented Jan 29, 2022 at 6:30
  • @VScode_fanboy It'd best be on a separate server.
    – W.O.
    Commented Jan 29, 2022 at 7:37
  • 1
    @ARogueAnt. stin.to/o3k51 or us21.chatzy.com/62949396861134, what looks better?
    – FriendlyFire
    Commented Jan 29, 2022 at 7:58
  • 3
    Relevant: meta.stackexchange.com/a/375120/175002
    – AncientSwordRage
    Commented Jan 29, 2022 at 12:00

1 Answer 1

Reset to default
40

On Friday the 28th around 23:30 UTC, Stack Overflow was hit by a DDoS attack. Unlike previous attacks this month, this one ramped up very quickly and brought SE Network sites down. Once alerted, the SRE Team began analyzing the nature of the attack and took steps to mitigate it.

We were able to stabilize site traffic the first time around 00:50 UTC. The attacker discovered the mitigations we put in place, adapted, and resumed the attack around 02:50 UTC. Due to the active and evolving nature of this attack, we took a more targeted approach to mitigating the attack and were able to stabilize site traffic again around 04:00 UTC.

You can see by the graph below, the attacker attempted to adapt again but was unsuccessful for the remainder of the weekend.

Bandwidth graph showing DDoS attack, with the bytes transferred showing as two blocks of activity and a final sharp peak as described previously.

Improve this answer
11
  • 4
    Thank you and the team who responded, for all the hard work in keeping the site running. Do you have any details to share later down the line or further steps planned?
    – AncientSwordRage
    Commented Jan 31, 2022 at 15:32
  • 2
    In what way is this related to: "Unusual number of requests" error on users tab when page number is 1000 or more
    – Luuklag
    Commented Jan 31, 2022 at 15:36
  • 4
    @AncientSwordRage, we are going through normal post mortem rituals after the incident but I'm not sure if there will be any more information to share in the future.
    – Josh Zhang StaffMod
    Commented Jan 31, 2022 at 15:39
  • 2
    @Luuklag, we are unable to positively identify the attacker but Friday's attack and the one you linked are different in nature.
    – Josh Zhang StaffMod
    Commented Jan 31, 2022 at 15:40
  • 30
    True story: responding to this DDoS interrupted my Friday DnD 5e campaign w/ friends. My level 5 Tiefling Warlock got put into NPC / follower mode for 90 mins while I jumped in and helped the team. DDoSers: so freaking inconsiderate of others' time :P.
    – Haney
    Commented Jan 31, 2022 at 15:43
  • What software do you use to get that graph?
    – Henry WH Hack v3.0
    Commented Jan 31, 2022 at 15:44
  • Good job, I only noticed a couple of 2-3 minute downtimes.
    – bad_coder
    Commented Jan 31, 2022 at 15:46
  • 1
    @HenryWHHackv3.0b I'd guess grafana, though that's from messing with it for a future project that's on hold so I could be wrong
    – Journeyman Geek
    Commented Jan 31, 2022 at 15:48
  • 1
    @HenryWHHackv3.0b that specific graph is from our Fastly (CDN) dashboard. We also use SignalFX (Splunk Observability).
    – Josh Zhang StaffMod
    Commented Jan 31, 2022 at 15:52
  • 1
    Are the DDoS attacks back again right now? I get “This service is unavailable.” a few times and if the pages load, then only slowly.
    – Sebastian Simon
    Commented Feb 4, 2022 at 13:11
  • 1
    @SebastianSimon based on the similar comment, I'd say that yes, looks like same attacker.
    – Shadow Wizard
    Commented Feb 7, 2022 at 7:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .