Update on the ongoing DDoS attacks and blocking Tor exit nodes

Question

For the past month, Stack Overflow has been hit by weekly DDoS attacks that progressively grew in size and scope. In each incident, the attacker(s) have been changing their methodology and responding to our countermeasures. Initially, we were able to detect and mitigate the attacks before any performance degradation could be noticed, but the latest attacks ramped up very quickly and the site was brought down before we could react.

While we cannot go into specifics on each attack in order to maintain OPSEC and not tip off the attackers, we can say that each individual attack has been using different IP addresses and targeting different aspects of the site. During an outage, our top priority is always getting the site back up and running. After traffic has been stabilized, we perform a post-mortem for the incident where we assess and improve upon the actions we have taken.

During the outage last Sunday, we noticed that a large amount of the DDoS traffic originated from Tor exit nodes. The decision to block Tor exit nodes did not come lightly; in fact, Teresa, our CTO was on the call when we discussed remediation methods. Due to the persistent nature of the attack and our desire to bring the site back up as fast as possible, we made the decision to block all DDoS traffic endpoints, including these Tor exit nodes.

We did not target, nor set out to block all traffic from Tor; that’s not something Stack has ever done. However, due to the shared nature of Tor exit nodes, some of them were also routing DDoS attacks to our sites and were blocked. We have tried removing these blocks between attacks, but this action has resulted in further site outages as DDoS efforts continue to originate from these exit nodes. Unfortunately, blocking the Tor exit nodes also blocks legitimate users from using them. An immediate solution for users who find themselves blocked is to access our site from other IP addresses, via home internet, work internet, or other VPN services.

We are continuing to evaluate the situation and will keep our community updated. Thank you for your patience and understanding.

Update 2022-02-10:

You may see Tor access improve over the next few days as we do some testing. The changes may be temporary depending on how everything goes. We will continue to keep everyone updated as the situation evolves.

Update 2022-02-11:

There was another DDoS attack that briefly affected the site last night; we were able to test some of the changes we have made, and we are happy to report that none of the DDoS traffic originated from Tor.

Update 2022-05-19:

I wrote a blog post sharing some lessons learned: https://stackoverflow.blog/2022/05/16/stack-under-attack-what-we-learned-about-handling-ddos-attacks/

Answer 1

During the last outage when Fastly's servers were down, Stack Overflow and many other websites were down. During that time, in Twitter, I saw a tweet that said that they are going to temporarily route traffic directly to the backend Stack Overflow servers. I guess it was a situation where they didn't have any other choice.

Fastly themselves say that they mitigate DDoS attacks in here. (Maybe it requires a separate plan?). They do not seem to have mitigated the attack to Stack Overflow.

All these things seem to say that the attack was directly launched onto Stack Overflow's servers' IP addresses (without Fastly being in between).

Note: AFAIK, it seems that Stack Overflow is still protected using Fastly.

Answer 2

For the past 3 weeks we have been working to further strengthen our DDoS mitigation while also accounting for Tor traffic. At this point, we feel that we have been able to strike a good balance between protecting the site from DDoS attacks without blocking Tor traffic. Although we cannot rule out the possibility of blocking Tor exit nodes in the future if they are used as part of a DDoS attack, we will work to unblock them as soon as possible after we are able to mitigate the attack.

Unfortunately, we’re still unable to go into specifics about the nature of the DDoS attacks and what we have implemented to mitigate them. Our monitoring shows that we are still being attacked, but fortunately, we have been able to mitigate them before any major disruption occurs.

Again, we’d like to thank the community for your patience and understanding.