For the past month, Stack Overflow has been hit by weekly DDoS attacks that progressively grew in size and scope. In each incident, the attacker(s) have been changing their methodology and responding to our countermeasures. Initially, we were able to detect and mitigate the attacks before any performance degradation could be noticed, but the latest attacks ramped up very quickly and the site was brought down before we could react.
While we cannot go into specifics on each attack in order to maintain OPSEC and not tip off the attackers, we can say that each individual attack has been using different IP addresses and targeting different aspects of the site. During an outage, our top priority is always getting the site back up and running. After traffic has been stabilized, we perform a post-mortem for the incident where we assess and improve upon the actions we have taken.
During the outage last Sunday, we noticed that a large amount of the DDoS traffic originated from Tor exit nodes. The decision to block Tor exit nodes did not come lightly; in fact, Teresa, our CTO was on the call when we discussed remediation methods. Due to the persistent nature of the attack and our desire to bring the site back up as fast as possible, we made the decision to block all DDoS traffic endpoints, including these Tor exit nodes.
We did not target, nor set out to block all traffic from Tor; that’s not something Stack has ever done. However, due to the shared nature of Tor exit nodes, some of them were also routing DDoS attacks to our sites and were blocked. We have tried removing these blocks between attacks, but this action has resulted in further site outages as DDoS efforts continue to originate from these exit nodes. Unfortunately, blocking the Tor exit nodes also blocks legitimate users from using them. An immediate solution for users who find themselves blocked is to access our site from other IP addresses, via home internet, work internet, or other VPN services.
We are continuing to evaluate the situation and will keep our community updated. Thank you for your patience and understanding.
You may see Tor access improve over the next few days as we do some testing. The changes may be temporary depending on how everything goes. We will continue to keep everyone updated as the situation evolves.
There was another DDoS attack that briefly affected the site last night; we were able to test some of the changes we have made, and we are happy to report that none of the DDoS traffic originated from Tor.
I wrote a blog post sharing some lessons learned: https://stackoverflow.blog/2022/05/16/stack-under-attack-what-we-learned-about-handling-ddos-attacks/