We wanted to share some of what's been happening on the network lately behind the scenes. We do our best to keep fires from being visible but that's not always possible and when it happens my goal is to be as open as possible.
In short, we've enabled captchas for anonymous users when they search Q&A. If you'd like more details, read on!
So, attackers. In recent weeks we've been hit with progressively escalating attacks targeting our search engine. These attacks have varied and escalated in both complexity and volume. In short, someone is attempting to DDoS us from over 10,000 endpoints. This has caused abnormally high load on our search clusters used for both site search and search on
/jobs as well.
While we've tried to play nice with what at first looked like crawlers making questionable choices, after deploying changes to
robots.txt instructing them to back off, continued abuse, and user agent/IP range hits against it, it became obvious this was a malicious attack against our infrastructure.
Ultimately, this was the cause of a brief site outage Monday (October 19, 2020) which was a cascading impact of search queries executed (normally quickly) while a database connection was held open. When queries suddenly took far longer, this resulted in connection pool exhaustion across our web tier and resulted in collateral damage outside search in the wider Q&A application. We have removed this interaction by eagerly closing connections in those paths (less optimal, but safer).
In addition, we have enabled captchas for anonymous users hitting our search endpoints to eliminate the traffic coming from automated bots, but still allow for legitimate user traffic. We'll now require a captcha solved when hitting search anonymously (which is valid for 5 minutes, we may tweak this). It's not a usability tradeoff we'd like to make, but it is a necessary one at this phase. Our search resources simply are not infinite and to make them scale to 10-100x their current size only to serve botnets isn't a good use of anyone's resources either.
We hope this results in only minimal disruption; if anyone has any specific questions please let us know and we’ll answer ’em if we can.