Recent Incident with Email/Password Log In - January 9th, 2024
Starting on Tuesday, January 9th at 20:45 UTC, we received an alert that our OpenId Provider was receiving timeout exceptions to its Redis instance. This resulted in users being unable to log in or sign up using an email and password across all Stack Exchange network sites (including Stack Overflow) and Stack Overflow for Teams (Free, Basic, and Business). Other authentication providers (e.g. Google, GitHub, Facebook, etc.) remained unaffected.
To mitigate this issue, we attempted to failover to a different region, but the failover environment was experiencing similar issues. We tried various other measures to reestablish the connection between the application and Redis. Eventually, we determined the issue was due to the DNS servers being configured incorrectly. We switched the DNS servers referenced by the application and that ultimately resolved the issue.
During the incident, there were some confusing signals in our observability pipeline that delayed our ability to effectively diagnose and remediate the issue. Specifically, the health check endpoint on our OpenId Provider application was returning a successful response (albeit rather slowly) even though the Redis instance was unreachable. The health check endpoint also failed to log the errors from Redis. Both of these issues have since been addressed.
We’ve also improved our alerting for both the active region and our failover region. This will help ensure that we have a healthy failover instance that we can use to mitigate any future incidents more quickly.
We would like to thank our community for their patience and understanding during this incident.