When we make a change to a post, we inform our API that the post has been changed so that it doesn't return an outdated cached version. Normally, this is a relatively small list of things since not that much changes (thousands of posts an hour).
As part of the HTTPS migration though, we needed to replace each site's links to http:// to their https:// equivalents (so you didn't eat an extra, insecure 301 to get there). On all the other sites, this went just fine. They were small enough that it went smoothly.
But Stack Overflow yielded some surprises. The way we find http:// links is by using our Elastic index and then crawling those posts to replace them. Unfortunately, Samo made this very efficient - it went very fast and slammed a ton of items into that cache breaker queue concurrently. The piece that checks the queue does a ZRANGE on redis and instead of getting the usual few-thousand items, it got well over 100k. This was enough to cause redis trouble on that node and result in cascading failure of the API usage on that redis connection.
Here's what that impact looked like: 
...so yeah, that was bad, and unexpected, and it happened so quickly it was finished before we realized what the cause was. It also self-healed before we could do anything.
We're sorry this happened - it's one of those things that gives a loving surprise when deployed at scale (this batch was 1.8 million posts across the network).
