I opened the Android app and was met by the "Loading stack exchange sites" splash screen. The spinner froze, then a black screen appeared, then the entire app froze. I tried a few more times and was met with the same results.

The full site also works fine on my phone, so it appears to just be the app.

On the iPhone app, I keep getting a "Request timed out. Tap to retry." message when loading the question body. The title gets loaded, as is the HNQ list and my own notifications, but no posts. The mobile site is working.

Requests to the API never load:

  • 1
    @ShadowWizard I tried the new app. It sucks. It might just be because I'm used to the SE setup, but to me, the new layout is just so poor I couldn't use it. Uninstalled it as soon as I realized it was stealing notifications from the SE app. And how can they shut it down if the new app can't be used for any other SE sites? That would be a terrible decision! May 23, 2017 at 12:12
  • @ShadowWizard I'm still getting notifications from the SE app, but if I click on them, I'm met with the situation described above. May 23, 2017 at 12:15
  • I can confirm that the app doesn't load for me as well May 23, 2017 at 12:18
  • 3
    The API is down.. and that also affects the apps.
    – Floern
    May 23, 2017 at 12:22
  • @Floern Ahh, that makes sense. Where can I hear about stuff like that? I didn't see an announcement. May 23, 2017 at 12:26
  • @Floern Source? May 23, 2017 at 12:28
  • 1
    @PatrickHofman Well no requests are getting through! My website uses the API and it's returning 504 timeouts (after 30(?) seconds).
    – Tim
    May 23, 2017 at 12:29
  • @PatrickHofman the apps use the public API, which I can't access at the moment, and my chatbot gets timeouts since about 2 hours
    – Floern
    May 23, 2017 at 12:29
  • I can call the API from here. Tried some endpoints. Maybe it is just a partial outage on the API. May 23, 2017 at 12:32
  • @PatrickHofman Yeah, I just got through as well.
    – Tim
    May 23, 2017 at 12:34
  • Post jacked! Although I suppose this may be a better description of the problem. May 23, 2017 at 12:34
  • @Carcigenicate Sorry, I wanted to combine it into one post and it describe the problem more accurately for people who are also searching for this issue. Feel free to rollback my edits!
    – Tim
    May 23, 2017 at 12:35
  • So the API isn't down @Tim May 23, 2017 at 12:35
  • 2
    Should be back up now.
    – Oded
    May 23, 2017 at 13:17
  • 3
    @Oded answer with some juicy details about what went wrong (and who is to blame!) would be welcome then. :-) May 23, 2017 at 13:18

1 Answer 1


When we make a change to a post, we inform our API that the post has been changed so that it doesn't return an outdated cached version. Normally, this is a relatively small list of things since not that much changes (thousands of posts an hour).

As part of the HTTPS migration though, we needed to replace each site's links to http:// to their https:// equivalents (so you didn't eat an extra, insecure 301 to get there). On all the other sites, this went just fine. They were small enough that it went smoothly.

But Stack Overflow yielded some surprises. The way we find http:// links is by using our Elastic index and then crawling those posts to replace them. Unfortunately, Samo made this very efficient - it went very fast and slammed a ton of items into that cache breaker queue concurrently. The piece that checks the queue does a ZRANGE on redis and instead of getting the usual few-thousand items, it got well over 100k. This was enough to cause redis trouble on that node and result in cascading failure of the API usage on that redis connection.

Here's what that impact looked like: LocalDC cache SLOWLOG

...so yeah, that was bad, and unexpected, and it happened so quickly it was finished before we realized what the cause was. It also self-healed before we could do anything.

We're sorry this happened - it's one of those things that gives a loving surprise when deployed at scale (this batch was 1.8 million posts across the network).

  • Would you add an intermediary/rate limit to prevent it from happening again?
    – Braiam
    May 23, 2017 at 13:40
  • 4
    @Braiam If we need to rebake that many posts again - yes. It's only happened once in 9 years, so no plans to do that again any time soon :) May 23, 2017 at 13:43
  • 5
    Thanks for taking to time to write-up what happened. I love reading incident reports like this. Always interesting to read about what goes on behind the scenes of a site I use everyday. May 23, 2017 at 13:44
  • As I said already: https, it's always https. May 23, 2017 at 13:46
  • 1
    So essentially, your code was just too good :P
    – Tim
    May 23, 2017 at 18:10
  • that's totally F#'s fault twitter.com/Nick_Craver/status/852498557714255873
    – m0sa
    May 24, 2017 at 7:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .