22

I opened the Android app and was met by the "Loading stack exchange sites" splash screen. The spinner froze, then a black screen appeared, then the entire app froze. I tried a few more times and was met with the same results.

The full site also works fine on my phone, so it appears to just be the app.

On the iPhone app, I keep getting a "Request timed out. Tap to retry." message when loading the question body. The title gets loaded, as is the HNQ list and my own notifications, but no posts. The mobile site is working.


Requests to the API never load:

  • 1
    @ShadowWizard I tried the new app. It sucks. It might just be because I'm used to the SE setup, but to me, the new layout is just so poor I couldn't use it. Uninstalled it as soon as I realized it was stealing notifications from the SE app. And how can they shut it down if the new app can't be used for any other SE sites? That would be a terrible decision! – Carcigenicate May 23 '17 at 12:12
  • @ShadowWizard I'm still getting notifications from the SE app, but if I click on them, I'm met with the situation described above. – Carcigenicate May 23 '17 at 12:15
  • I can confirm that the app doesn't load for me as well – Mad Scientist May 23 '17 at 12:18
  • 3
    The API is down.. and that also affects the apps. – Floern May 23 '17 at 12:22
  • @Floern Ahh, that makes sense. Where can I hear about stuff like that? I didn't see an announcement. – Carcigenicate May 23 '17 at 12:26
  • @Floern Source? – Patrick Hofman May 23 '17 at 12:28
  • 1
    @PatrickHofman Well no requests are getting through! My website uses the API and it's returning 504 timeouts (after 30(?) seconds). – Tim May 23 '17 at 12:29
  • @PatrickHofman the apps use the public API, which I can't access at the moment, and my chatbot gets timeouts since about 2 hours – Floern May 23 '17 at 12:29
  • I can call the API from here. Tried some endpoints. Maybe it is just a partial outage on the API. – Patrick Hofman May 23 '17 at 12:32
  • @PatrickHofman Yeah, I just got through as well. – Tim May 23 '17 at 12:34
  • Post jacked! Although I suppose this may be a better description of the problem. – Carcigenicate May 23 '17 at 12:34
  • @Carcigenicate Sorry, I wanted to combine it into one post and it describe the problem more accurately for people who are also searching for this issue. Feel free to rollback my edits! – Tim May 23 '17 at 12:35
  • So the API isn't down @Tim – Patrick Hofman May 23 '17 at 12:35
  • 2
    Should be back up now. – Oded May 23 '17 at 13:17
  • 3
    @Oded answer with some juicy details about what went wrong (and who is to blame!) would be welcome then. :-) – Shadow Wizard May 23 '17 at 13:18
34

When we make a change to a post, we inform our API that the post has been changed so that it doesn't return an outdated cached version. Normally, this is a relatively small list of things since not that much changes (thousands of posts an hour).

As part of the HTTPS migration though, we needed to replace each site's links to http:// to their https:// equivalents (so you didn't eat an extra, insecure 301 to get there). On all the other sites, this went just fine. They were small enough that it went smoothly.

But Stack Overflow yielded some surprises. The way we find http:// links is by using our Elastic index and then crawling those posts to replace them. Unfortunately, Samo made this very efficient - it went very fast and slammed a ton of items into that cache breaker queue concurrently. The piece that checks the queue does a ZRANGE on redis and instead of getting the usual few-thousand items, it got well over 100k. This was enough to cause redis trouble on that node and result in cascading failure of the API usage on that redis connection.

Here's what that impact looked like: LocalDC cache SLOWLOG

...so yeah, that was bad, and unexpected, and it happened so quickly it was finished before we realized what the cause was. It also self-healed before we could do anything.

We're sorry this happened - it's one of those things that gives a loving surprise when deployed at scale (this batch was 1.8 million posts across the network).

  • Would you add an intermediary/rate limit to prevent it from happening again? – Braiam May 23 '17 at 13:40
  • 4
    @Braiam If we need to rebake that many posts again - yes. It's only happened once in 9 years, so no plans to do that again any time soon :) – Nick Craver May 23 '17 at 13:43
  • 5
    Thanks for taking to time to write-up what happened. I love reading incident reports like this. Always interesting to read about what goes on behind the scenes of a site I use everyday. – Carcigenicate May 23 '17 at 13:44
  • As I said already: https, it's always https. – Shadow Wizard May 23 '17 at 13:46
  • 1
    So essentially, your code was just too good :P – Tim May 23 '17 at 18:10
  • that's totally F#'s fault twitter.com/Nick_Craver/status/852498557714255873 – m0sa May 24 '17 at 7:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .