18

In 2016, Stack Overflow had an outage due to catastrophic backtracking in a regex. There's a tweet about the outage, which links to the postmortem on stackstatus.net. That link is now dead. I've tried replacing http with https, and adding the www. subdomain, but none of these let me read the postmortem.

This postmortem link is all over the internet, and as recently as a few months ago I was able to follow it to read the postmortem. I've tried looking for an alternate link, by browsing the incident history on stackstatus.net, but it claims there were no incidents in that time period. In fact, I can't find any incidents from before May of 2022!

It looks like this might be related to Improvements to site status and incident communication - great, there's a new system. But is the archive of the old system available somewhere? I'd love to be able to link to this postmortem, because it's a great example of the hidden danger of innocuous-looking regexes even in the hands of competent professionals.

4
  • 1
    I've updated my answer with this detail: we've fixed all the content here to point links at the Tumblr content.
    – Aaron Bertrand Staff
    Commented Sep 30, 2022 at 20:52
  • 1
    ..with the exception of links that intentionally demonstrate the problem, like the link in the question, which currently leads to an (empty) 404 page on the new status page site.
    – Aaron Bertrand Staff
    Commented Sep 30, 2022 at 21:09
  • ...and now a redirect is in place, so all links to www.stackstatus.net/post/... should land on stackstatus.tumblr.com/post/...
    – Aaron Bertrand Staff
    Commented Nov 8, 2022 at 13:43
  • @AaronBertrand Nice, thanks for the update.
    – amalloy
    Commented Nov 8, 2022 at 16:06

2 Answers 2

18
+100

Background

The status page is hosted independently of our network so that, when our entire infrastructure is down, you can still see the status. The original choice of using Tumblr was indeed an "interesting" one, but it served its purpose for several years.

One of the challenges we had with Tumblr, Twitter, bug/support requests here, etc. is keeping them all up to date while simultaneously trying to solve the issue. This can amount to significant pressure, and different credentials for company social media accounts added to that complication. Some of us didn't have (ready) access to one or both accounts, and sometimes we'd forget to post updates in one place or another due to chaos, fatigue, assuming another person would do it, or just simple memory. After the incident was resolved, the first thing most people would do is go back to bed. As many of you have pointed out in the past, all of this could lead to disjointed and unsatisfactory transparency, and a sometimes delayed notification that stability had been restored - apart from, you know, F5. :-)

So, we didn't change hosts because of better pricing or because we didn't like Tumblr or whatever. We changed to FireHydrant with the goal of smoothing out automation and having tighter integration with our incident system for incidents going forward. Some of that is described here. When we create, update, or resolve an incident with a specific priority in FireHydrant, we want it follow a runbook so that, without further intervention, it will conditionally and automatically (among other things, of course):

Please bookmark those, as the goal is for you to see updates there before you'll see a response from us on meta.

We are continuing to perfect this process and refine our runbooks so that all of the automation works the way we expect, and we appreciate your patience as we fine tune and improve. That's not sales bro gibberish; I do understand there is a lot of passion around this topic, and I hope you trust that we are working on making this better, but also that there will inevitably be a learning curve as we ramp up our new system.

Now to the stated problem

As for existing content from past incidents, I don't know that importing old content into the new system was ever even considered until today. We know where the content is, as Animuson has described, and there are potentially ways we can help solve the problem of old links pointing to non-existent pages in the new system. However, we don't control the server, so it would require intervention from at least FireHydrant and maybe GCP also. Some limitations:

  • We can't just import the content to the new system and make old links magically work again. FH has its own URL generation system, and I'm still working with them to resolve this other issue.
  • We can't just redirect (a suggestion I made off-the-cuff in an earlier comment), because we don't have control over the server, the application(s) it's running, or host header and other configuration settings. I have brought up multiple potential workarounds (even putting static HTML files with vintage meta redirects at the old URLs), but so far none seem promising.
  • UPDATE 2022-11-08: FireHydrant has implemented redirects (e.g. this link now lands on Tumblr. Still, the next bullet remains partially true:
  • We wouldn't want to redirect to Tumblr because we don't know that our account there (or Tumblr itself) will be available indefinitely.
  • We won't be able to solve the "http/no www issue" described here and here, no matter what we do with the content. If an old link points to http://stackstatus.net, the link will just never work for some people.
  • While we continue working with the vendor on a more seamless solution, we have pinned a tweet (thanks @Sonic!) explaining the easy (but manual) fix, and have asked for similar content or other pointers on the 404 page.

Unfortunately the best recommendation I have in the short term is to locate (or generate) versions of the post-mortems in the Wayback Machine (like this one Luuklag created), and manually update your own links and bookmarks. I know this is not ideal; having done this many times, I know this is not a trivial amount of work for you, either.

A fix for content here, at least

We have in the meantime manually updated all of the existing posts and comments on meta, with the exception of posts from recent months talking specifically about the www/no https issues, to point directly at stackstatus.tumblr.com for all posts prior to June 2022. If you see a link that actively still points to stackstatus.net here, or if you see broken links on other network sites, feel free to update them directly, or point me at them and I can help. Longer-term, should Tumblr (or our account there) tumble, we should have enough notice that the process will be easy enough to repeat by replacing links with their web.archive.org equivalent.

As for incident vs. post-mortem

Going forward, I consider the status page as having a charter that is completely different from post-mortems:

  • The status page is simply meant to inform you of one of the following things:

    • "Yes, there is an incident ongoing"
    • "Yes, there was an incident from <time1> to <time2>, but it is now resolved."
    • "Everything is currently fine. If you are experiencing issues, it might be something else."

    This is really all the status page needs to indicate, especially retroactively. At least IMHO. During the incident, while it might be nice to keep you in the loop on root cause and what we're doing about it, the on-call folks are likely going to focus more on the resolution than anything else.

  • A post-mortem is meant to dig into the details of the outage or instability, and explain the cause / resolution / future prevention - to be transparent (at least to the extent that we can), to satisfy curiosity, and even to help others avoid the same issues. To me this doesn't belong in the incident system, but rather in an answer here (under the maintenance notification, when relevant, or to any bug/support request), or a blog post, or both. Aside from separation of duties, and that "site is down" or "service is degraded" has the same impact to you regardless of cause or current scrambling, this also gives us a lot more time to formulate the right answer, articulate the details of the mitigation, investigate and decide on courses of action for future prevention, and present it all in a way that is richer in formatting than pulling plain text out of a comment field from the incident system.

The TL;DR here is simply to not expect long, drawn-out analysis on status page incidents (or on Tumblr); expect those details here, or in follow-up blog posts we'll post here.

7
  • 1
    @Luuklag sorry we are fighting each other for edits, mobile is not cooperating.
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 20:46
  • 1
    No problem, just got curious to see what you changed, and spotted a stray y. Mobile indeed is less then helpful when you got a long post.
    – Luuklag
    Commented Sep 28, 2022 at 20:48
  • 1
    You might consider publishing a pinned tweet from the Stack Status account stating that old links from before the change to the status website are broken and instructions on how to modify them if one wants to view them. Commented Sep 28, 2022 at 22:14
  • 1
    @Sonic I'll add it to the list, thanks!
    – Aaron Bertrand Staff
    Commented Sep 29, 2022 at 2:24
  • 1
    Two things - given an ok, and the slightest bit of direction, the community is really good at fixing links. The other is - at the job I was in, one of the unofficial tasks I had was literally running interference between my boss and the DBAs/SAs and other support folks - so I'd say anything above "We're aware its broken" would/should be best effort, and if anyone's being too pushy let us know :D Commented Sep 29, 2022 at 12:35
  • How times have changed in the XML world, writing an interval [t1, t2] not expressing it with square brackets used to be considered an offense.
    – bad_coder
    Commented Sep 29, 2022 at 21:50
  • 2
    Thanks @Sonic, here's the tweet.
    – Aaron Bertrand Staff
    Commented Sep 30, 2022 at 13:53
19

The old Stack Status page was hosted on Tumblr so as long as that account still exists, you can still access the complete archive of stuff that was originally posted there: https://stackstatus.tumblr.com/ - The custom domain just no longer points to the Tumblr blog.

The postmortem you seek is still available here: https://stackstatus.tumblr.com/post/147710624694/outage-postmortem-july-20-2016

It might make sense to link to the Tumblr blog in some form as an archive of before we moved, but I'll leave that to the SRE team to decide.

11
  • 6
    For histories sake, I just forced the waybackmachine to save a snapshot of the blog. For reference you can find the archived page here
    – Luuklag
    Commented Sep 27, 2022 at 22:16
  • So you moved to new host and started from scratch, without even taking any old content. Any reason? Too complicated? New platform doesn't support migrating old stuff? Commented Sep 28, 2022 at 7:42
  • 2
    @ShadowTheKidWizard Just a couple of thoughts. New host is firehydrant. I don't know if they support importing old stuff, but I'm having difficulties flipping bits on new stuff created inside their platform, so I'm not sure how it would handle external content. I wasn't involved directly in the transition, but I don't know that it was a priority to even consider importing the old content. The focus was getting the page working and smoothing out the automation for events going forward. Incidents IMHO are separate from the post-mortems - the latter are often better suited for blog posts.
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 11:35
  • 1
    @ShadowTheKidWizard ...or as answers to the maintenance notifications and/or bug/support questions here. Or both/all. The incident platform is meant to advise people that an incident is occurring or that they aren't crazy and yes there was an issue but it's resolved now. Going into the low-level details of cause/resolution to be transparent and satisfy curiosity and even help others avoid the same issues is a separate charter, I think, and is also better solved by platforms with more flexible formatting, graphics, markdown, etc. Firehydrant just literally pulls text from incidents.
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 11:49
  • 1
    @AaronBertrand excellent points, I totally agree. So, bottom line, how can I trigger the process of "migrating" old post-mortem posts to the blog? Commented Sep 28, 2022 at 11:57
  • 1
    @ShadowTheKidWizard You could submit a request I suppose, but now that we know where the content is, it may be tough to justify the curation effort of copying it (correctly). :-) A more reasonable (or at least more immediate) compromise might be to set up 301s for the old post-mortem links to redirect to tumblr.
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 12:02
  • @AaronBertrand question is, aren't there plans to shut down the tumblr account? If not, and that's a stable archive then yes, it can be used as reference, but I had the feeling that it's going to be closed at some point. Commented Sep 28, 2022 at 12:07
  • 3
    @ShadowTheKidWizard I don't know the long-term plans, but it wouldn't take much to locate (or generate) them in the wayback machine, and have the redirects go there. If you don't trust that...
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 12:09
  • 2
    @AaronBertrand as usual, thanks for your invaluable insights. You might also share some of this in a new answer, e.g. "The focus was getting the page working and smoothing out the automation for events going forward" which explains why you changed host, it's really not trivial to know or understand. Commented Sep 28, 2022 at 12:17
  • 1
    @ShadowTheKidWizard The other aspect that I don't know we'll be able to solve is where old links point to https://www.stackstatus.net with no https/www, as mentioned here and here. So existing links that are formatted that way may never work no matter what we do with the content.
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 12:18
  • 1
    @ShadowTheKidWizard You got it. :-)
    – Aaron Bertrand Staff
    Commented Sep 28, 2022 at 13:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .