I marked this complete since we've resolved it and keep an eye on it when launching new sites, but I think some details are in order because it's not 100% obvious why this happens. Here are some details:
It's a build race condition with the favicon sprite map generation and the cache breaker at the CSS level of PNG copies where the CDN caches an old PNG due to hitting a web server further in rotation. Here's what happens in a build:
We deploy web servers 1-9 in order, like this:
- Copy static resources to
- Update the website/app on
So the new content should be there before the website/app is sending new cache breakers to clients, right? Almost. At the app level, yes. At the CSS level, no. The
?v=ab1234 cache breakers you see are a SHA256 hash of the file. The website/app also has a copy of the static content. On meta, it's so it can be served locally and meta is usable even if the CDN is down (see how it's
/content here and it's from
cdn.sstatic.net on Stack Overflow?). This is so you fine guys and gals can tell us, ya know, when the CDN is FUBAR.
Okay so, new static content is on
cdn.sstatic.net (backed by those 9 web servers) before the web app has it. So before the first web app roll on
web01 starts sending new cache breakers, all the new files are there. That's how we can cache static files for 7 days on clients and break them when we need to. This works pretty well and there's nothing all that special about it. When the CDN request for the new file/cache breaker it hasn't seen before happens it can hit
web09...any of those have the new file and we're good.
But, this isn't true for images in stylesheets - this is still a problem we're working on. The race happens in the first of the 2 rollouts, like this:
Let's say the static content phase is on
web04 have new CSS and static content.
- A CDN cache node is missing
.css from the web app with the current (not new) cache breaker, so it asks for the content.
- This origin request for content hits
web02, and it gets new CSS
- That new CSS has a new cache breaker for the image inside of it (we do the same SHA256 during the build of CSS)
- The user getting the CSS then requests that new PNG file/cache breaker combo, thir request isn't on the CDN either (being new and all), they land on
- They just loaded an old image with a new cache breaker, for 7 days. And the CDN cached it.
Sooooo, that's how it happens. What are we doing about it?
Well, my plan is to copy PNG content as a new phase 0 to the static deploy, which means everything that depends on a cache breaker is there before a new cache breaker is served, even at the CSS level. The problem with doing that is it's expensive (time-wise) in the build. But, with all the unified theming work we've just deployed across the network, we're also dropping a lot of PNGs. This greatly lessens the time required to deploy in such a new phase and makes it probably reasonable.
We're almost there - hang tight, we're working on preventing this scenario entirely.
Side note: if anyone's curious about the cache breaker file mismatches the other direction: can that happen? Yes sure, you may get new CSS with the old app/cache breaker if a CDN edge is missing it during a deploy. It can happen, and you can cache it for 7 days. But, that's a good way to fail. Because your long-lived local content is about to match the HTML it goes with as the deploy finishes...so a simple next-page-load fixes it. Of all the options, that's the best way to fail. It's still rare, but it's also self-recovering.