Review queues not working because of JavaScript error

Question

For some users, reviewing is not possible at the moment; the queues show up blank. Both active queues and completed review items are not visible, probably because of a JavaScript error:

I'm using the latest macOS and can reproduce it in Firefox, Safari and Chrome. For me it seems to work fine in Chrome on Windows, but other users report problems with Windows too.

Answer 1

My bad! I was canary testing some JS infrastructure changes on one of our nine web servers (namely NY-WEB09, which as the name suggests is located in our data centre in... New Jersey).

The bug which caused your issue is kind of subtle, so please bear with me while I explain!

• All of our scripts (on the non-Meta network sites) live on a separate domain called sstatic.net. When the server writes a script tag to the HTML it uses the script's location in the sstatic.net domain.

• sstatic.net is served by our CDN (Content Delivery Network), which is provided by a company called Fastly.

• The first time anyone downloads a (JS, CSS, etc) file through sstatic.net, Fastly grabs it from our web server, but for subsequent requests Fastly caches that file for us on their servers. Fastly has edge nodes all around the world — the idea is, if you live in Australia you'd prefer it if your browser didn't have to roundtrip to New Jersey for every JS file, since that might take a long time.

• But when we release new JS, we don't want to serve the cached version since it may be out of date! When the app tier writes a script tag into the page, it sticks a hash of the JS file's content into the request's query string to function as a cache-breaker. (That's why you'll see something like ?v=4da18762a4fa when you look at a JS file's URL in the debugger.) This prompts Fastly to pull the latest version of the file from our data centre again, since the file's URL no longer matches the version which Fastly has in its cache.

• I wanted to test my infrastructure change on one of our web servers, so as to impact only 1/9th of our users if something went wrong. I deployed the new version of the JavaScript to the CDN, but only deployed the new cache-breaker hashes and backend code to NY-WEB09. I convinced myself that this would mean users who hit NY-WEB09 would get the new JS and users on the other 8 web servers would get the (older) JS from the CDN cache, corresponding to the older hash.

• That second assumption turned out to be mistaken. It seems like there are certain circumstances in which the CDN will get the latest version of the file even if the cache-breaker hash matches an older version. Your session was hitting NY-WEB04, so you had the old backend code (and the old cache-breaker hash) but you were getting the new JS, which failed because it was expecting to be paired with the new backend code.

• I'm not exactly sure what leads the CDN to grab the latest version of a file even when it "should" have a cache hit. It seems to have only affected a small proportion of users. Perhaps it was a regional thing; perhaps Fastly spins nodes up and down every now and then; I'm not sure.

Lesson learned: I knew canary testing front-end changes was trickier than back-end changes, but it turns out to be even trickier than that! I now know that with our current setup we have no way to run two versions of the JS side by side, since it's not safe to rely on the cache-breaker hash to deliver an old version. Next time I deploy a JS change I'll make sure to put it on all the web servers at once. #YOLO

The good news is, I'm currently working on setting up Webpack (that's the infrastructure change I'm testing!), which does a better job of managing cached assets than our current homemade glue-and-string setup. In particular, Webpack has a configuration option to output each version of a file under a different filename (rather than the same filename with a query string appended). Hopefully by the end of my project we'll be able to test JS changes side-by-side without risking this bug again.

In the mean time, I've rolled the CDN and all nine app servers back to the code's main branch. If you clear your browser cache you should get the correct version of the code. Should.

Finally, thank you for reporting this! This is exactly why we run canary tests. It's really useful to hear from the community when something's not working. Although in this case the code we were testing was fine and the bug was caused by the canary test itself. Whoops!