26

For some users, reviewing is not possible at the moment; the queues show up blank. Both active queues and completed review items are not visible, probably because of a JavaScript error:

I'm using the latest macOS and can reproduce it in Firefox, Safari and Chrome. For me it seems to work fine in Chrome on Windows, but other users report problems with Windows too.

12
  • 1
    I noticed this on my mac in Chrome, FF and Safari, as well as mobile Chrome, though surprisingly on the Windows machine I remote desktop to in the office it appears to work in FF
    – Sam Onela
    Oct 18 at 17:32
  • It happens on Windows too, tested on Opera GX, Brave, Chrome and Firefox. It is strange because SO queues are working well for me, but SOpt queues are not. Oct 18 at 17:43
  • 1
    Sam and I discussed this a little in chat. However my review queues randomly started working again with a ctrl-f5. (Firefox Linux.)
    – Peilonrayz
    Oct 18 at 17:44
  • 1
    This may have also broken comments on election nominations, but it's extremely intermittent (... in terms of users, I've seen 2 repros and 3 norepros) Oct 18 at 17:47
  • 1
    Yeah, I'm not experiencing this myself, either. I'm not sure what to poke at here. :(
    – Catija StaffMod
    Oct 18 at 17:49
  • 2
    @Catija it may be a regional issue. I'm in the Netherlands but the Windows VM I tested on is hosted in the US.
    – Glorfindel Mod
    Oct 18 at 17:50
  • I'm networked in Europe (and can repro), but the other repro I've seen was in the US as well. Oct 18 at 17:55
  • Checking approved edits also doesn't display the reviewed item. (Confirmed on Win 10 Firefox 93).
    – bad_coder
    Oct 18 at 17:59
  • 1
    I'm getting the same error out of https://cdn.sstatic.net/Js/ask.en.js?v=4da18762a4fa:1 in both FF and Chrome on Windows ... is that worth a separate bug report? In my case is blocking me from asking a question. Oct 18 at 18:21
  • Possibly related report on SO of user unable to ask a question: Cannot post a question
    – zcoop98
    Oct 18 at 18:23
  • 1
    I'm in the upper midwest US. Can't comment or flag comments on SO proper. Uncaught TypeError: Cannot read properties of null (reading 'innerText') in full.en.js.
    – isherwood
    Oct 18 at 19:57
  • It appears to be working for me now on my mac
    – Sam Onela
    Oct 18 at 20:30
26

My bad! I was canary testing some JS infrastructure changes on one of our nine web servers (namely NY-WEB09, which as the name suggests is located in our data centre in... New Jersey).

The bug which caused your issue is kind of subtle, so please bear with me while I explain!

  • All of our scripts (on the non-Meta network sites) live on a separate domain called sstatic.net. When the server writes a script tag to the HTML it uses the script's location in the sstatic.net domain.

  • sstatic.net is served by our CDN (Content Delivery Network), which is provided by a company called Fastly.

  • The first time anyone downloads a (JS, CSS, etc) file through sstatic.net, Fastly grabs it from our web server, but for subsequent requests Fastly caches that file for us on their servers. Fastly has edge nodes all around the world — the idea is, if you live in Australia you'd prefer it if your browser didn't have to roundtrip to New Jersey for every JS file, since that might take a long time.

  • But when we release new JS, we don't want to serve the cached version since it may be out of date! When the app tier writes a script tag into the page, it sticks a hash of the JS file's content into the request's query string to function as a cache-breaker. (That's why you'll see something like ?v=4da18762a4fa when you look at a JS file's URL in the debugger.) This prompts Fastly to pull the latest version of the file from our data centre again, since the file's URL no longer matches the version which Fastly has in its cache.

  • I wanted to test my infrastructure change on one of our web servers, so as to impact only 1/9th of our users if something went wrong. I deployed the new version of the JavaScript to the CDN, but only deployed the new cache-breaker hashes and backend code to NY-WEB09. I convinced myself that this would mean users who hit NY-WEB09 would get the new JS and users on the other 8 web servers would get the (older) JS from the CDN cache, corresponding to the older hash.

  • That second assumption turned out to be mistaken. It seems like there are certain circumstances in which the CDN will get the latest version of the file even if the cache-breaker hash matches an older version. Your session was hitting NY-WEB04, so you had the old backend code (and the old cache-breaker hash) but you were getting the new JS, which failed because it was expecting to be paired with the new backend code.

  • I'm not exactly sure what leads the CDN to grab the latest version of a file even when it "should" have a cache hit. It seems to have only affected a small proportion of users. Perhaps it was a regional thing; perhaps Fastly spins nodes up and down every now and then; I'm not sure.

Lesson learned: I knew canary testing front-end changes was trickier than back-end changes, but it turns out to be even trickier than that! I now know that with our current setup we have no way to run two versions of the JS side by side, since it's not safe to rely on the cache-breaker hash to deliver an old version. Next time I deploy a JS change I'll make sure to put it on all the web servers at once. #YOLO

The good news is, I'm currently working on setting up Webpack (that's the infrastructure change I'm testing!), which does a better job of managing cached assets than our current homemade glue-and-string setup. In particular, Webpack has a configuration option to output each version of a file under a different filename (rather than the same filename with a query string appended). Hopefully by the end of my project we'll be able to test JS changes side-by-side without risking this bug again.

In the mean time, I've rolled the CDN and all nine app servers back to the code's main branch. If you clear your browser cache you should get the correct version of the code. Should.

Finally, thank you for reporting this! This is exactly why we run canary tests. It's really useful to hear from the community when something's not working. Although in this case the code we were testing was fine and the bug was caused by the canary test itself. Whoops!

7
  • 3
    Some users are reporting broader issues with site functionality after this fix went out. Oct 18 at 22:55
  • Humorous note: "NY-WEB09, which as the name suggests is located in (...) New Jersey" shouldn't it be called NJ-WEB09 then?
    – bad_coder
    Oct 18 at 23:46
  • 17
    @bad_coder That's the joke.
    – Catija StaffMod
    Oct 18 at 23:50
  • 4
    @bad_coder SE's "New York" datacenter is physically located in New Jersey. (For a long time, it was actually in New York, but they moved and never changed the name to avoid breaking stuff.) Oct 19 at 0:58
  • I've cleared cache and comments are still broken. Also, the dynamic header bar icons (messages, achievements) still don't work. Same console error I reported above: Uncaught TypeError: Cannot read properties of null (reading 'innerText')
    – isherwood
    Oct 19 at 13:50
  • @Benjamin Are there still issues with this? Still having multiple problems
    – Nick
    Oct 19 at 14:06
  • Per this update, all my issues on SO seem resolved.
    – isherwood
    Oct 19 at 15:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .