Loading posts in real time is broken in some cases due to a breaking change in the format data is returned

Question

At some point in the last 16, or so, hours a change was made to the data returned from the Stack Exchange internal /posts/ajax-load-realtime/{list of semicolon separated postIDs} endpoint. The response changed from plain HTML text to a JSON encoded Object with at least two properties: Html and VotesCastJson. Unfortunately, when this change was made, SE's codebase wasn't searched for every place where this endpoint was used and/or those places were not actually changed.

I haven't done an exhaustive check to see everywhere which this change breaks things (e.g. there may additional things which relied upon the internal HTML structure provided). In addition, there have been reports of other issues in other areas of the site which could easily be related to this or similar endpoint changes. However, one place where this endpoint is used is StackExchange.realtime.reloadPosts([array of post IDs]). The code for that function assumes that it's still going to receive straight HTML text, rather than a JSON encoded Object with the HTML text in the Html property. This, of course, breaks everything which relies on StackExchange.realtime.reloadPosts().

You can test to see if this is working by opening up the browser console for this question, or any question, and typing StackExchange.realtime.reloadPosts([$('.question').data('questionid')]), which should reload the HTML for this question. At the time of posting this question, instead of the correct operation it will produce a Syntax Error.

Note: the new VotesCastJson property appears to be intended to reduce the need to call the /posts/${postId}/votes endpoint to get the votes for the post that's being reloaded, but the data returned in that property is invalid in all cases which I've checked.

Answer 1

This has been fixed. This was a bug that came up during some modulification. You all are exactly right. There was a mismatch in what was being sent vs. what was expected. The original change caught it in most places, but it missed this one. This change went out on Monday.

I took a quick search through related JavaScript code and I think this was the only place that was broken. If there's more that's broken that we've missed, please let us know!

Thanks to both Makyen and VLAZ for making this easy to reproduce and fix. I really appreciate it.

Answer 2

AS OF 2022-03-30 THE ISSUE SEEMS TO BE FIXED BY STACK EXCHANGE

THE USERSCRIPT IS NOT NEEDED ANY MORE. IT CAN BE SAFELY UNINSTALLED.

Makyen's fix does not work for me. I suspect there was an update that broke it but did not fix the behaviour. The script is correct - it monkey patches the StackExchange.realtime.reloadPosts() method, however the problem is that internally the SE code directly calls the function:

if ($('#review-content').length === 0) {
    reloadPosts([post.id]);
}

Instead of using the (overridden) method call.

(see Line 10886 in the dev distribution of full.js)

I made a fix via a different route - using $.ajaxSetup and the dataFilter property to directly get the data before it is processed and attempt to extract the HTML portion of it.

Install the userscript (Source on GitHub)

Same guidance as Makyen's:

Once SE actually fixes the issue in their code, you'll need to disable or uninstall the userscript.

Code

// ==UserScript==
// @name          Quick and dirty fix for reloading posts 
// @namespace     StackExchangeFix
// @description   Hook directly through jQuery and Interceptdata returned from /ajax-load-realtime/ to pre-process for later calls. 
// @match         https://stackoverflow.com/*
// @match         *://*.stackoverflow.com/*
// @match         *://*.superuser.com/*
// @match         *://*.serverfault.com/*
// @match         *://*.askubuntu.com/*
// @match         *://*.stackapps.com/*
// @match         *://*.stackexchange.com/*
// @match         *://*.mathoverflow.net/*
// @exclude       *://api.*
// @exclude       *://blog.*
// @exclude       *://chat.*
// @exclude       *://data.*
// @exclude       *://stackoverflow.com/advertising*
// @exclude       *://stackoverflow.com/talent*
// @exclude       *://stackoverflow.com/teams*
// @grant         none
// @version       1.1
// @author        VLAZ
// @run-at        document-end
// ==/UserScript==


(function() {
  const equalCaseInsensitive = (a, b) =>
    a?.localeCompare(b, undefined, {sensitivity: "base"}) === 0;
  
  function patchReload(data, type) {
      //only intercept reloads
      if (!this.url.startsWith("/posts/ajax-load-realtime/"))
        return data;

      //these requests seem broken as they claim the result is HTML but it's actually JSON
      if (!equalCaseInsensitive(type, "html"))
        return data;

      try {
        //try to parse
        const realData = JSON.parse(data);

        //verify it's the correct shape
        if ("Html" in realData && "VotesCastJson" in realData)
            return realData.Html;

        return data;
      } catch (e) { 
        // parsing error: ignore and return the original
        return data;
      }
    }

  const prevDataFilter = $.ajaxSetup().dataFilter || (x => x);
  
  const passThroughFix = function(...args) {
    //apply fix before passing the data downstream
    const newData = patchReload.apply(this, args);
    
    const newArgs = [newData].concat(args.slice(1));
    
    //apply downstream filter if any    
    return prevDataFilter.apply(this, newArgs);
  };

  $.ajaxSetup({ dataFilter: passThroughFix });
})();