15

From the symptoms, this is identical to the issue I reported in Random pages across SE feel "dead", due to SE WebSocket misbehaving: sometimes connecting, sending heartbeat, but no data, which was resolved, but has now returned. That post has considerably more background information, which I won't repeat here.

About 10% of the time when one connects to the SE WebSocket, the WebSocket is non-functional in a way that is only identifiable by the lack of valid data. These non-functional WebSockets result in SE pages feeling "dead". The pages get no updates for votes, post edits, deletions, new comments, etc. The problem occurring on a particular page load is random. If the page is reloaded, the problem will disappear, about 90% of the time (i.e. if the WebSocket connection is closed and reestablished).

The basic problem is that Stack Exchange's WebSocket, which SE uses to communicate from their servers to JavaScript running on their webpages, will, about 10% of the time, correctly connect, generate no actual errors, but never deliver any actual data, other than a heartbeat message1, which SE uses to determine if the page is still listening.

You can use the code in the snippet below to test for the issue. I'm currently seeing the following error rate:

enter image description here

I initially identified the prior occurrence of this issue due to how it affected SmokeDetector. While SmokeDetector is likely seeing the current problem, code was added to SmokeDetector the first time I reported this issue to work around the problem and continue normal operation. This time, I noticed it in SE pages and identified it due to having previously investigated.

Mitigation

The code to mitigate the issue for SmokeDetector has been running since shortly after my first report, so this only affects SmokeDetector minimally. For normal viewing of Stack Exchange pages, just reload the page.

Duplication

Testing

The code below opens a WebSocket and sends a subscription to the main "active" WebSocket feed. A test round is considered to pass if data is received on the WebSocket prior to the heartbeat message,1 which SE sends 5 minutes after the WebSocket is opened. If the WebSocket sees the heartbeat without having seen any other valid messages, then that is considered a failure. The code then closes the WebSocket and runs another round.

It's possible that activity on all of SE is low enough such that no data should have been seen in the 5 minutes between the test round's WebSocket being established and the heartbeat signal. However, that is very unlikely. If it did occur, then SmokeDetector would also see that lack of activity and reboot, while reporting that fact into Charcoal HQ

You can try connecting to the SE WebSocket for yourself using the Stack Snippet below:

(() => {
  'use strict';

  let webSocket = null;
  let noAdd = true;
  let startTime = 0;
  const passEl = document.getElementById('pass');
  const failEl = document.getElementById('fail');
  const failPercentEl = document.getElementById('failPercent');
  const startTimeEl = document.getElementById('startTime');
  const elapsedTimeEl = document.getElementById('elapsedTime');
  const diffLabel = {
      days: 'd',
      hours: 'h',
      minutes: 'm',
      seconds: 's',
  };
  const diffOrder = [
      'days',
      'hours',
      'minutes',
      'seconds',
  ];

  function incrementAndShowPercent(addPass, addFail) {
    if (noAdd) {
      return;
    }
    const pass = +passEl.textContent + addPass;
    passEl.textContent = pass;
    const fail = +failEl.textContent + addFail;
    failEl.textContent = fail;
    const failPercent = Math.round((10000 * fail) / (pass + fail))/100
    failPercentEl.textContent = failPercent;
    if (fail) {
      failEl.classList.add('isFailing');
      failPercentEl.classList.add('isFailing');
    }
    /* The elapsed time code was copied, then significantly modified, from
     * https://codereview.stackexchange.com/a/160240/53535 by Przemek
     * https://codereview.stackexchange.com/users/97934/przemek */
    const elapsed = (Date.now() - startTime) / 1000;
    const diff = {
      days: Math.floor(elapsed / 86400),
      hours: Math.floor((elapsed / 3600) % 24),
      minutes: Math.floor((elapsed / 60) % 60),
      seconds: Math.floor(elapsed % 60),
    };
    const elapsedText = diffOrder.reduce((sum, key) => {
      if (sum || diff[key] || key === 'seconds') {
          sum += ` ${diff[key]}${diffLabel[key]}`;
      }
      return sum;
    }, '').trim();
    elapsedTimeEl.textContent = elapsedText;
  }

  function registerWebSocket() { // eslint-disable-line no-unused-vars
    if (webSocket) {
      if (typeof webSocket.removeEventListener === 'function') {
          //There can be additional events that mess up the count.
          webSocket.removeEventListener('message', socketOnMessage);
      }
      if (typeof webSocket.close === 'function') {
          // Close any existing WebSocket.
          webSocket.close();
      }
    }
    webSocket = new WebSocket('wss://qa.sockets.stackexchange.com');
    webSocket.addEventListener('message', socketOnMessage);
    webSocket.addEventListener('open', () => {
      //console.error('WS: Open:');
      webSocket.send('155-questions-active');
      console.log('vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv');
      console.log('WS: Open & Subscribed to active questions');
      noAdd = false;
    });
    webSocket.addEventListener('close', event => {
      //console.log('WS: Closed: event.code', event.code, '::  event:', event);
      console.log('WS: Closed');
    });
    webSocket.addEventListener('error', event => {
      console.error('WS: Error:');
      console.log('WS: Error: event', event);
    });
  }

  function socketOnMessage(event) {
    const messageObject = JSON.parse(event.data);
    const data = messageObject.data;
    let dataObject = data;
    try {
      dataObject = JSON.parse(data);
    } catch(error) {
      // Ignore any errors
    }
    messageObject.data = dataObject;
    console.log('WS: messageObject:', messageObject);
    if (messageObject.action === 'hb') {
      //At least for the 155-questions-active socket, the hb is every 5 minutes.
      incrementAndShowPercent(0, 1);
      noAdd = true;
      console.error('WS: heartbeat Received:');
      //console.error('WS: Responding to heartbeat:');
      //webSocket.send(data);
      registerWebSocket();
    } else {
      incrementAndShowPercent(1, 0);
      noAdd = true;
      registerWebSocket();
    }
  }

  document.getElementById('start').addEventListener('click', () => {
    passEl.textContent = 0;
    failEl.textContent = 0;
    failEl.classList.remove('isFailing');
    failPercentEl.textContent = 0;
    failPercentEl.classList.remove('isFailing');
    startTime = Date.now();
    startTimeEl.textContent = (new Date(startTime)).toISOString();
    registerWebSocket();
  });

  document.getElementById('stop').addEventListener('click', () => {
    webSocket.close();
  });
})();
span:not([id]) {
  margin-left: 10px;
}
span.extraSpace {
  margin-left: 20px;
}
#fail.isFailing,
#failPercent.isFailing {
  color: red;
  font-weight: bold;
}
#pass,
#failPercent {
  color: #00d000;
  font-weight: bold;
}
<button id="start">Start/Restart</button>
<button id="stop">Stop</button>
<span> Pass: <span id="pass">0</span></span>
<span> Fail: <span id="fail">0</span></span>
<span>Fail %: <span id="failPercent">0</span> %</span>
<span class="extraSpace">Started: <span id="startTime"></span></span>
<span class="extraSpace">Elapsed: <spam id="elapsedTime"></span></span>
</br>You may want to open the Web Console (F12), as it's easier to read the output there.</br>
If you don't see anything in the console for 5 minutes after "WS: Open & Subscribed to active questions", that means that attempt to connect (probably) failed.</br>


  1. The "heartbeat" is a message that SE sends which says "hb". After receiving the heartbeat, the consumer of the WebSocket must send some data to indicate that the WebSocket is still active. The "heartbeat" is administrative and doesn't have any data for the user to consume.
15
  • 4
    I suspect the post here about 19 hours ago of The list of related questions is shown empty is related to this issue. Also, I've noticed the same problem happening again for me as when the problem occurred earlier, e.g., where the list of newest questions of this site sometimes doesn't show me the latest ones until I refresh the page. – John Omielan Jul 10 '20 at 4:16
  • @Tantalus'touch. To get a more accurate failure rate, you really have to let the test run for an extended period of time. To classify a run as a failure requires 5 minutes of no activity on the WebSocket, but each run that's a pass can take as little a second or two, up to almost 5 minutes, depending on the level of activity on SE. Note: it's quite unusual for there to be minutes of time between activity on the SE WebSocket, but in testing with SmokeDetector, I have seen some times of the week when there can be 2 to 3 minutes of no activity. – Makyen Jul 10 '20 at 5:06
  • 1
    After 2 hours failure rate 10.4 % here. UK, England, SP:BT. – Tantalus' touch. Jul 10 '20 at 11:08
  • 1
    Oh, I kept it running for a while and forgot about it. I currently have “Pass: 222 Fail: 23 Fail %: 9.39 % Started: 2020-07-10T08:52:57.385Z Elapsed: 2h 14m 7s”. I have definitely also noticed that this issue re-appeared in the past few weeks. – Sebastian Simon Jul 10 '20 at 12:19
  • 1
    Has this been fixed now? On a second run I had about 2300 passes and 0 failures. – Sebastian Simon Jul 10 '20 at 15:22
  • 1
    “Pass: 577 Fail: 0 Fail %: 0 % Started: 2020-07-10T15:23:53.764Z Elapsed: 42m 1s” — seems status-completed for now? – Sebastian Simon Jul 10 '20 at 16:06
  • 2
    @user289905 I, also, am no longer seeing errors. I suspect something happened on SE's side at about 11:11 UTC, which is when the testing I had running saw the WebSocket be closed from SE's side. That implies they did some work on the servers at about that time. – Makyen Jul 10 '20 at 16:49
  • 2
    19.36 Z, n=967, 0% fail. Seems resolved, will keep an eye on it through the next few days maybe more. – Tantalus' touch. Jul 10 '20 at 21:03
  • 1
    Pass: 1044 Fail: 0 Fail %: 0 % Started: 2020-07-11T06:37:54.299Z Elapsed: 2h – rene Jul 11 '20 at 8:38
  • 1
    Pass: 2353 Fail: 0 Fail %: 0 % Started: 2020-07-11T06:37:54.299Z Elapsed: 4h 20m 40s – rene Jul 11 '20 at 10:59
  • 1
    Pass: 3481 Fail: 0 Fail %: 0 % Started: 2020-07-11T06:37:54.299Z Elapsed: 6h 11m 30s – rene Jul 11 '20 at 12:49
  • 2
    It definitely looks like the issue has been resolved. The yesterday's 11:11 UTC time I mentioned above also corresponds to the approximate time, 2020-07-10T11:38:20Z, when SmokeDetector experienced its WebSocket to SE being closed by SE's servers. So, SE doing work on the servers responsible for the WebSocket at that time is quite likely. – Makyen Jul 11 '20 at 13:11
  • Is this still an issue today? – Sonic the Curiouser Hedgehog Aug 10 '20 at 6:16
  • @SonictheMaskedWerehog I haven't been seeing this particular WebSocket issue. There have been a couple of times in the last week where SmokeDetector may have experienced it. SD auto-recovers from this issue without providing confirmation that this is the issue it's recovering from (i.e. it doesn't wait the full five minutes to see that there was no traffic; it waits a shorter period). – Makyen Aug 10 '20 at 6:42
  • @SonictheMaskedWerehog OTOH, I've been seeing another issue much more prevalently which could also be described as a "dead" page. The other issue is that jQuery isn't loading prior to SE's stub.<language_locale>.js file. When that happens, none of SE's JavaScript on the page is operational, and the page feels even more "dead" than when the WebSocket was broken. I've been this other issue several times a day over the last few days. I asked in SOCVR if anyone else was seeing the same problem, but nobody said they'd been seeing it, so I've assumed it's likely an issue on my machine/browser/etc. – Makyen Aug 10 '20 at 6:42

You must log in to answer this question.

Browse other questions tagged .