18

From the symptoms, this is identical to the issue I reported in:

Both of those were resolved, but the issue has now returned (again). The first post has considerably more background information, which I won't repeat here. The second post is largely similar to this question, but with error rates specific to what was seen at that time.

This issue, as with the other prior issues, about 11% of the time when one connects to the SE WebSocket, the WebSocket is non-functional in a way that is only identifiable by the lack of valid data. [Early testing indicated a wider range of possible values for % error rate (5% to 25%). Longer and more recent testing appears to be converging to about 11%.] These non-functional WebSockets result in SE pages feeling "dead". The pages get no updates for votes, post edits, deletions, new comments, etc. If the problem occurs on a particular page load is random. If the page is reloaded, the problem will disappear, about 80% of the time (i.e. if the WebSocket connection is closed and reestablished).

The basic problem is that Stack Exchange's WebSocket, which SE uses to communicate from their servers to JavaScript running on their webpages, will, about 11% of the time, correctly connect, generate no actual errors, but never deliver any actual data, other than a heartbeat message1, which SE uses to determine if the page is still listening.

You can use the code in the snippet below to test for the issue. I'm currently seeing the following error rates:

Results after a few hours:

Test A (started about 1 hour prior to test B):
WebSocket testing results showing 348 pass results and 45 failures for a 11.45% failure rate.

Test B:
WebSocket testing results showing 246 pass results and 33 failures for a 11.83% failure rate.

I initially identified the prior occurrence of this issue due to how it affected SmokeDetector. While SmokeDetector is likely seeing the current problem, code was added to SmokeDetector the first time I reported this issue to work around the problem and continue normal operation. This time, I noticed it in SE pages and identified it due to having previously investigated.

Mitigation

The code to mitigate the issue for SmokeDetector has been running since shortly after my first report, so this only affects SmokeDetector minimally. For normal viewing of Stack Exchange pages, just reload the page.

Duplication

Testing

The code below opens a WebSocket and sends a subscription to the main "active" WebSocket feed. A test round is considered to pass if data is received on the WebSocket prior to the heartbeat message,1 which SE sends 5 minutes after the WebSocket is opened. If the WebSocket sees the heartbeat without having seen any other valid messages, then that is considered a failure. The code then closes the WebSocket and runs another round.

It's possible that activity on all of SE is low enough such that no data should have been seen in the 5 minutes between the test round's WebSocket being established and the heartbeat signal. However, that is very unlikely. If it did occur, then SmokeDetector would also see that lack of activity and reboot, while reporting that fact into Charcoal HQ

You can try connecting to the SE WebSocket for yourself using the Stack Snippet below:

(() => {
  'use strict';

  let webSocket = null;
  let noAdd = true;
  let startTime = 0;
  const passEl = document.getElementById('pass');
  const failEl = document.getElementById('fail');
  const failPercentEl = document.getElementById('failPercent');
  const startTimeEl = document.getElementById('startTime');
  const elapsedTimeEl = document.getElementById('elapsedTime');
  const diffLabel = {
      days: 'd',
      hours: 'h',
      minutes: 'm',
      seconds: 's',
  };
  const diffOrder = [
      'days',
      'hours',
      'minutes',
      'seconds',
  ];

  function incrementAndShowPercent(addPass, addFail) {
    if (noAdd) {
      return;
    }
    const pass = +passEl.textContent + addPass;
    passEl.textContent = pass;
    const fail = +failEl.textContent + addFail;
    failEl.textContent = fail;
    const failPercent = Math.round((10000 * fail) / (pass + fail))/100
    failPercentEl.textContent = failPercent;
    if (fail) {
      failEl.classList.add('isFailing');
      failPercentEl.classList.add('isFailing');
    }
    /* The elapsed time code was copied, then significantly modified, from
     * https://codereview.stackexchange.com/a/160240/53535 by Przemek
     * https://codereview.stackexchange.com/users/97934/przemek */
    const elapsed = (Date.now() - startTime) / 1000;
    const diff = {
      days: Math.floor(elapsed / 86400),
      hours: Math.floor((elapsed / 3600) % 24),
      minutes: Math.floor((elapsed / 60) % 60),
      seconds: Math.floor(elapsed % 60),
    };
    const elapsedText = diffOrder.reduce((sum, key) => {
      if (sum || diff[key] || key === 'seconds') {
          sum += ` ${diff[key]}${diffLabel[key]}`;
      }
      return sum;
    }, '').trim();
    elapsedTimeEl.textContent = elapsedText;
  }

  function registerWebSocket() { // eslint-disable-line no-unused-vars
    if (webSocket) {
      if (typeof webSocket.removeEventListener === 'function') {
          //There can be additional events that mess up the count.
          webSocket.removeEventListener('message', socketOnMessage);
      }
      if (typeof webSocket.close === 'function') {
          // Close any existing WebSocket.
          webSocket.close();
      }
    }
    webSocket = new WebSocket('wss://qa.sockets.stackexchange.com');
    webSocket.addEventListener('message', socketOnMessage);
    webSocket.addEventListener('open', () => {
      //console.error('WS: Open:');
      webSocket.send('155-questions-active');
      console.log('vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv');
      console.log('WS: Open & Subscribed to active questions');
      noAdd = false;
    });
    webSocket.addEventListener('close', event => {
      //console.log('WS: Closed: event.code', event.code, '::  event:', event);
      console.log('WS: Closed');
    });
    webSocket.addEventListener('error', event => {
      console.error('WS: Error:');
      console.log('WS: Error: event', event);
    });
  }

  function socketOnMessage(event) {
    const messageObject = JSON.parse(event.data);
    const data = messageObject.data;
    let dataObject = data;
    try {
      dataObject = JSON.parse(data);
    } catch(error) {
      // Ignore any errors
    }
    messageObject.data = dataObject;
    console.log('WS: messageObject:', messageObject);
    if (messageObject.action === 'hb') {
      //At least for the 155-questions-active socket, the hb is every 5 minutes.
      incrementAndShowPercent(0, 1);
      noAdd = true;
      console.error('WS: heartbeat Received:');
      //console.error('WS: Responding to heartbeat:');
      //webSocket.send(data);
      registerWebSocket();
    } else {
      incrementAndShowPercent(1, 0);
      noAdd = true;
      registerWebSocket();
    }
  }

  document.getElementById('start').addEventListener('click', () => {
    passEl.textContent = 0;
    failEl.textContent = 0;
    failEl.classList.remove('isFailing');
    failPercentEl.textContent = 0;
    failPercentEl.classList.remove('isFailing');
    startTime = Date.now();
    startTimeEl.textContent = (new Date(startTime)).toISOString();
    registerWebSocket();
  });

  document.getElementById('stop').addEventListener('click', () => {
    webSocket.close();
  });
})();
span:not([id]) {
  margin-left: 10px;
}
span.extraSpace {
  margin-left: 20px;
}
#fail.isFailing,
#failPercent.isFailing {
  color: red;
  font-weight: bold;
}
#pass,
#failPercent {
  color: #00d000;
  font-weight: bold;
}
<button id="start">Start/Restart</button>
<button id="stop">Stop</button>
<span> Pass: <span id="pass">0</span></span>
<span> Fail: <span id="fail">0</span></span>
<span>Fail %: <span id="failPercent">0</span> %</span>
<span class="extraSpace">Started: <span id="startTime"></span></span>
<span class="extraSpace">Elapsed: <spam id="elapsedTime"></span></span>
</br>You may want to open the Web Console (F12), as it's easier to read the output there.</br>
If you don't see anything in the console for 5 minutes after "WS: Open & Subscribed to active questions", that means that attempt to connect (probably) failed.</br>


  1. The "heartbeat" is a message that SE sends which says "hb". After receiving the heartbeat, the consumer of the WebSocket must send some data to indicate that the WebSocket is still active. The "heartbeat" is administrative and doesn't have any data for the user to consume.
  • On my home boxx: Pass: 47 Fail: 3 Fail %: 6 % Started: 2020-10-13T17:36:48.528Z Elapsed: 18m 44s – rene Oct 13 at 17:57
  • 9
    The 10-20% would be news to me, but a failure rate of ~11% would be explained by a ny-web05 issues we've seen on an off this week due to a series of cascading events. We're digging into it, but there's a few interlinked things causing the socket failure and we're not there yet...hopefully get time to wrap up that dig tomorrow (laptop death has limited me a bit this week). – Nick Craver Oct 13 at 18:04
  • Pass: 137 Fail: 12 Fail %: 8.05 % Started: 2020-10-13T17:36:48.528Z Elapsed: 1h 9m 54s – rene Oct 13 at 18:49
  • Pass: 17 Fail: 5 Fail %: 22.73 % Started: 2020-10-13T18:44:01.757Z Elapsed: 26m 23s – Luuklag Oct 13 at 19:10
  • Second run: Pass: 79 Fail: 5 Fail %: 5.95 % Started: 2020-10-13T19:33:12.698Z Elapsed: 31m 42s Which makes me think that the 11% @NickCraver mentioned might be about right. Due to the long time out period, shorter runs like this are difficult to use to establish a solid percentage, as the dead returns heavily influence the percentage on short time intervals – Luuklag Oct 13 at 20:05
  • A couple of hours ago, I started up an additional test run. That one is showing: Pass: 170 Fail: 20 Fail %: 10.53 % Started: 2020-10-13T18:21:41.176Z Elapsed: 1h 54m 21s. My first one (actually 2nd), which is still running appears to also be converging on a ~10% error rate (down from ~20%): Pass: 220 Fail: 33 Fail %: 13.04 % Started: 2020-10-13T17:14:00.255Z Elapsed: 3h 1m 55s. So, the initial ~20% appears to be a blip, which is not unexpected given that the errors are random. – Makyen Oct 13 at 20:23
  • 2
    @Luuklag Yeah, the it's strongly dominated by the 5 minute timeout to declare a failure. I could re-write the test to open multiple WebSocket connections at the same time and use the passing WebSockets to declare the failing one(s) as failures well prior to the 5 minute timeout. That would result in converging on an accurate error rate much more rapidly. While that would be fun/interesting, I have way too many other things I should be spending time on. As it is, the code achieves its primary purpose: demonstrate there's a problem on SE's side of the WebSocket, which they should look at. – Makyen Oct 13 at 20:31
  • 5
    @Xnero If you look at the comments on the original question I posted about this, it was stated that SE prefers a new question be posted if an issue reoccurs. I'm fine with having only a single question. That was, in fact, my first inclination. However, having thought about it, I can certainly see why they would want a new question for an issue that reoccurred, as the previous issues were resolved at some time after posting and a new question acts as a new problem ticket. – Makyen Oct 13 at 22:08
  • @NickCraver, I think I've noticed this in the Review Queue as well. It will say there is a review available but there's none for me - IF I either wait a few minutes and refresh or go to the History, choose an old deletion reason for a prior review, then use the [Next] button then the review is immediately presented; so it's not a review that I'm not eligible for. --- This happens least frequently (never?) in the Close queue and almost always in the LQ queue, and infrequently in the other two. – Rob Oct 13 at 22:17
  • I can reproduce the results. 10.74% after 2.5h and still running. It goes a bit between 8 and 15 it seems, but blips outside of those bounds should be expected. – Mast Oct 14 at 11:32
  • Pass: 503 Fail: 54 Fail %: 9.69 % Started: 2020-10-14T09:05:07.823Z Elapsed: 5h 5m 17s – Mast Oct 14 at 14:10
  • 1
    @Xnero No, I didn't. – Makyen Oct 14 at 17:05
  • 1
    @Xnero I don't know. I'm sorry, but I don't have any more information about deleted comments on MSE than you or any other user who's not an MSE moderator, CM, or employee with moderator access. All I know is: I saw it; IIRC, it was upvoted, most likely as an automatic part of the other close-vote; and it's now deleted. – Makyen Oct 14 at 20:40
  • 1
    Thanks for all the stats dumps here - our web sockets are spread across 9 servers, so with one being wonky we'd expect to see a ~11% bad behavior rate. Given the above dumps, that looks to be exactly the case. But, this is something very hard to repro, they've been running for weeks. So instead of a quick fix, we want to narrow it down and figure out what's wrong where so we can fix the root. Is anyone in chat we could run tests with today? We can try rotating out each node to find the culprit faster. We can do this without, but your test suite already going would be very useful and welcome. – Nick Craver Oct 15 at 11:19
  • Pass: 2928 Fail: 0 Fail %: 0 % Started: 2020-10-15T15:28:28.693Z Elapsed: 3h 12m 46s – rene Oct 15 at 18:41

You must log in to answer this question.

Browse other questions tagged .