From the symptoms, this is identical to the issue I reported in:
- Random pages across SE feel "dead", due to SE WebSocket misbehaving: sometimes connecting, sending heartbeat, but no data
- 2020-07-10 Happening again: Random pages across SE feel "dead", due to SE WebSocket misbehaving: sometimes connecting, sending heartbeat, but no data.
Both of those were resolved, but the issue has now returned (again). The first post has considerably more background information, which I won't repeat here. The second post is largely similar to this question, but with error rates specific to what was seen at that time.
This issue, as with the other prior issues, about 11% of the time when one connects to the SE WebSocket, the WebSocket is non-functional in a way that is only identifiable by the lack of valid data. [Early testing indicated a wider range of possible values for % error rate (5% to 25%). Longer and more recent testing appears to be converging to about 11%.] These non-functional WebSockets result in SE pages feeling "dead". The pages get no updates for votes, post edits, deletions, new comments, etc. If the problem occurs on a particular page load is random. If the page is reloaded, the problem will disappear, about 80% of the time (i.e. if the WebSocket connection is closed and reestablished).
The basic problem is that Stack Exchange's WebSocket, which SE uses to communicate from their servers to JavaScript running on their webpages, will, about 11% of the time, correctly connect, generate no actual errors, but never deliver any actual data, other than a heartbeat message1, which SE uses to determine if the page is still listening.
You can use the code in the snippet below to test for the issue. I'm currently seeing the following error rates:
Results after a few hours:
Test A (started about 1 hour prior to test B):
Test B:
I initially identified the prior occurrence of this issue due to how it affected SmokeDetector. While SmokeDetector is likely seeing the current problem, code was added to SmokeDetector the first time I reported this issue to work around the problem and continue normal operation. This time, I noticed it in SE pages and identified it due to having previously investigated.
Mitigation
The code to mitigate the issue for SmokeDetector has been running since shortly after my first report, so this only affects SmokeDetector minimally. For normal viewing of Stack Exchange pages, just reload the page.
Duplication
Testing
The code below opens a WebSocket and sends a subscription to the main "active" WebSocket feed. A test round is considered to pass if data is received on the WebSocket prior to the heartbeat message,1 which SE sends 5 minutes after the WebSocket is opened. If the WebSocket sees the heartbeat without having seen any other valid messages, then that is considered a failure. The code then closes the WebSocket and runs another round.
It's possible that activity on all of SE is low enough such that no data should have been seen in the 5 minutes between the test round's WebSocket being established and the heartbeat signal. However, that is very unlikely. If it did occur, then SmokeDetector would also see that lack of activity and reboot, while reporting that fact into Charcoal HQ
You can try connecting to the SE WebSocket for yourself using the Stack Snippet below:
(() => {
'use strict';
let webSocket = null;
let noAdd = true;
let startTime = 0;
const passEl = document.getElementById('pass');
const failEl = document.getElementById('fail');
const failPercentEl = document.getElementById('failPercent');
const startTimeEl = document.getElementById('startTime');
const elapsedTimeEl = document.getElementById('elapsedTime');
const diffLabel = {
days: 'd',
hours: 'h',
minutes: 'm',
seconds: 's',
};
const diffOrder = [
'days',
'hours',
'minutes',
'seconds',
];
function incrementAndShowPercent(addPass, addFail) {
if (noAdd) {
return;
}
const pass = +passEl.textContent + addPass;
passEl.textContent = pass;
const fail = +failEl.textContent + addFail;
failEl.textContent = fail;
const failPercent = Math.round((10000 * fail) / (pass + fail))/100
failPercentEl.textContent = failPercent;
if (fail) {
failEl.classList.add('isFailing');
failPercentEl.classList.add('isFailing');
}
/* The elapsed time code was copied, then significantly modified, from
* https://codereview.stackexchange.com/a/160240/53535 by Przemek
* https://codereview.stackexchange.com/users/97934/przemek */
const elapsed = (Date.now() - startTime) / 1000;
const diff = {
days: Math.floor(elapsed / 86400),
hours: Math.floor((elapsed / 3600) % 24),
minutes: Math.floor((elapsed / 60) % 60),
seconds: Math.floor(elapsed % 60),
};
const elapsedText = diffOrder.reduce((sum, key) => {
if (sum || diff[key] || key === 'seconds') {
sum += ` ${diff[key]}${diffLabel[key]}`;
}
return sum;
}, '').trim();
elapsedTimeEl.textContent = elapsedText;
}
function registerWebSocket() { // eslint-disable-line no-unused-vars
if (webSocket) {
if (typeof webSocket.removeEventListener === 'function') {
//There can be additional events that mess up the count.
webSocket.removeEventListener('message', socketOnMessage);
}
if (typeof webSocket.close === 'function') {
// Close any existing WebSocket.
webSocket.close();
}
}
webSocket = new WebSocket('wss://qa.sockets.stackexchange.com');
webSocket.addEventListener('message', socketOnMessage);
webSocket.addEventListener('open', () => {
//console.error('WS: Open:');
webSocket.send('155-questions-active');
console.log('vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv');
console.log('WS: Open & Subscribed to active questions');
noAdd = false;
});
webSocket.addEventListener('close', event => {
//console.log('WS: Closed: event.code', event.code, ':: event:', event);
console.log('WS: Closed');
});
webSocket.addEventListener('error', event => {
console.error('WS: Error:');
console.log('WS: Error: event', event);
});
}
function socketOnMessage(event) {
const messageObject = JSON.parse(event.data);
const data = messageObject.data;
let dataObject = data;
try {
dataObject = JSON.parse(data);
} catch(error) {
// Ignore any errors
}
messageObject.data = dataObject;
console.log('WS: messageObject:', messageObject);
if (messageObject.action === 'hb') {
//At least for the 155-questions-active socket, the hb is every 5 minutes.
incrementAndShowPercent(0, 1);
noAdd = true;
console.error('WS: heartbeat Received:');
//console.error('WS: Responding to heartbeat:');
//webSocket.send(data);
registerWebSocket();
} else {
incrementAndShowPercent(1, 0);
noAdd = true;
registerWebSocket();
}
}
document.getElementById('start').addEventListener('click', () => {
passEl.textContent = 0;
failEl.textContent = 0;
failEl.classList.remove('isFailing');
failPercentEl.textContent = 0;
failPercentEl.classList.remove('isFailing');
startTime = Date.now();
startTimeEl.textContent = (new Date(startTime)).toISOString();
registerWebSocket();
});
document.getElementById('stop').addEventListener('click', () => {
webSocket.close();
});
})();span:not([id]) {
margin-left: 10px;
}
span.extraSpace {
margin-left: 20px;
}
#fail.isFailing,
#failPercent.isFailing {
color: red;
font-weight: bold;
}
#pass,
#failPercent {
color: #00d000;
font-weight: bold;
}<button id="start">Start/Restart</button>
<button id="stop">Stop</button>
<span> Pass: <span id="pass">0</span></span>
<span> Fail: <span id="fail">0</span></span>
<span>Fail %: <span id="failPercent">0</span> %</span>
<span class="extraSpace">Started: <span id="startTime"></span></span>
<span class="extraSpace">Elapsed: <spam id="elapsedTime"></span></span>
</br>You may want to open the Web Console (F12), as it's easier to read the output there.</br>
If you don't see anything in the console for 5 minutes after "WS: Open & Subscribed to active questions", that means that attempt to connect (probably) failed.</br>- The "heartbeat" is a message that SE sends which says "hb". After receiving the heartbeat, the consumer of the WebSocket must send some data to indicate that the WebSocket is still active. The "heartbeat" is administrative and doesn't have any data for the user to consume.
