10

Here on my network, the WebSocket's port 80 is disconnected. So, the WebSocket connection for Stack Exchange sites does not work. It throws an error in the console always as shown below:

WebSocket connection to 'ws://sockets.ny.stackexchange.com/' failed: Unexpected response code: 502

But yesterday, 3rd Sept, 2013, the WebSocket hasn't thrown any error. It worked.

I confirmed from my network administrator that he has nothing to do with how it worked yesterday (though port 80 was still disconnected with WebSocket).

I am not asking why my WebSocket is throwing an error, but I am asking: why did it work properly yesterday?

If there is a hand of a Stack Exchange developer to make it work properly then I request them to keep those changes forever. So that the WebSocket will work for me too :D.

3
  • 5
    It was us, I'll do a full answer after the gym in a few hours...your sockets will return. Oct 4, 2013 at 10:14
  • @NickCraver you there? the websocket problem is in my office so I can't test till Monday.
    – Mr_Green
    Oct 4, 2013 at 15:09
  • I've been having the same problem, and I noticed that it was fixed yesterday. It's broken again today. I'd be happy to test any fixes. Oct 4, 2013 at 15:18

1 Answer 1

11

One issue we've had greatly delaying SSL is a bug in which when we reach a max socket allocation on the kernel for our load balancers from a single source. In retrospect it's a simple problem, just was a very convoluted one to find.

Why does this matter? Because many workplace or just plain bad proxies don't handle websockets correctly. They stick their nose in, don't understand what's happening and just break things. With SSL websockets they can't do that, they have to just forward the traffic along without interfering - which is what appears to be happening in your case and is the primary reason we're so adamant about deploying SSL for websockets.

Detailed answer:

First a preface: there is still a separate case of a proxy purposefully sitting in the way with either a local trusted or just bad certificate...you're out of luck in that case, there's nothing we can do if it's behaving badly.

So on our side, your SSL connection does this:

You --> SSL terminator --> HTTP front-end --> Webserver

It's more fun than that with the details, but that's generally what happens. Now what we're doing is replacing that nginx SSL terminator with native HAProxy (and enabling SSL for many more things at the same time). When doing so we have inter-process termination and proxy forwards from a few processes running SSL termination to the HTTP front-end's process, all happening on localhost. What was happening is when we reached a huge load of concurrent connections (yay websockets!) you exhaust the ephemeral points available for that local to local connection. If you want a more detailed explanation, Alecco Locco has an excellent one here.

This isn't a problem on Stack Overflow even, where we have ~about 1k simultaneous connections (we render really fast, so connections are brief). However, websockets stay around and we have 130k simultaneous connections as I write this.

So, how do we solve that? Well the answer's pretty simple, we expand that Terminator --> Front-end binding to be across more IPs/ports in HAProxy via round-robin, each binding can handle about 25k concurrent connections with default ephemeral limits, so we spread the load across n bindings.

Yesterday we ran a long-term test of this fix, and turn it off last night to patch our load balancers to the latest...well, everything. Tom is going to start porting our HAProxy module to Hiera and getting the latest configs sorted out. Hopefully soon after our New York meetup (starting in 9 days) we will be rolling out SSL websockets permanently.

2
  • Nick, the web-socket error has started again.. is it temporary?
    – Mr_Green
    Oct 15, 2013 at 12:47
  • It will happen until next week or the one after...we are working on the SSL rollout Oct 15, 2013 at 14:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .