13

Stack Exchange provides a way to watch a question (i.e. receive real-time information about up- and downvotes, new answers, new comments, edits, closures and deletions) via WebSockets. You just have to connect to qa.sockets.stackexchange.com and send the following message:

SITE_ID-question-QUESTION_ID

where SITE_ID is ID of the site the question belongs to (for example 1 for Stack Overflow), and QUESTION_ID is ID of the question.

I don't know if it's a bug or a feature, but you can watch multiple questions using a single WebSocket connection.

I would like to watch a large amount of questions, for example 10,000. Is this allowed?

Improve this question
  • I know there are restrictions in place specifically to deal with scrapers not throttling their requests at appropriate levels, so I'd expect that you'd end up with your IP blocked doing that kind of thing. – Servy May 10 '16 at 20:29
  • 1
    @Servy Yes, but I think that WebSockets don't require as much resources as HTTP requests, so the limit could be higher. – Michał Perłakowski May 10 '16 at 20:34
  • 2
    FWIW, Smokey had been holding on the question websockets for ten minutes after every report to monitor deletions. It's nowhere near ten thousand, but it's definitely more than one or two concurrently. – Undo May 10 '16 at 21:16
  • @Undo we could count that, just monitor how many reports we get at about 7 in the morning for ten minutes. – ArtOfCode May 10 '16 at 23:01
5

Because of lack of answer from any SE employee, I decided to test it by myself. Actually, it is possible to watch 10k questions with a single connection. However, you can't send too many messages at once — after sending about 450 messages without any delay, I was disconnected. Adding 1ms delay between messages is enough.

Of course the fact that the system allows to do this doesn't necessarily mean that it's OK to do this. I still await an official answer.

Improve this answer
3

At peak load, the SE servers see 500k simultaneous websocket connections. The average is more like 370k simultaneous.

A good rule of thumb: don't open so many sockets that you're a significant part of that number. Given the millions of users, significant is subjective, but I'd set it at no more than 1% of average load.

That gives you 3700 simultaneous we sockets to play with, which is probably more than enough for most purposes. If you're watching 10k questions, you may need to narrow down why you need to, and optimise the number you actually need to watch. Also - explicitly close the connections as soon as you can when you're done with the socket.

If you're watching questions on the same socket, that's more than enough - you can have 3333 open sockets, each watching 3 questions, to distribute the load.

I'm not an SE developer; this is my best estimate. Use at your own risk.

Improve this answer
  • I don't understand, why would I use more than one socket? – Michał Perłakowski May 11 '16 at 5:44
  • @Gothdo to spread the load across connections. – ArtOfCode May 11 '16 at 7:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .