I think this is a bad idea, because the presence of close votes does not mean something is off topic the same way that closing a question does.
For example,
On The Great Outdoors we currently have a question with 4 close votes. However, when it was sent to the review queue, all of the votes were to leave it open and before the comments were cleared, the comment disagreeing with closing had been upvoted 11 times.
There is also a meta post on the question now, and the votes on that all point to leaving it open.
In this case, removing it from the Hot Network Questions list because 4 users think its off topic instead of listening to the majority who think its on topic and vote to leave it open would not be fair.
If 4 close votes were enough to remove it from the list, then what I would do is vote to close, instantly edit and vote to reopen and then send up the bat signal asking other users to reopen. That seems like a lot of unnecessary work and users might get confused as to why their question suddenly dropped off the list.
For any problems it may cause, the Hot Network List helps increase the reputation of users and encourage participation, which in turn will help the site keep on track.
It seems to me that this is trying to solve a different question, off topic questions making onto the list but this isn't going to solve that.
Instead
- Upvote good content so that there are more users who can vote to close.
- Write meta posts clearly stating what is and what is not on topic.
- Keep the site chat active so you can ask other users to vote to close off topic questions.
TLDR: Off topic questions getting onto the list isn't going to be solved by letting a minority of users determine what gets onto it.