Prevent specific sites from being overrepresented in the hot questions list [duplicate]

Question

Some sites appear far more often in the hot questions list than other sites. I don't have any hard data on that, as that kind of data is just not publicly available, but I'm pretty sure my subjective impression on that is reasonably accurate.

Those sites don't appear more often because they are inherently more interesting, but because they have certain voting and answering patterns that favor them in the hot questions algorithm. I would argue that the goal of the hot questions list is to present questions from a large variety of SE sites to the whole network, and to show possibly interesting questions to users that would not have found them otherwise.

To achieve this goal, the spots in the hot questions list should be distributed among all SE sites. The algorithm should try to get a good sampling of all sites, and it should try to normalize the results for varying voting behaviour. At the moment, sites that intrinsically get more votes and more answers per question seem favored in the algorithm.

So I think one part of the algorithm should try to identify questions that get an unusally high amount of votes and views for that site, and give the smaller sites a chance this way.

I'm not sure on how exactly this weighting should be done, as a complete normalization would likely not work across the huge difference in scale for all the SE sites.

As a specific example that the current algorithm is broken I'll present the hot questions list from right now, where Math.SE provides 11 our of the top 30 questions. I don't think any site should occupy a third of the whole list.

Here's another instance where one site is overrepresented at the top of the list:

Answer 1

It was bad

Old system was designed so that it tended to inaccurately favor questions from sites that differ much from Stack Overflow, particularly smaller and ones of conceptual / subjective-ish nature (for example, Programmers and Workplace).

Compared to SO, smaller sites have much less power to handle content moderation problems involved in hot questions. Additional problems are caused by the fact that answer flagging system is heavily, unfairly tilted to serve only answers to concrete technical questions.

Combined, these factors make it so that amount of answers in hot questions at some sites tends to be higher compared eg to SO. Since hotness formula rather indiscriminately factors amount of answers into the score, questions from smaller / subjective-ish sites tend to get higher score which pushes them closer to the top of hot list, compared to eg those from Stack Overflow.

This often had an effect of some sites looking like overrepresented in hot list.

It is not right there now

Recently, substantial changes were made to hot list selection (SE team even claims that changes made managed to resolve long standing, painful issue of sticky hot questions). However substantial were the changes, it would be wrong to assume that these resolved the issue with overrepresented sites.

This is first because selection algorithm is neither published nor finalized, not even logged, so it would be generally wrong to make any assumptions on how it works - except for those based on official claims of SE team (note that even these claims are better be tested, as these may be based on incorrectly understood / implemented algorithm).

Second, and probably most important, is that whatever efforts are under way, corrections to address the issues outlined here are neither their goal nor a priority (note how this very request has none of the status-* tags on it). As a result, one should not expect changes to anyhow resolve the issue of overrepresented sites.

Per my observations, mentioned issues indeed remain even after all the changes, see eg screen shot made yesterday, with 4 questions from single site (Workplace) bumped into sidebar:

You can make a difference

Given that SE team does not seem to be interested to address this issue, it looks appropriate to point out that there is a way for concerned users to put their own effort into addressing this issue.

For this, take into account that hot questions listed at sidebar are indeed picked from a larger source list, link to which is provided in sidebar list header. This larger list contains 100 questions that are spread fairly evenly across different sites - in other words, there is no over/under-representation there.

In mentioned source list, one can pick posts from some high-traffic site that seems underrepresented on sidebar (Stack Overflow looks a prominent example) and vote them up so that these posts get more exposure (assuming enough reputation to upvote or association bonus on that site).

• With more exposure and more eyeballs, these posts get good chance to collect even more upvotes from passers by visiting it from sidebar, which even can turn this into self-sustaining process, similar to one that caused effect of "sticky hot" questions mentioned above. In theory, there is an "aging factor" in hotness formula that is intended to tune this down but in practice it turns out inefficient in many hot questions.

Pushing more questions from underrepresented site(s) higher in the hot list decreases exposure of overrepresented sites.

This sort of balances the load carried by hot questions and redirects that load to where it can supposedly be handled better - to larger sites, having more moderators, 10K and 20K users and better options to manage troublesome answers brought in by visitors attracted from hot list.

The more one upvotes, the more is the impact. This is especially true when question has 4, 5 or more answers and is less than 7 hours old. Given formula details, with a bit of luck, even a single voter can make a surprising bump of the question to the top of hot list (giving it more chances to start collecting upvotes from others and stick in hot list longer).

Answer 2

The simplest way to prevent hot questions monopolization would be to decrease the weight of the subsequent hot questions from the same site by the factor dependent on the number of hot questions already featured.

Let's say the factor would be linear. First 2 questions doesn't change anything, but the 3rd question from the same site would have 'hotness' decreased by 2, the fourth by 3 etc.

Example:

• Math has 4 questions of hotness 800, 700, 600, 500
• SuperUser has 3 questions of hotness 600, 500, 400
• Travel and Outdoors have 2 questions of hotness 350, 300
• there are 6 slots in hotness formula

Solution:

• Math questions get 2 slots (with hotness 800 and 700)
• Weight of 3rd and 4th questions on Math is reduced to 300 and 250, so the next 2 slots get SuperUser with weights 600 and 500
• Weight of 3rd question from SuperUser is reduced to 200, so the last 2 slots got a question from Travel and a question from Outdoors

Without weight reduction only Math and SuperUser would got the slots.

Answer 3

I agree that it would be nice to see more range in the hot questions list, but the algorithm as it stands is already a good representation of what's interesting and relevant to SE as a whole (questions of relative site scale notwithstanding).

Normalizing for different voting behaviors is interesting. Though those behaviors are likely to evolve over time since they're more cultural than codified. Studying the voting culture of each site over time could lead to a method for algorithmically abstracting it out of the equation.

(I have something to contribute in this regard but I'm currently on a bad connection. Edit coming soon with more details.)

---

The hotness algorithm: What formula should be used to determine "hot" questions?