Reorder questions picked for hot list based on adjusted hotness score (discard some answers by voting evidence)

Question

Executive Summary

The 100 questions are selected for the "Hot Network Questions" list displayed in the sidebar of each site. These questions are designed to showcase other sites in the Stack Exchange network.

Those questions should act as ambassadors from that site to the Stack Exchange Network, but due to the way the questions are calculated, bad data points result in poor examples staying on the list for a long time.

A previous request to fix this problem was marked as status-declined due to technical issues, so this question is requesting to change the order of the list without increasing the resources required to implement it.

TL;DR When votes of 20... 30... 100 users clearly indicate that only one or two answers are popular, it does not make sense to pretend that other answers are popular too.

Background

The Stack Exchange Network Hot Questions are viewed by hundreds of thousands of eyes. Many people's introduction to an SE site is influenced by the hot questions they see attributed to that site.

If the questions that are selected are good ambassadors for that site, the influx of new users is positive. If bad data points get selected, new users will follow the example of the hot question which can overwhelm budding community moderation, having a negative impact.

Bad Data Points

There are two things that can cause problems with the current hot questions list:

1. Ordering of "mature" questions in the list is driven by random bits of garbage picked by popular posts instead of answers with solid score.
2. Close votes are not taken into account. Note how this contrasts to close votes impact in twitter bot.

The current formula increases 'hotness' as the number of answers increases, even if voting evidence (score) indicates that those answers are of low interest. Bad answers are not an indication of a good question, and should not be used to determine the hotness of a question.

Questions that have close votes are treated as if there are no close votes. Smaller communities with questions asked at times when users with privileges are not around may have questions that would normally be closed getting upvotes before the community can moderate and improve the question. Close votes should be counted against questions in that list.

Technical Limitations

The previous feature-request was declined because the proposed formula would increase the resources required to make the query, and that was preventing its implementation.

Approach proposed here addresses above concerns by providing only a modification to the calculation and not any change to the database structure and it is limited only to fixed amount of questions, to guarantee that performance impact is under control.

Propose that the same parameters as before be used to select the 100 hottest questions, but the ranking of those 100 questions should be re-ordered with the following conditions:

• Answers with a score lower than (TopAnswerScore / 10) - 1 should be discarded from the hotness calculation
• Close votes on the question should reduce the total hotness of the question by 20% (so a closed question would have a 0 hotness score)

This will prevent questions from sticking to the top of the hot list too long by accumulating bad answers, and allow a couple quick close votes to prevent a bad example question from being broadcast across the network.

Answer 1

Yes, yes, and absolutely yes

Because of this question which got more views in 24 hours than we the entire site gets page views many days (question A), we have a lot of new users. That leads to people upvoting questions like this (question B) which are off-topic, has 4 close votes, but is still on the hot questions list!

Small communities cannot handle moderating thousands of new users coming in. We either need to allow:

1. Trusted users to protect/lock questions
2. Trusted users to add a tag to remove a question from the hot list
3. Mail notifications to community managers to be on-call in the site chatroom when a question gets over X amount of abnormal views

Or something. Like making this feature request happen. As-is you are hurting some communities with features that are supposed to be for our benefit, and it is unfortunate.

Update 10:49 UTC (or so) As of now, the question B has been put on hold, but is still popping up in the hot questions list...

Answer 2

For the sake of completeness, besides mentioned technical limitations decline justification listed some other points as well. These points were discussed in comments under decline message and for your convenience, a brief summary is provided below:

• in this case, not knowing the implementation details makes it almost impossible for you guys to solve this one.
  ...a lot of the research here is based on a false premise.

Feature request is based primarily on studies performed prior to ones addressed above:

• Don't let questions stick to the top of the hot questions list forever
• Answers quality in hot questions

Mentioned studies, in turn, involve analysis of observed issues based on implementation agnostic, black-box kind of research: answers quality, how long questions reside in hot list.

• Specifically, we already cap the number of answers that contribute to the score at 10. This means that any answer after 10 does not contribute to the hotness score.
• In fact, views don't contribute either. They turned out to be inefficient to query, so somebody at some point just removed them from the calculation. See above, removal of views has proven to be inefficient in addressing a problem. Really, when just stuffing the question with meh answers bumps it hotness as much as allowed currently

Neither change to cap at 10 nor ignoring views didn't help, as evidenced by persisting problems.

And, really, how could it help when 5-8 additional meh answers having ridiculously low score could artificially double / triple hotness score of a question having 1-2 answers that are proven to be popular by high score.

^{Regarding "lemming" meh answers, worth mentioning that at SO these seem to be less of a problem but since vast majority of posts in hot list come from smaller sites, this doesn't help.}

suggest opening a more open-ended feature-request that demonstrates the problem you perceive ("Hot questions stay at the top of the supercollider for too long", or similar) and then we can start playing with the formula.

Item spelled as suggested above has been submitted to Feedback request: New top bar and MultiCollider redesign:

^{Submitted here in order to ensure that involved feature requests (listed below) are included in the list of feedback items.}

##Hot questions stay at the top of the supercollider for too long

Above wording is as has been recommended by David Fullerton♦ here...

---

Finally - to dot all is and cross all ts - it is also worth noting that suggested feature doesn't address cases when question has multiple highly upvoted answers (such as some Apple.SE examples in sticky list). Whether "sticky popularity" of these is a problem that needs addressing and if yes, then how to address it, is out of scope of this request.

Answer 3

First off, this doesn't avoid the technical challenges present in the last suggestion: there's no denormalized "top answer score" column available, so calculating this would require querying all answers attached to each question across all sites. Oh, yeah - there's no denormalized "pending close votes" column either.

So this isn't really feasible. You can stop reading now, if you want - the rest of this answer just disputes the premise.

Unintended consequences

I did a quick simulation of your criteria on a few sites (SO, SU, TWP and Progse) just to see if there'd be a dramatic difference in the results... There wasn't. More or less the same questions came back, in more or less the same order; a few questions gained or lost position further down the list, but the single most noticeable difference was that the "hotness score" for all questions suffered on the sites that weren't Stack Overflow. The most noticeable effect of implementing this would be an increase in questions from SO to the detriment of other sites; given we're already heavily penalizing SO questions in order to give other sites a chance, this would be a pretty counter-productive change.

Answers as indicators of "hotness" vs. "hot answers"

You appear to be searching for a way of finding "hot" questions based on the presence of "hot" answers. (Aside: [there do exist lists of hot answers) This is an interesting idea, although if you check the 10K tools regularly you'll know that popular answers generally go hand-in-hand with popular questions so the outcome is unlikely to be dramatically different from what we have today - indeed, it might well just increase the number of problematic or "sticky" questions in the list.

Philosophically though, this isn't even particularly relevant. This isn't a list of answers; it's a list of questions. And answer score is a fairly narrow indication of the popularity of a question: they align cleanly when there's an awesome Eric Lippert metaphor to be had, but this just treats answers as a function of popularity, not as an independent indicator. Let's consider a few common classes of questions:

1. Bikeshed questions: Everyone has an opinion here, so a good many of them try to post it. Voting tends to follow the "find an opinion that matches your own and up-vote it" model. Let's face it: these questions should usually just be closed.

2. Hard, subtle or thought-provoking questions: Everyone has an answer here too, and most of them are demonstrably wrong. Answer scores vary wildly between top and average.

3. Trivial questions: There's one answer, and everyone knows it, so everyone posts it immediately. FGITW comes into play here, but if folks put a bit of effort into their answers it's possible to end up with something fairly similar to the bikeshed: everyone's just voting on their favorite way to present an answer.

Answers are an indication of popularity in all three cases. Penalizing "non-hot" answers can actually hit #2 the hardest. The trap you most want to avoid is the naive rewarding of questions that get the most "hot" answers, as this is a recipe for bikeshed promotion: the current system does this a good bit of the time, but your proposed changes actually make it worse by promotion questions that don't have wrong answers. Therefore, I would be opposed to this change even if it were technically feasible.

I sympathize with the folks participating on sites whose primary topic is seemingly filled with bikeshed questions, but them's the breaks: if your favorite topic has been driven off of several existing sites due to the problems it caused there, you have to kinda expect these problems when you sign up for a site dedicated to it.