# Reorder questions picked for hot list based on adjusted hotness score (discard some answers by voting evidence)

### Executive Summary

The 100 questions are selected for the "Hot Network Questions" list displayed in the sidebar of each site. These questions are designed to showcase other sites in the Stack Exchange network.

Those questions should act as ambassadors from that site to the Stack Exchange Network, but due to the way the questions are calculated, bad data points result in poor examples staying on the list for a long time.

A previous request to fix this problem was marked as due to technical issues, so this question is requesting to change the order of the list without increasing the resources required to implement it.

TL;DR When votes of 20... 30... 100 users clearly indicate that only one or two answers are popular, it does not make sense to pretend that other answers are popular too.

### Background

The Stack Exchange Network Hot Questions are viewed by hundreds of thousands of eyes. Many people's introduction to an SE site is influenced by the hot questions they see attributed to that site.

If the questions that are selected are good ambassadors for that site, the influx of new users is positive. If bad data points get selected, new users will follow the example of the hot question which can overwhelm budding community moderation, having a negative impact.

### Bad Data Points

There are two things that can cause problems with the current hot questions list:

1. Ordering of "mature" questions in the list is driven by random bits of garbage picked by popular posts instead of answers with solid score.
2. Close votes are not taken into account. Note how this contrasts to close votes impact in twitter bot.

The current formula increases 'hotness' as the number of answers increases, even if voting evidence (score) indicates that those answers are of low interest. Bad answers are not an indication of a good question, and should not be used to determine the hotness of a question.

Questions that have close votes are treated as if there are no close votes. Smaller communities with questions asked at times when users with privileges are not around may have questions that would normally be closed getting upvotes before the community can moderate and improve the question. Close votes should be counted against questions in that list.

### Technical Limitations

The previous was declined because the proposed formula would increase the resources required to make the query, and that was preventing its implementation.

Approach proposed here addresses above concerns by providing only a modification to the calculation and not any change to the database structure and it is limited only to fixed amount of questions, to guarantee that performance impact is under control.

Propose that the same parameters as before be used to select the 100 hottest questions, but the ranking of those 100 questions should be re-ordered with the following conditions:

• Answers with a score lower than (TopAnswerScore / 10) - 1 should be discarded from the hotness calculation
• Close votes on the question should reduce the total hotness of the question by 20% (so a closed question would have a 0 hotness score)

This will prevent questions from sticking to the top of the hot list too long by accumulating bad answers, and allow a couple quick close votes to prevent a bad example question from being broadcast across the network.

-
thanks to GlenH7, MichaelT and especially to jmac for help in drafting this request –  gnat Jan 15 '14 at 16:51

## Yes, yes, and absolutely yes

Because of this question which got more views in 24 hours than we the entire site gets page views many days (question A), we have a lot of new users. That leads to people upvoting questions like this (question B) which are off-topic, has 4 close votes, but is still on the hot questions list!

Small communities cannot handle moderating thousands of new users coming in. We either need to allow:

1. Trusted users to protect/lock questions
2. Trusted users to add a tag to remove a question from the hot list
3. Mail notifications to community managers to be on-call in the site chatroom when a question gets over X amount of abnormal views

Or something. Like making this feature request happen. As-is you are hurting some communities with features that are supposed to be for our benefit, and it is unfortunate.

Update 10:49 UTC (or so) As of now, the question B has been put on hold, but is still popping up in the hot questions list...

-
It's noteworthy that the OP of your Question A, "'friendly' way to let manager know...", has not visibly interacted with the question, or even been back to the site, since an hour or so after posting, which is now two days ago. –  Josh Caswell Jan 18 '14 at 20:53
There was another question that was put on hold yet still sat on the hot list at the same time. Had 4 close votes and was the top of the bottom half of the hot network questions list. The Question A above had a very significant impact on traffic to The Workplace as evidenced here too. Look at the long-term graph for natural growth, and note the giant disturbing spikes that hot questions cause. –  jmac Jan 20 '14 at 7:19
The last edit hints at why bringing close votes into this is fairly pointless: this data is heavily cached. If you're expecting to be able to knock a question out of the list with a vote, you're gonna spend a long time gritting your teeth waiting for it to happen. –  Shog9 Feb 9 '14 at 22:01

For the sake of completeness, besides mentioned technical limitations decline justification listed some other points as well. These points were discussed in comments under decline message and for your convenience, a brief summary is provided below:

• in this case, not knowing the implementation details makes it almost impossible for you guys to solve this one.
...a lot of the research here is based on a false premise.

Feature request is based primarily on studies performed prior to ones addressed above:

Mentioned studies, in turn, involve analysis of observed issues based on implementation agnostic, black-box kind of research: answers quality, how long questions reside in hot list.

• Specifically, we already cap the number of answers that contribute to the score at 10. This means that any answer after 10 does not contribute to the hotness score.
• In fact, views don't contribute either. They turned out to be inefficient to query, so somebody at some point just removed them from the calculation. See above, removal of views has proven to be inefficient in addressing a problem. Really, when just stuffing the question with meh answers bumps it hotness as much as allowed currently

Neither change to cap at 10 nor ignoring views didn't help, as evidenced by persisting problems.

And, really, how could it help when 5-8 additional meh answers having ridiculously low score could artificially double / triple hotness score of a question having 1-2 answers that are proven to be popular by high score.

Regarding "lemming" meh answers, worth mentioning that at SO these seem to be less of a problem but since vast majority of posts in hot list come from smaller sites, this doesn't help.

suggest opening a more open-ended feature-request that demonstrates the problem you perceive ("Hot questions stay at the top of the supercollider for too long", or similar) and then we can start playing with the formula.

Item spelled as suggested above has been submitted to Feedback request: New top bar and MultiCollider redesign:

Submitted here in order to ensure that involved feature requests (listed below) are included in the list of feedback items.

## Hot questions stay at the top of the supercollider for too long

Above wording is as has been recommended by David Fullerton♦ here...

Finally - to dot all is and cross all ts - it is also worth noting that suggested feature doesn't address cases when question has multiple highly upvoted answers (such as some Apple.SE examples in sticky list). Whether "sticky popularity" of these is a problem that needs addressing and if yes, then how to address it, is out of scope of this request.

-

First off, this doesn't avoid the technical challenges present in the last suggestion: there's no denormalized "top answer score" column available, so calculating this would require querying all answers attached to each question across all sites. Oh, yeah - there's no denormalized "pending close votes" column either.

So this isn't really feasible. You can stop reading now, if you want - the rest of this answer just disputes the premise.

## Unintended consequences

I did a quick simulation of your criteria on a few sites (SO, SU, TWP and Progse) just to see if there'd be a dramatic difference in the results... There wasn't. More or less the same questions came back, in more or less the same order; a few questions gained or lost position further down the list, but the single most noticeable difference was that the "hotness score" for all questions suffered on the sites that weren't Stack Overflow. The most noticeable effect of implementing this would be an increase in questions from SO to the detriment of other sites; given we're already heavily penalizing SO questions in order to give other sites a chance, this would be a pretty counter-productive change.

### Answers as indicators of "hotness" vs. "hot answers"

You appear to be searching for a way of finding "hot" questions based on the presence of "hot" answers. (Aside: [there do exist lists of hot answers) This is an interesting idea, although if you check the 10K tools regularly you'll know that popular answers generally go hand-in-hand with popular questions so the outcome is unlikely to be dramatically different from what we have today - indeed, it might well just increase the number of problematic or "sticky" questions in the list.

Philosophically though, this isn't even particularly relevant. This isn't a list of answers; it's a list of questions. And answer score is a fairly narrow indication of the popularity of a question: they align cleanly when there's an awesome Eric Lippert metaphor to be had, but this just treats answers as a function of popularity, not as an independent indicator. Let's consider a few common classes of questions:

1. Bikeshed questions: Everyone has an opinion here, so a good many of them try to post it. Voting tends to follow the "find an opinion that matches your own and up-vote it" model. Let's face it: these questions should usually just be closed.

2. Hard, subtle or thought-provoking questions: Everyone has an answer here too, and most of them are demonstrably wrong. Answer scores vary wildly between top and average.

3. Trivial questions: There's one answer, and everyone knows it, so everyone posts it immediately. FGITW comes into play here, but if folks put a bit of effort into their answers it's possible to end up with something fairly similar to the bikeshed: everyone's just voting on their favorite way to present an answer.

Answers are an indication of popularity in all three cases. Penalizing "non-hot" answers can actually hit #2 the hardest. The trap you most want to avoid is the naive rewarding of questions that get the most "hot" answers, as this is a recipe for bikeshed promotion: the current system does this a good bit of the time, but your proposed changes actually make it worse by promotion questions that don't have wrong answers. Therefore, I would be opposed to this change even if it were technically feasible.

I sympathize with the folks participating on sites whose primary topic is seemingly filled with bikeshed questions, but them's the breaks: if your favorite topic has been driven off of several existing sites due to the problems it caused there, you have to kinda expect these problems when you sign up for a site dedicated to it.

-
nothing challenging performance wise is needed over what is there now, because "hotness correction" is limited to posts already picked by current algorithm. Take 100 questions and less than 1000 answers already selected as of now, adjust their score, reorder and feed top 3-to-26 into sidebar. That's it. O(1), performance "cost" is negligible –  gnat Feb 9 '14 at 21:35
You're kinda missing the point here, @gnat: it's not that this data is slower to obtain (although it is) - we'd have to query every site's database in order to even get it. That's just not feasible - implementing this would mean denormalizing additional data and somehow keeping it updated quickly enough for it to be meaningful, which seems like a huge amount of work for something that would almost certainly have detrimental effects on the results anyway. –  Shog9 Feb 9 '14 at 21:38
you query some data already don't you - to create these funny looking tooltips in hot network list: "This question has been arbitrarily awarded 100.982 hotness points". Feature request only suggest to use the same data, that is already queried and used, nothing else –  gnat Feb 9 '14 at 21:45
"The same data" doesn't include individual answer scores or close votes. You have the site, creation date, score, last activity date, answer count, deletion date, closed date, tags and answer score. That's it. –  Shog9 Feb 9 '14 at 21:46
I see, all right. Let's see if I understand it correctly this time: performance cost of this feature request is that of re-querying 100 questions (we're talking these that are listed) with answers ordered by votes - is that right? –  gnat Feb 9 '14 at 22:05
No. That's the perf cost if we don't actually expect the list to change based on the altered criteria. Normally, those 100 results are pulled from a much, much larger pool of questions, with the "hotness" score calculated for each of them so that they can be ranked accordingly. But if we're expecting the same 100 results and just want them ordered differently, then this would suffice - in fact, you could probably estimate the cost of this using the public API. Of course, if we're just getting the same results, then... what's the point? –  Shog9 Feb 9 '14 at 22:10
yes, this request is limited only to 100 questions that have been "pre-selected" using current formula. ("Global" score adjustment has been suggested, discussed and declined by David per prior request.) The point is to manage sidebar exposure: take 100 questions already selected as of now, adjust their score, reorder and feed top 3-to-26 into sidebar. –  gnat Feb 10 '14 at 5:54
Yeah, that's... Unlikely to do much of anything at all. We could slap a fixed penalty on questions from TWP and have a much greater effect on what shows up in the list with considerably less effort and overhead. –  Shog9 Feb 10 '14 at 7:23
I don't understand how penalty is supposed to help when particular question sticks to top and starts quickly collecting 5...10...20 lemming answers? Also, what about other sites complaining about similar issues (Programmers, Math, Code Golf, UX...)? are these also to be slapped –  gnat Feb 10 '14 at 7:57
If you refer to the formula that drives this, you'll see that the only way a question can end up sticky is to maintain a score higher than most of the other questions on the network - either by accumulating answers and/or votes at an unusually high rate, or by simply not having any real competition for a good long while. Think of it as one balloon floating higher than the rest of a bunch. So, you tie a bit of lead to the string... @gnat –  Shog9 Feb 11 '14 at 17:57
as far as I can tell performance cost of suggested feature is acceptable, is it? Also you mentioned that simulations show that impact is small and so are associated risks, correct? (btw minor impact is intentional; for a bit more impact/risk you can consider cut at something like TopAnswerScore/5-2 instead) Wonder why you don't just give it a try, are you afraid of something? Are you afraid that it will succeed and break your dream of completely dropping hot list? Or that it'll fail and I will push for hot list removal myself? –  gnat Feb 13 '14 at 5:47
Thanks for the additional breakdown of hot questions and the related effects that could potentially occur from changing the algorithms. –  GlenH7 Feb 14 '14 at 16:45
correct if I misunderstood, the primary reason for decline is that suggested feature doesn't address the difference of thought provoking questions from bikeshed and trivial ones. (Gee it took me almost half to even discover this point. Can't tell if this is because it was obscured by other considerations or maybe I was simply blinded by the passion to tweak hotness formula) –  gnat Jul 6 '14 at 11:57
you make some interesting points that I didn't consider these when making this feature request, I even initially accepted because of that. However I recently learned that SE doesn't even keep simple log of what happens in the hot list. This makes it look like pure speculation doesn't it. I mean we can't tell if your reasoning is right or wrong because we simply have no data to help us decide. That's... sad. You are of course totally free to pick the reason to decline... –  gnat Aug 24 '14 at 14:01
...however I am also free to decide whether to accept an explanation for decline or not –  gnat Aug 24 '14 at 14:02