In “network hot” questions formula, discard answers when voting evidence indicates that these are not good data points

Question

TL;DR When votes of 20... 30... 100 users clearly indicate that only one or two answers are popular, it does not make sense to pretend that other answers are popular too.

---

In current version of “network hot” questions formula (AnswerCount * Qscore) ^*, all answers up to 10 are assumed to equally contribute to question "hotness score", including even those downvoted into oblivion.

Suggest discarding answers when there is a strong evidence that these do not provide good data points for question popularity, such as answers with score less than 1/10... 1/20... 1/100 ^{* * *} of top voted one.

Note that hotness formula "specification" ^* assumes exclusion of the answers that are not considered "good data points". An example of this is a justification for discarding accepted answers:

Note that accepted answers weight not at all in hotness. This is intentional, as I feel accepted answers are a fine social contract, but not a good data point for question or answer quality.

As far as I can tell, indiscriminate inclusion of low score answers in questions having lots of views and votes also goes against another underlying assumption in formula "specification":

one assumes... there will be a lot more voting on the answer

Suggest to discard answers having score ^* less than (TopAnswerScore/10-1).

• ^{This way, answers at -2 or less are ignored when question has answers with non-negative score. When some answer reaches score 10 (qualifies for Nice Answer badge), formula would start discarding answers with negative score. When there is an answer at +20, formula would ignore those having less than +2, when there is an answer at +100, it would ignore those scoring less than +9, and so on...}

---

Negative impact of counting low quality answers is closely related to popularity of MultiCollider^* and Hot Questions sidebar^* which use hotness score to arrange list of "hot questions"^* displayed to Stack Exchange users.

When SE users visit the questions from the top of sidebar (previously from collider), some of them choose to add their answers to it. Since this audience involves hundreds users, amount of answers brought into question could quickly increase by 5, 10, 20...

By indiscriminately counting these answers into hotness score (even when voting evidence suggests opposite), formula pushes impacted questions closer to the top of sidebar, which in turn brings more visitors, who in turn add their answers, which in turn push the question even higher at the sidebar and so on, and so on, over and over again, creating a positive feedback loop ^* of uncontrolled artificial increase of the hotness score.

This uncontrolled hotness growth in turn blocks a time decay mechanism "embedded" in the formula (MAX(QAgeInHours + 1, 6) ^ 1.4), which in its turn also contributes to positive feedback loop mentioned above.

A popular question quickly acquiring 9/10 "noisy", low interest answers, can stand on the top of sidebar for hours, even when the number of really popular answers remains the same, nor there is a substantial increase in the number of views (several hundreds views from sidebar do not come close to thousands views that come when question becomes fairly popular from outside of Stack Exchange).

Discarding answers proven to be "insufficiently hot" would allow for more efficient functioning of time decay component of the formula.

---

The last but not the least, contribution of under-scored posts into "question hotness" (which in turn blocks intended time decay) forces questions with multiple low quality answers stick for a long time at the top of the sidebar, making wrong impression on what kind posts are welcome at Stack Exchange.

This makes it look like good questions are those having many meh answers, the effect that is amplified by these questions being highly visible to sidebar audience - hundreds and thousands of SE users. Misguided users spread acquired attitude further into other questions and answers, posting stuff that follows what they saw at the "cool" ("hot") questions.

As far as I can tell, this jeopardizes the very idea ^* of making an Internet a better place.

Please stop counting proven low score answers in hotness formula. Please roll the dice fairly, let user voting and time decay contribute to hotness score as intended. Please promote to sidebar audience less brain-damaging content to learn from.

---

Update. Functionally limited, but performance-friendly alternatives of this feature request:

• Reorder questions picked for hot list based on adjusted hotness score (discard some answers by voting evidence)
• Make hot questions with multiple answers age away faster on smaller / subjective-ish sites

Answer 1

TL;TMWtB

That's a brilliant idea! and would bring some much needed balance back into the hotness values! This change should really cut down on the run-away questions.

_{* Too long; tell me what to believe}

---

Data to back the assertions

Being an engineer, I have this crazy predisposition for having numbers to back assertions when making decisions. So I gathered some data, and I'm editing my answer to reflect what I found.

Near the end of August 2013, I noticed a question that looked like it would ride on top of the hotness collider for a while. Please note, I'm not suggesting anything was wrong with that question. I just thought it would provide some useful data for this meta question.

This is what I found. Apologies in advance for having to break up some of the images.

Baseline

Times are GMT, and I'm not certain about the values I calculated for the denominator. The denominator is supposed to be the age-drag effect on the equation. Essentially, I took the calculated numerator and divided by the Collider value in order to determine what the denominator needed to be. The problem is that most of the time the denominator (age-drag effect) actually boosts the question for about the first 7 hours of the Q's life.

I assumed it was safe to ignore the denominator since it wasn't fully relevant to the proposed change.

Looking at the "percentage of numerator from ..." columns, we see that QScore * NumAnswers / 5 grows in overall impact to the collider value. In effect, the quality of the answers (aka the sum of their scores) is outweighed by the score of the question and the total number of answers. So we see that noisy answers help out the question's hotness value even though they don't help the question itself. Quite telling is the scores on the answers themselves - that's a long tail of low to no up-votes on answers.

Non-zero scored answers

So now what happens if we ignore all the zero scored (or lower) answers?

Our number of answers to consider drops off and holds steady at 9 throughout the time I watched this question. Notice that the relative effect of QScore * NumAnswers / 5 versus SumAnswers stays pretty consistent. My interpretation is that ignoring zero scored answers from the Collider formula would be a good step forward in curbing excessive hotness scores.

However, there's a problem in that the result of QScore * NumAnswers / 5 will always remain zero now until an answer gets an up-vote. That seems a little unfair to a fledgling question. Even a highly up-voted question won't gain traction in this category until an answer gets an up-vote.

10% threshold

So let's see what happens when we implement the 10% threshold as suggested by the OP (gnat).

I think the first thing to notice is how SumAnswers starts to outweigh the other factors for the numerator as the question ages. Personally, I think this is the right approach for us to have. As a question ages and attracts answers, it really should be the quality of those answers that determine how hot a question should remain.

As an additional benefit, the impact from total views is a little greater and doesn't decay quite as quickly. With our current collider implementation, the number of views quickly drops off as a factor in the hotness score. In this case, the total views retains a little over half its original impact.

In case you're still following along, you might be wondering what's the impact of my suggestion to have negative voted answers drag down the hotness value? Honestly, it's not all that much. At the end, only 2 questions had a negative value at -1 each, so take 2 points off of SumAnswers. QScore * NumNegAnswers / 5 yields 17 which is about an additional 20% off the final numerator value from my observations.

Summary

Granted, it's only one question and one set of observations. And you can't draw a trend from only one point.

But I think there's enough information to show that this suggestion is worth considering. It would slow down how quickly hotness scores can ratchet up without punishing high quality questions. In fact, it would probably give up-and-coming high quality questions a better chance since the impact of the answer scores has a greater weight.

So please let the record show that engineers and their equations sometimes lead us to clearer discussions around change. :-)

---

Previous, and now a bit dated, thoughts

I think you have a solid suggestion in reining in the issues with the hotness formula, but I don't think your approach goes far enough. [edit: looks like the data shows it would be pretty decent without my suggestion]

Instead of just ignoring negatively scored answers, they should drag the hotness value down. Any answer with a negative score should have an equal, but opposite, effect on the hotness value. So whatever effect it would have had as a positively voted answer should be flipped around.

One of the problems that you're trying to address and describe in the under-damped feedback cycle is that poor questions attract poor answers and the current formula encourages that behavior. My suggestion is to extend your request and move into an over-damped control system so the formula will punish attracting poor answers.

This also provides an incentive to the community to down-vote the poor answers knowing that it will reduce the potential popularity of a poor question.

Adding my suggestion would get you the cooling effect you're looking for in a quicker fashion.

Answer 2

Sorry, marking this as status-declined.

We love seeing people contributing in concrete ways (especially backed up by research!) but in this case, not knowing the implementation details makes it almost impossible for you guys to solve this one.

1. The hotness formula that's used for the network hot question list is not the same as the one for the "hot" page. Just superficially comparing the two pages shows you that -- the hot page favors questions that are "instantaneously" hot. The collider algorithm does a lot to correct this, documented here. Because of that, a lot of the research here is based on a false premise.

2. Specifically, we already cap the number of answers that contribute to the score at 10. This means that any answer after 10 does not contribute to the hotness score.

3. In fact, views don't contribute either. They turned out to be inefficient to query, so somebody at some point just removed them from the calculation.

4. This query is very much based on "what data is easily queryable?". We already have a denormalized column for AnswerCount, so using it in the formula is easy. Forcing the query to actually consider each answer and count it or not based on a formula is not possible in a query that has to cover so many posts. Adding a denormalized field just for this seems like overkill. It'd be much easier to attack this using the columns already in the DB.

Since this question is based on such a specific modification to the algorithm, I'm marking it status-declined. If you want to pursue this, I suggest opening a more open-ended feature-request that demonstrates the problem you perceive ("Hot questions stay at the top of the supercollider for too long", or similar) and then we can start playing with the formula.

Answer 3

For those interested to support, or oppose, or adjust proposed change, here is more detailed explanation of suggested "cut value":

(TopAnswerScore/10-1)

Above is mostly based on my studies of questions listed in two meta posts:

• Answers quality in hot questions
• Don't let questions stick to the top of the hot questions list forever

_{For the sake of completeness, there were other multiple discussions involved, but the main focus was on the data in above two anyway.}

Studying low quality *) answers in questions listed in above posts made me believe that these tend to have low score.

Also, it looked like the way how current formula tries to take this into account in sum(Ascores) is insufficient due to indiscriminate stuffing of these into Qanswers * Qscore: eg in question scored +100, any answer with zero score makes same contribution as 20 upvotes to other answers.

I tried to figure how formula could be adjusted to better take into account answers that have low score.

---

My first approach was to consider ignoring negative score answers. This idea is extremely natural, for it's hard to imagine how these could be considered "popular". I think if I proposed it as MSO back then, we'd have modified formula several months ago.

The problem with it was, it didn't work on the data I studied. There were just too few answers with negative score to make any substantial difference. My guess is, this is due to how sympathy upvotes work in hot questions. Negative score answer tends to stand out among the rest and when there are hundreds, thousands eyeballs on these, there is a good chance for someone to "like".

• ^{It was sort of revelation that happened when I observed particular hot question. Among many others, there were 3 answers of different degree of crappiness, each voted accordingly at -1, -2, -3. Few hours later, all 3 answers were upvoted back to zero, and kept staying there, buried in the sea of other mediocre, low score answers.}

Okay, I tried to figure how it will work if the answers are cut at zero score. No luck either, there were too few to make a difference. Random upvotes in high view questions made it practically impossible to consider a cut at any constant value.

Cut like at +2 would probably be OK at the question with 1K views, but at one with 3K views, it would fail miserably and positive feedback loop of fake popularity would kick in again. At 3K views, one would rather cut at something like +5, but oh, such a cut would make no sense at lower views questions, with views at or below 1K. Dead end.

---

Messing with ideas of cut at some constant score has led me to believe that cut would better be proportional to the score of the top voted post (question or answer). I tried TopPost/100 and it made good sense on the set provided in sticky questions post but made too little difference in examples in answer quality post, so I decided to see how it would work if I cut harder.

At about this time, I also made a change that I would want it to be TopAnswer rather than TopPost, for I wanted cut to be based on comparison of more uniform kinds of posts (comparing score between answers felt more reliable than between answers and question).

If memory serves, cuts between TopAnswer/5 and TopAnswers/15 worked fairly well on my data sets, so I decided to pick a single value of these, the one that would be easier to explain and understand.

TopAnswer/10 made it, as it blends well in Stack Exchange "value system": 10x score difference is one that makes a Nice Answer, same difference is between Nice and Great answer, all right.

---

Okay, let it be cut at about TopAnswerScore/10, now I needed to figure how to score questions at early stages, when there are just too few votes to make any sense of this?

My studies of how hotness score works made so far led me to believe that at early stages, current formula works quite well (try to study it yourself, you'll find out that it starts to really mess things only when question gets to higher views / votes) so I basically wanted it to work until there are sufficient votes to reliably switch to "new cut".

I wanted current formula to work until at least one of the answers get votes about to qualify it for Good Answer badge, ie until about +25. I figured that if I subtract 1, that will make a good approximation, that made the final version: TopAnswer/10 - 1. Quoting self,

This way, answers at -2 or less are ignored when question has answers with non-negative score. When some answer reaches score 10 (qualifies for Nice Answer badge), formula would start discarding answers with negative score. When there is an answer at +20, formula would ignore those having less than +2, when there is an answer at +100, it would ignore those scoring less than +9, and so on...

---

_{*) "low quality answers" -- my evaluation of quality was subjective. Simply speaking, I read the answers and tried to figure whether these make a fair attempt to address the question asked and provide an explanation. I tried to abstain of opinions, ie one-liner slogans lacking an explanation were considered low quality even when I agreed 200%.}

---

Performance considerations

As soon as I arrived at conclusion that cut would better be proportional to the score of the top voted post, I took a while to ponder on performance. This algorithm looked substantially harder than current formula and even without knowing if it's feasible or not, I wanted a variant that would be comparable to current one in terms of performance.

For comparable approach, I took into account that 1) I really need an evidence-based score for a limited amount of posts (just those at collider) and 2) current formula tends to get in trouble only when posts are near the top of collider.

Above brought me to idea to pre-select candidates for collider using current formula (which is already proven to have acceptable performance) and only re-calculate and reorder these pre-selected posts, like just few hundreds questions.

My idea for amount of questions to pre-select was like 2x (3x, 4x) of amount needed for collider (the more the better, as long as performance remains acceptable). The point of 2x-3x-4x multiplier is to ensure that "candidates pool" includes questions that have been out of collider. As a result, even if currently hot questions have their initial approximate (old-fashioned) score screwed up by low interest answers, these are still guaranteed to compete against questions that gained popularity in a more natural way.

Feature request based on performance considerations above is: Reorder questions picked for hot list based on adjusted hotness score (discard some answers by voting evidence).

Performance considerations - age decay

Update: another performance-friendly way to approximate suggested feature is to make aging factor depend on the amount of answers, so that questions with more answers (5, 10, 20, 40...) start aging away sooner and stronger.

Underlying concept is the same as in proposed feature, that is new answer is either 1) voted high enough to compensate for increased age decay and keep popularity high, or 2) makes the hotness score decrease faster, thus lowering question exposure and decreasing chances for further damage.

Purpose is also the same, to make for a more robust, self-correcting process, when low quality answers contribute to resolving the problem instead of making it worse.

Feature request based on performance considerations above is: Make hot questions with multiple answers age away faster on smaller / subjective-ish sites.

Answer 4

I think this is a great idea though any algorithm in the ballpark would probably be fine. For example (Qscore*(Sum of AnswerScores)) would probably be fine and at least a little easier to implement. It would also allow great answers (which we are theoretically optimizing for, pearls not sand and whatnot) to bring a question into the collider, and presumably we want to show off the great answers at least as much as the great questions.

Answer 5

Personally I think this is a great idea. The collider in my experience clearly habits to attach to a fixed set of questions which all tend to have the same thing in common: Being the sorts of things everybody has an opinion about. If answers that lack distinct quality are not counted in the chance to enter the collider, this will have a clear and distinct affect on questions of that nature, which are frequently not particularly good to begin with.

The synergy between questions that attract many answerers and the advertising in the collider can clearly only have one affect, which is going to be good when the many answerers attracted are adding quality up-voted content. When it's everyone adding their two-cents-opinions in and doing no benefit to the information archive that is SE however, it clearly benefits everyone to stem the tide into the question that the collider brings.