What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

I've noticed that some very highly upvoted questions stay in the list of hot questions (also displayed in the Stack Exchange menu on the top left) for a very long time, often for several days. One example would be this question on Programmers which is still in the hot questions list (though pretty much at the bottom) and was posted around 6 days ago.

Sometimes those are interesting questions, but often they are popular mostly because they appear in the hot questions list gaining views and are about a topic where everyone has an opinion (and a vote) to give. I'm fine with promoting the questions via the MultiCollider, but I don't think those highly-voted questions are that good, they don't need to be in the hot questions list for several days.

I think those questions should be thrown out earlier and should not stay in the hot questions list for several days. It makes the hot questions list less useful as you very often encounter the same questions again, even if you check back a day later. I thought the idea was to show a broad mixture of possibly interesting questions from all around the SE network, not to semi-permanently highlight questions that are already popular.

I propose to either throw out questions from the hot questions list once they are around 36-48 hours old, or at least significantly increase the time-dependence of the hotness from that point on.

share|improve this question
4  
This is a problem with the hottest questions on StackExchange.com not with the MultiCollider. But you have a good point, some questions and sites are probably over-represented in the list –  Ivo Flipse Aug 6 '11 at 12:55
4  
"This is also a question type that is discouraged on most sites, but which leads to a lot of answers and a lot of voting, and thereby often a high hotness score." - I think that can be solved by getting rid of the "# of answers" multiplier from the hotness equation. But of course that's just my biased opinion. –  Mysticial Sep 10 '12 at 10:00
    
issues with hot questions are discussed from a bit different angle at Programmers meta: Answers quality in hot questions "...protect your own users from scale ...human interaction, many to many interaction, doesn't blow up like a balloon..." –  gnat Jan 17 '13 at 12:39
1  
I completely agree with Mad Scientist they stay to long we should do some thing –  dragonloverlord Apr 15 '13 at 15:23
    
for the record: an item intended to address involved issues has been submitted to Feedback request: New top bar and MultiCollider redesign –  gnat Oct 25 '13 at 11:55
4  
I just thought I'd update this post by saying that we're going to be looking into some algorithm changes in the next few weeks that should help with this issue. –  Jeremy T Jan 7 at 22:48
1  
I need a program where the user inputs an array of doubles and the program outputs the array sorted has been in hot list in between Dec 27 and Jan 7, does it qualify to be added to the list here? –  gnat Jan 7 at 23:00
1  
@JeremyTunnell If you're looking at the algorithm, I've a few more suggestions in another meta post on things that could be improved there. –  Mad Scientist Jan 8 at 6:55
1  
This post has been generally addressed, but we've got more things coming to better target questions. –  Jeremy T Feb 25 at 18:48
    
related: Ease up on the Hot Network Question Randomness "the hot questions list is basically a list of 100 or so questions on the server. All we are doing is randomizing the list of 100 before we pull some off the top instead of pulling the top X every time." –  gnat Feb 25 at 19:04
2  
@JeremyT I'd be interested in a meta post on the new formula. It is almost impossible to post any specific criticism on the new formula as we don't know it and the randomness means we also can't really guess it. –  Mad Scientist Feb 25 at 21:19
    
related: Add an audit log to record when particular question enters and leaves hot list (FWIW it feels like the issue of "sticky" questions is only somewhat tamed, but not really resolved by features like sidebar shuffling and community protection) –  gnat Aug 23 at 15:13

5 Answers 5

One problem with hot questions seems to be that there is no way for "hotness algorithm" to differentiate genuine popularity from fake one, that is from popularity introduced by the algorithm itself.

  • The issue probably wasn't even noticeable when current algorithm was introduced in 2008, since SO wasn't very popular back then. But between Aug 2010 (when formula was last modified) and July 2011, amount of SO users doubled and audience of collider grew accordingly; it became high enough to reveal the problem. And as SO popularity (and collider audience) grows, problems with current "hotness score" will only become more and more prominent.

This is most likely what causes some questions to stick to the top for too long: the algorithm simply is incapable to detect cases when popularity is its own byproduct.

Feature requests to address this issue are:


In 9 examples of "sticky" questions listed so far, 83 of total 254 answers (about 1/3) have score less than 1/100 of top scored post. Given amount of views received by these questions, these hardly have anything to do with popularity. Still, "hotness formula" counts these as contributing to popularity.

  • Note recently added 10th example of "sticky question" shows the same pattern: about 2/3 answers (13 of 20) are scored less than 1/100 of top voted post (despite about 90K views) - each adding solid Qscore/5 to numerator part of the formula. With Qscore value above 900 in this example, this amounts to more than 2000 points (13 * 900 / 5) that are most likely fake.

Look, here is how it is probably supposed to work...

  • ...Good question is posted, gets a portion of upvotes, hotness increases. Some time later, good answer is posted, gets its portion of upvotes, along with some additional upvotes to the question triggered by the bump, again, contributing to the hotness.
     
    Then, yet another good answer gets posted, cycle repeats and so on until it's over. That's pretty simple, and to keep things rolling smoothly in the list one just has to tweak things a little to ensure that each next portion upvotes weights less hotness wise, giving it a smooth reasonable decay.
     
    There can also be some minor infrequent deviations, bad answers and unfair voting, mostly from site / tag newcomers but these are expected to be corrected by community regulars the usual way.

For questions that are not too hot (likely 2-3 clicks away from top of the list) it's natural to see things working exactly as intended. Answers and comments quality is mostly maintained by site / tag regulars (business as usual), collider brings moderate amount of interested newcomers from other sites with their views, votes and fresh perspective, everything is nice and fair. The last but not the least, "hotness" decays naturally, just as expected.

But when question gets closer to the top of collider, things change...

http://i.stack.imgur.com/X4q2Y.png

...And fun starts. Amount of views and contribution of newcomers from other sites / tags much increases, along with the risk of things getting out of control.

You know, everyone would want to answer a question, but for community regulars, this natural desire is balanced by established norms. Regulars understand what kind answers are good and likely to bring upvotes and they avoid posting garbage that usually gets downvoted: this sort of sets the quality norm.

But wait, the question is now on top, it attracts a lot of visitors whose desire to answer is not balanced by local community norms. At this point "intended life-cycle of a hot question" breaks.

Hotness algorithm assumes the question gets bumped mostly by good answers and that infrequent deviations are corrected by local community through voting and comments. Thing is though, this does not work on top of the collider...

  • ...There are just too many new visitors to keep things under local community control, and there are just too many new voters and commenters to get things going as designed.
     
    What happens is - someone posts populist garbage (recipe provided below for those interested), it gets a nice portion of populist upvotes and supportive comments, a portion sufficient to cover any downvotes and critique from site / tag regulars - a portion sufficient for other newcomers to believe answers like that are welcome and rewarded (broken windows anyone?).
     
    So comes next and next and next round of garbage answers, bumping the question, bringing mindless upvotes, over and over and over again.

Hotness algorithm calculates this as genuine popularity, keeps it accordingly high at collider, bringing more visitors that are breaking things further and so on and so on. Note it's not limited to newcomers only, everyone is invited to the party. Community regulars can clearly see that usual quality norms are broken in favor of populist garbage and that they are free to follow the New Order just like anyone else.

That's how fake popularity makes shit stick to the ceiling and helps to keep it there.


It is quite interesting to observe how "sticky" questions appear like breaking assumptions on which hotness formula is based. These assumptions are laid out in SE blog post Alternate Sorting Orders:

...it is quite rare for us to get a question with more than 30 answers...
By the time a question gets to more than 30 answers, and has tons of voting, it’s arguably not a very appropriate question for Stack Overflow.

I find it fairly fascinating that assumptions made more than 3 years ago still work so well... on regular questions. Questions that could have too many answers - non-constructive garbage, gorilla-vs-shark, shopping, all the flavors of vague narq are quickly detected and closed. Regular open questions ("appropriate" ones) tend to have much less than 30 answers indeed.

But when we look at "sticky" questions, things seem to be different. These questions aren't "inappropriate", otherwise these would be closed long time ago. Still, 10 examples of sticky questions have more than 270 answers total, about 27 average per question, pretty close to "magic 30" mentioned in blog post.

  • What gives? Well, evidence demonstrated by voting results, as shown above, brings things back to perspective: only about 1/3 of these answers have score above "1/100 of top scored post". This means, at least 2/3 answers in sticky questions are just a garbage that somehow leaked in (probably because "fake hotness" gave question an unnaturally high exposure in collider).

If honestly, I don't know how to handle that. I doubt that algorithm can be made smart enough to differentiate good answer from populist garbage.

I doubt that standard "community attention" approach (that works so well in regular / moderately hot questions) will solve that... To be precise, this could probably be the way to go in low traffic communities getting hot questions once, maybe twice a month - but if community is getting "a hit" (pun intended) once a week at least, this would turn things into cumbersome 24x7 police duty.

It would be interesting to try automatically protecting questions that get close to the top of collider but I can't tell if this will work or not.

It would be also interesting to see what would happen if "hotness formula" would ignore answers with non-positive score, or those with less than 1/10 of maximum score - simply speaking, these where assumption of "popularity" somewhat contradicts results of voting of readers.


Promised recipe for successful populist garbage:

  • The rules are classical: know your audience. In hot question, your audience will be many (thousands) community newcomers, unfamiliar with norms, plus several site regulars. Newcomers are your sheep, these will bring you wool and milk (upvotes and supportive comments). Regulars are your dogs, you need to make sure they can't bite you too hard (can't flag / 20Kers won't vote-to-delete your answer).

  • For the "sheep", you need a populist slogan, a pitch, a catch phrase that will trigger sufficient support ("git is fantastic" is a good bet for programmers communities). For the "dogs", you need basic handwaving skills - just enough blah blah to make sure answer isn't flaggable plus make it read sufficiently smoothly to avoid triggering vote-to-delete in case if 20Ker skims through it.

That's it. Nothing else is needed to get a portion of cheap rep in a hot question. Easy isn't it.

Note you'll be not alone, collider will bring more and more guys like that, posting similar stuff, making it look even hotter, bringing more views and more chances for upvotes to crap like this, which in turn will make it hotter and so on.

Oh and don't worry about getting downvotes from community regulars - there will be so much more upvotes from newcomers to keep balance in your favor. A nice thing is, newcomers with less than 1K reputation won't be able to see how votes split, so your answer at +15/-10 will be displayed to them as solid positive +5, good enough to collect more upvotes from those ticked by your pitch.

  • Just in case if you feel guilty, there's no need for that: you use system as designed.
     
    Think of it: for stuff deemed potentially harmful, they have all these limits 1, 2, 3, 4 - picked, tweaked and set up to prevent it. There's nothing like that here. The door is open, the road is clear.
     
    Just... go for it. Have some fun, blow it up. Let that devil out
share|improve this answer
4  
Thanks - this recipe works awesome! Git makes me tingle. –  psr Jan 18 '13 at 0:18

I agree that two weeks was extreme for that question, but as you can see, it is no longer present.

That is also a very, very rare event. Can you think of another time that has happened since we introduced the MultiCollider?

I don't think we need to rearchitect the entire se.com cross-site hotness algorithm on the basis of a few hundred year floods.

share|improve this answer
2  
The first of my 2 examples also was about a week in the Multicollider. I acutally think 2 days in the MultiCollider are enough, mabe even a bit less. I would find it more useful if the questions are changed more often. I don't think the algorithm needs a big change, just an additional cutoff date to remove older questions. –  Mad Scientist Aug 11 '11 at 10:33
    
An easy way is to add an internal overall hotness counter, let that overall hotness counter increment each X minutes (perhaps half/full hour) and once the counter hits a certain amount of overall hotness it is no longer allowed to be displayed. You could set it such way that if it counts up to X days (taking the average of the top entry) that it's no longer showed. Only a few lines of code if you ask me; but of course, it depends on whether it has been a hundred years flood or will appear again in the future. –  Tom Wijsman Aug 12 '11 at 16:42
6  
-1 - 10 examples collected by OP in between Jul '11 and Jan '13 mean this happens every month or two. To me, this hardly qualifies as "very, very rare event" –  gnat Jan 29 '13 at 8:20
    
@gnat Two weeks is 14 days. Are any of his examples 14 days? Let's try counting again, shall we? –  Jeff Atwood Jan 29 '13 at 9:03
13  
@JeffAtwood does that mean anything less than 14 days is OK? For the record, examples currently listed were stuck at the top for 16 days, 4, 6, 10, 12, 10, 9, 10, 10 - is that normal? Note question text describes the issue as "several days", not as 14-days-or-more –  gnat Jan 29 '13 at 9:42
    
bounty awarded for inventing formula that has been proven to work reasonably well in the past (and still does a fairly good job at early picking of potentially popular questions) –  gnat Feb 22 '13 at 8:01

At the moment, nearly every question on hot list of P.SE is a "100 year flood" question and if the dates on the last activity of the questions is any indication, will be so for quite a long time.

Screen shot of the hot tab on P.SE at about 2:40 UTC May 1st

Anyone want to make a bet when the "What should every programmer know about web development" question will fall off the top without having anyone tweak it?

The second question on that list hasn't been touched in nearly a month. A question that hasn't had any activity since it was closed on the 17th is the 6th on the list. None of the questions in the list shown have new answers (ok, one, maybe two had a new answer that was deleted) but rather edits to existing questions and answers, fixing typos and the like - [1] [2].

As it stands, the 'hot' page is misleading of what is actually hot. It furthermore has a potentially less than ideal user experience for new users who come and add (often poor) answers to questions that are rather thoroughly answered as it is (that are then down voted and/or deleted).


Just for fun, here's a snapshot of MSO's hot questions at about the same time...

MSO hot questions, 3:00 UTC May 1

There is more activity on MSO than P.SE to create some churn to that, but that top question is sitting there for a rather long time. The problem gets even worse when you have more votes (2k votes vs 366) and views (197k views vs 87k views) on a site that gets about 2/3 the questions/day of MSO but 5x the views/day further helping inflate the views to the hot questions.

share|improve this answer

Possible way to improve the algorithm, if you believe Gnat's analysis. Have the hotness formula consider votes and answers by "community regulars" (defined in some plausible way, based on reputation, or activity on the site, or views on the site) with much more weight than "drive by" voters or answerers. If it's too computationally expensive it could always only be run on questions near the hotness threshold under the original formula.

This wouldn't help in detecting answers where site regulars jumped in and started pandering to gather reputation, but would help with votes and answers from people that may not understand the community norms.

share|improve this answer
    
it would help if formula at least accounted for close votes from 3K community regulars as proposed here –  gnat Mar 4 at 18:49

How about letting votes "age"? Instead of measuring hotness by just counting votes, use some technique like exponential averaging to make older votes count for less than more recent votes. Exponential averaging is easy to implement.

Some questions wil be higher on the hot list than they should be. Voting is an imperfect process. If this technique will help even a little, then it's worth considering.

share|improve this answer
    
Votes aging / time decay is already in the hotness formula, in the form of Qage value. It is intended to decay not only votes but views as well. At "non sticky" questions, this works like a charm. Week old question with 5 answers, 1K views and 100 votes has much lower score than one day old one, having same 5 answers, 1K views and 100 votes –  gnat Aug 8 '13 at 14:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .