One problem with hot questions seems to be that there is no way for "hotness algorithm" to differentiate genuine popularity from fake one, that is from popularity introduced by the algorithm itself.
- The issue probably wasn't even noticeable when current algorithm was introduced in 2008, since SO wasn't very popular back then. But between Aug 2010 (when formula was last modified) and July 2011, amount of SO users doubled and audience of collider grew accordingly; it became high enough to reveal the problem. And as SO popularity (and collider audience) grows, problems with current "hotness score" will only become more and more prominent.
This is most likely what causes some questions to stick to the top for too long: the algorithm simply is incapable to detect cases when popularity is its own byproduct.
Feature requests to address this issue are:
In 9 examples of "sticky" questions listed so far, 83 of total 254 answers (about 1/3) have score less than 1/100 of top scored post. Given amount of views received by these questions, these hardly have anything to do with popularity. Still, "hotness formula" counts these as contributing to popularity.
- Note recently added 10th example of "sticky question" shows the same pattern: about 2/3 answers (13 of 20) are scored less than 1/100 of top voted post (despite about 90K views) - each adding solid
Qscore/5 to numerator part of the formula. With Qscore value above 900 in this example, this amounts to more than 2000 points (13 * 900 / 5) that are most likely fake.
Look, here is how it is probably supposed to work...
- ...Good question is posted, gets a portion of upvotes, hotness increases. Some time later, good answer is posted, gets its portion of upvotes, along with some additional upvotes to the question triggered by the bump, again, contributing to the hotness.
Then, yet another good answer gets posted, cycle repeats and so on until it's over. That's pretty simple, and to keep things rolling smoothly in the list one just has to tweak things a little to ensure that each next portion upvotes weights less hotness wise, giving it a smooth reasonable decay.
There can also be some minor infrequent deviations, bad answers and unfair voting, mostly from site / tag newcomers but these are expected to be corrected by community regulars the usual way.
For questions that are not too hot (likely 2-3 clicks away from top of the list) it's natural to see things working exactly as intended. Answers and comments quality is mostly maintained by site / tag regulars (business as usual), collider brings moderate amount of interested newcomers from other sites with their views, votes and fresh perspective, everything is nice and fair. The last but not the least, "hotness" decays naturally, just as expected.
But when question gets closer to the top of collider, things change...
...And fun starts. Amount of views and contribution of newcomers from other sites / tags much increases, along with the risk of things getting out of control.
You know, everyone would want to answer a question, but for community regulars, this natural desire is balanced by established norms. Regulars understand what kind answers are good and likely to bring upvotes and they avoid posting garbage that usually gets downvoted: this sort of sets the quality norm.
But wait, the question is now on top, it attracts a lot of visitors whose desire to answer is not balanced by local community norms. At this point "intended life-cycle of a hot question" breaks.
Hotness algorithm assumes the question gets bumped mostly by good answers and that infrequent deviations are corrected by local community through voting and comments. Thing is though, this does not work on top of the collider...
- ...There are just too many new visitors to keep things under local community control, and there are just too many new voters and commenters to get things going as designed.
What happens is - someone posts populist garbage (recipe provided below for those interested), it gets a nice portion of populist upvotes and supportive comments, a portion sufficient to cover any downvotes and critique from site / tag regulars - a portion sufficient for other newcomers to believe answers like that are welcome and rewarded (broken windows anyone?).
So comes next and next and next round of garbage answers, bumping the question, bringing mindless upvotes, over and over and over again.
Hotness algorithm calculates this as genuine popularity, keeps it accordingly high at collider, bringing more visitors that are breaking things further and so on and so on. Note it's not limited to newcomers only, everyone is invited to the party. Community regulars can clearly see that usual quality norms are broken in favor of populist garbage and that they are free to follow the New Order just like anyone else.
That's how fake popularity makes shit stick to the ceiling and helps to keep it there.
It is quite interesting to observe how "sticky" questions appear like breaking assumptions on which hotness formula is based. These assumptions are laid out in SE blog post Alternate Sorting Orders:
...it is quite rare for us to get a question with more than 30 answers...
By the time a question gets to more than 30 answers, and has tons of voting, it’s arguably not a very appropriate question for Stack Overflow.
I find it fairly fascinating that assumptions made more than 3 years ago still work so well... on regular questions. Questions that could have too many answers - non-constructive garbage, gorilla-vs-shark, shopping, all the flavors of vague narq are quickly detected and closed. Regular open questions ("appropriate" ones) tend to have much less than 30 answers indeed.
But when we look at "sticky" questions, things seem to be different. These questions aren't "inappropriate", otherwise these would be closed long time ago. Still, 10 examples of sticky questions have more than 270 answers total, about 27 average per question, pretty close to "magic 30" mentioned in blog post.
- What gives? Well, evidence demonstrated by voting results, as shown above, brings things back to perspective: only about 1/3 of these answers have score above "1/100 of top scored post". This means, at least 2/3 answers in sticky questions are just a garbage that somehow leaked in (probably because "fake hotness" gave question an unnaturally high exposure in collider).
If honestly, I don't know how to handle that. I doubt that algorithm can be made smart enough to differentiate good answer from populist garbage.
I doubt that standard "community attention" approach (that works so well in regular / moderately hot questions) will solve that... To be precise, this could probably be the way to go in low traffic communities getting hot questions once, maybe twice a month - but if community is getting "a hit" (pun intended) once a week at least, this would turn things into cumbersome 24x7 police duty.
It would be interesting to try automatically protecting questions that get close to the top of collider but I can't tell if this will work or not.
It would be also interesting to see what would happen if "hotness formula" would ignore answers with non-positive score, or those with less than 1/10 of maximum score - simply speaking, these where assumption of "popularity" somewhat contradicts results of voting of readers.
Promised recipe for successful populist garbage:
The rules are classical: know your audience. In hot question, your audience will be many (thousands) community newcomers, unfamiliar with norms, plus several site regulars. Newcomers are your sheep, these will bring you wool and milk (upvotes and supportive comments). Regulars are your dogs, you need to make sure they can't bite you too hard (can't flag / 20Kers won't vote-to-delete your answer).
For the "sheep", you need a populist slogan, a pitch, a catch phrase that will trigger sufficient support ("git is fantastic" is a good bet for programmers communities). For the "dogs", you need basic handwaving skills - just enough blah blah to make sure answer isn't flaggable plus make it read sufficiently smoothly to avoid triggering vote-to-delete in case if 20Ker skims through it.
That's it. Nothing else is needed to get a portion of cheap rep in a hot question. Easy isn't it.
Note you'll be not alone, collider will bring more and more guys like that, posting similar stuff, making it look even hotter, bringing more views and more chances for upvotes to crap like this, which in turn will make it hotter and so on.
Oh and don't worry about getting downvotes from community regulars - there will be so much more upvotes from newcomers to keep balance in your favor. A nice thing is, newcomers with less than 1K reputation won't be able to see how votes split, so your answer at +15/-10 will be displayed to them as solid positive +5, good enough to collect more upvotes from those ticked by your pitch.
- Just in case if you feel guilty, there's no need for that: you use system as designed.
Think of it: for stuff deemed potentially harmful, they have all these limits 1, 2, 3, 4 - picked, tweaked and set up to prevent it. There's nothing like that here. The door is open, the road is clear.
Just... go for it. Have some fun, blow it up. Let that devil out
