Better criteria for the hot questions list

Question

The list of hot questions is shown on stackexchange.com and the MultiCollider (the dropdown in the top left). The idea behind it is, as far as I understand it, to present interesting questions across the whole network. I often find interesting posts that I wouldn't have seen otherwise when browsing the hot questions list, but I think that there is a lot of potential to improve the selection.

There are several aspect that could be improved in my opinion:

Questions can stay in the list for too long, making the list a bit too static for regular SE users. I've previously posted a feature request about this, and I still think that no question should be able to stay in the hot questions list for a whole week.

The current criteria are prone to select problematic questions. They value a high number of answers, which you find for example in list questions. So questions like this example from Math.SE have it easy to get into the list, while this type of question is regarded as problematic by most of the SE network and explicitly disallowed on many sites.

Being in the list of hot questions is self-perpetuating. Questions on the list get significantly more exposure, which means more votes, which then improves the position in the list of hot questions. The votes from outside users are also problematic as they represent mostly the popular appeal of the posts, not necessarily a judgement of the quality by an expert.

I've some ideas on what could be changed:

Exclude community wiki questions from the list. Those are usually big list-style questions that aren't a good example for high-quality content across the network.

Put less value on a high number of answers. The current method values (as far as I understand it) a high number of answers and a high total score of all answers. This preferentially puts more subjective or list-type questions to the top. A good question that got one high-quality answer that is highly upvoted shouldn't be at a disadvantage compared to a popular question that gets lots of opinions. Maybe counting only the first two or three answers and their combined score would be enough.

Weight external votes differently from internal votes. External votes from users that discover the question via the hot questions list shouldn't be counted in full, they mostly represent the additional exposure and the mass-appeal of the question, not necessarily the quality. But they also shouldn't be disregarded completely, a question that doesn't get any votes from the external users is probably too specialized to be of interest to a wide audience.

Normalize votes for each site. The voting behaviour varies a lot between different sites. Currently sites that have an above average number of votes are favoured compared to other sites. Normalizing the votes on a per-site base would put a stronger emphasis on the number of votes a specific questions gets compared to other questions on the same sites. It would prevent certain sites from being overrepresented due to their general voting behaviour.

These are just some rough ideas, I'm open to more suggestions.

Answer 1

##Executive Summary

All SE sites are currently far smaller than Stack Overflow. The amount of eyes the Hot Questions can bring to a smaller or younger site can overwhelm budding community moderation. Hopefully any new system will be sure to minimize harm to the communities these questions were (I assume) designed to get attention for.

1. Maximize promoted question quality
2. Minimize harm to community moderation for popular questions

To do that, I propose the following two changes:

1. Questions with active close votes/flags should have those weighted
2. Hot questions should be auto-protected

##Background

I looked at the 10 high-view questions from the past month on The Workplace:

1. I worked for a company affiliated with a very publicly failed project. Should I take it off my resume?
2. How do I avoid driving with a coworker who drives dangerously?
3. If companies can afford a counter offer, why don't they pay that before the employee departs?
4. What should I do when I don't have anything to say in daily stand up meetings?
5. Asked for 3 references but I don't have 3!
6. Is it appropriate to ask about a company's work from home policy during an interview?
7. How to respond to "Why do you deserve a promotion"?
8. Is it okay to ask how many women are on the team I'd be working with?
9. How much time in between companies you are in will you be considered a "Job Hopper?"
10. What should I consider before wearing Google Glass to a job interview when I do development for them?

• 30% of these questions were closed (2, 3, 10)
• The more answers a question gets, the more negative scored/deleted answers there are

##Weight Flags/Close Votes Against Questions

Different SE communities manage quality in different ways. On The Workplace many members leave a comment to help the asker refine their question, and hold off on downvotes in order to not scare off newer users. The issue is that by not downvoting a question that could use improvement, it becomes more likely to hit the hot question train, and then improvement of the question becomes a nightmare. So 30% of our hot questions over the past month all got closed, which is less than ideal, because we want the attention to go to good polished questions rather than works in progress.

If close votes count against the hotness score, then we can vote to close while trying to improve the post, and not worry about it getting a dozen answers in 24 hours as the entire SE network flocks in.

##Auto-Protect Hot Questions

When a question does get popular, it tends to get more answers. If these are good answers, great! But the number of answers tends to correlate pretty well with how many will be downvoted and/or deleted. And that's not a good use of visitor resources and not likely to increase the quality of our content.

This is the sort of thing Protected Questions were designed to solve:

So, in the future, if you see a question that is attracting a lot of drive-by noise answers, please flag it for moderator attention. We’ll turn on protection. The protection bar is extraordinarily low right now — you only need >= 10 reputation to post an answer to a protected question — but we think this is enough to cut down dramatically on answer noise for certain unlucky, but obviously popular, questions.

Smaller sites don't necessarily have a mod around 24 hours a day, so I see two potential solutions:

1. Allow trusted users (perhaps requiring 2 votes) to protect a question prior to 24 hours
2. Allow all sites to have a threshold of answers above which the question gets auto-protected (much like too many answers automatically turns them all to community wiki)

Answer 2

I propose a system similar to that implemented by Reddit. From Wikipedia (my emphasis)

Front page rank, for both the general front page and for individual reddits, is determined by the age of the submission, positive ("upvoted") to negative ("downvoted") feedback ratio and the total vote count. Dozens of submissions cycle through these front pages daily.

Such that there is some function of a question's age, number of votes, ratio of votes, and (unique to Stack Exchange) number of upvoted answers. The results of this function would rank questions across the network by "hotness" and they would be displayed accordingly. I think the most critical part of the function is the age. A question that is older should require more votes to keep it on the hot list.