I've long been fascinated with the concept of the Greatest Hits page, but I find it lacking:
The page is remarkably static. This is especially problematic since
it turns out to be expensive to build. Once you've seen it, there's
little reason to go back.
The algorithm behind the ranking is a touch obscure:
The current algorithm divides the number of page views with the total
amount of question and answer feedback received (adding a bonus for
high view counts), excluding questions with less views than the median
:- 245.
The resulting list is interesting, but I can't picture how it works.
It's not obvious what one is supposed to do with the list.
To correct those problems, I think we actually need three separate lists to replace the current page:
1. Actual greatest hits as picked by the listening audience.
The simplest problem to tackle is the algorithm. It turns out that statisticians have discovered the optimal strategy for ranking things that have received upvotes and downvotes: "Score = Lower bound of Wilson score confidence interval for a Bernoulli parameter". (Go read the article; that's honestly the most complicated bit and it's not really important that you know what it means.) We are looking only at anonymous feedback of particular posts.
In theory, the top posts might be questions, but for the sites I've checked, they are mostly answers. For instance, the top result on The Workplace is the top answer to What should I do when my dream company is interested in me before my skills are developed? On Writers, it's the top answer to How can I catch more errors when I proofread? As for Stack Overflow, the 34th result suggests using an XML parser, but the top result is a fairly comprehensive comparison between AWS and Heroku.
To be clear, these might not be the sort of posts active users value. (Case in point, this rather terse answer is the second best thing on Writers according to anonymous users. It kinda seems spammy, but if you want some software to convert your screenplay between Celtx and Final Draft, this just might save your bacon.)
This list isn't necessarily useful for showing off the very best of a site, but it does give a better picture of how people are being helped in the long tail.
2. Recent popular questions.
Unfortunately, we don't have an easy way to track traffic on questions over time. Fortunately, anonymous voting correlates very strongly with views. So we can look at questions that have attracted a lot of drive-by readers in the last X days. We learn that people are playing a lot of Clash of Clans still. Looking at Judaism, it turns out that the sabbath year started recently.
If the answers people are getting are subpar when they come with those particular questions, it might be a good idea to improve them. But it can also just be useful to know what's caught the public interest on your site lately.
3. Fixing embarrassing posts.
Finally, there's a use case that the current Greatest Hits page serve, but not well: potentially out-of-date or otherwise bad posts. By reversing what we are looking for from the first list, we can find some on Stack Overflow, such as this reasonable-seeming answer that happens to be horribly, horribly wrong. (Thankfully it's also downvoted to oblivion. Unfortunately, it's an accepted answer.) Another example is this potentially dangerous answer on Unix & Linux.
Where should these lists reside?
Trying to think of how these lists could be presented, I wonder if jmac's idea of linking them to site reviews is not a good one. Not that I think these questions are particularly good ones to review; rather, I think a link to the queries might help people understand the strengths and weaknesses of their site. It might make sense to skip the reviews for graduated sites, but still post on meta periodically so that people have an opportunity for introspection.
