17

A few years back, the team rolled out a Greatest Hits page that purported to display the "public face" of a site's (or an individual user's) content. This was never linked to from within the UI, because... Well, we weren't sure what to do with it.

That's probably not gonna change; in fact, we might be killing the page entirely - it's extremely slow, and no one uses it on a regular basis. Possibly because it isn't linked to anywhere, or possibly because... Well, what the hell do you do with that information?

There's another problem too: the page differentiates itself by including views and anonymous feedback in the ranking - but the vast majority of the time, views and anonymous votes correspond pretty closely to lots of actual votes. So the highest-scored questions on the site also tend to be the greatest hits. There are exceptions - this question has a ton of views and anon feedback, but very few votes - but they're just that: very rare exceptions.

So here's my question, in three parts:

  1. Is there any real value in trying to determine what a site's greatest hits - or maybe we should say "Google Favorites" - are? And... Is /questions/greatest-hits doing that at all?
  2. If so, how can we encourage folks to actually put that information to good use? That is, where should this page be linked to from?
  3. If not, is there any real point in keeping this page around?
  • 2
    Exceptions are not so very rare on Math, where 10 out of 50 questions on the first page have single-digit score. – user259867 Nov 4 '14 at 1:46
  • 2
    The anonymous feedback data has since been added to Data Explorer anyway right? So I guess that solves the "We're collecting the data but no one sees it" aspect of that page's reason for being. – Tim Stone Nov 4 '14 at 1:56
  • I guess you just need to ask yourself: "Why would anyone find this page useful?" If the answer is "Uhh... because, umm, DATA!" then it's probably not really necessary to keep around. – Doorknob Nov 4 '14 at 2:01
  • 1
    I know what I use it for - but I have plenty of other ways of getting that same information, @Doorknob. Heck, as Tim notes, the rest of you do too - which is kind of the point of part #2: we don't need the page unless there's a specific use we want to encourage for it. – Shog9 Nov 4 '14 at 2:32
  • 1
    I see no added value to maintaining a list of the questions that already annoyingly pop up everywhere whether you want them to or not. – Jason C Nov 4 '14 at 2:44
  • 2
    just kill it, unless you find a way to handle The Trouble With Popularity (what I observe with hot questions and community protection rather strongly indicates that as of now, you can't handle it) – gnat Nov 4 '14 at 14:23
  • 1
    Just glancing at the Stack Overflow ones, unless there is a better way of crunching them it is instead "what old questions does everyone upvote"... like Title: "What is the maximum value for a int32?" body: "I can never remember that number. I need a memory rule." and answer "It's 2,147,483,647. Easiest way to memorize it is via a tattoo." - do we really want to show off those as "these are good examples of questions?" – user213963 Nov 7 '14 at 7:35
14

I have tried to use that page in the past to find very popular questions with answers that could be improved. I actually found out that I answered a question that now has more than a hundred thousand views, but only a question score of 14. The question was about soursop as a cancer treatment, and after seeing just how much traffic this one was getting I looked again in more detail to make sure that my answer was good. The addition I made was a much stronger warning against it as I found some evidence for harmful sideeffects from a component of soursop.

I think that there is considerable value in general in a mechanism that would allow us to identify questions that receive a lot of traffic consistently (as opposed to questions that got a traffic spike once). Targeting those questions for improvement should yield a better ROI than simply targeting all posts on the site.

I personally found the anonymous feedback too hard to interpret and rarely useful. I think that there is value in a page attempting to list the most popular questions and selectively improving those.

10

I've long been fascinated with the concept of the Greatest Hits page, but I find it lacking:

  1. The page is remarkably static. This is especially problematic since it turns out to be expensive to build. Once you've seen it, there's little reason to go back.

  2. The algorithm behind the ranking is a touch obscure:

    The current algorithm divides the number of page views with the total amount of question and answer feedback received (adding a bonus for high view counts), excluding questions with less views than the median :- 245.

    The resulting list is interesting, but I can't picture how it works.

  3. It's not obvious what one is supposed to do with the list.

To correct those problems, I think we actually need three separate lists to replace the current page:

Actual greatest hits as picked by the listening audience.

The simplest problem to tackle is the algorithm. It turns out that statisticians have discovered the optimal strategy for ranking things that have received upvotes and downvotes: "Score = Lower bound of Wilson score confidence interval for a Bernoulli parameter". (Go read the article; that's honestly the most complicated bit and it's not really important that you know what it means.) We are looking only at anonymous feedback of particular posts. In theory, the top posts might be questions, but for the sites I've checked, they are mostly answers. For instance, the top result on The Workplace is the top answer to What should I do when my dream company is interested in me before my skills are developed? On Writers, it's the top answer to How can I catch more errors when I proofread? As for Stack Overflow, the 34th result suggests using an XML parser, but the top result is a fairly comprehensive comparison between AWS and Heroku.

To be clear, these might not be the sort of posts active users value. (Case in point, this rather terse answer is the second best thing on Writers according to anonymous users. It kinda seems spammy, but if you want some software to convert your screenplay between Celtx and Final Draft, this just might save your bacon.)

This list isn't necessarily useful for showing off the very best of a site, but it does give a better picture of how people are being helped in the long tail.

Recent popular questions.

Unfortunately, we don't have an easy way to track traffic on questions over time. Fortunately, anonymous voting correlates very strongly with views. So we can look at questions that have attracted a lot of drive-by readers in the last X days. We learn that people are playing a lot of Clash of Clans still. Looking at Judaism, it turns out that the sabbath year started recently.

If the answers people are getting are subpar when they come with those particular questions, it might be a good idea to improve them. But it can also just be useful to know what's caught the public interest on your site lately.

Fixing embarrassing posts.

Finally, there's a use case that the current Greatest Hits page serve, but not well: potentially out-of-date or otherwise bad posts. By reversing what we are looking for from the first list, we can find some on Stack Overflow, such as this reasonable-seeming answer that happens to be horribly, horribly wrong. (Thankfully it's also downvoted to oblivion. Unfortunately, it's an accepted answer.) Another example is this potentially dangerous answer on Unix & Linux.

Where should these lists reside?

Trying to think of how these lists could be presented, I wonder if jmac's idea of linking them to site reviews is not a good one. Not that I think these questions are particularly good ones to review; rather I think a link to the queries might help people understand the strengths and weaknesses of their site. It might make sense to skip the reviews for graduated sites, but still post on meta periodically so that people have an opportunity for introspection.

  • 3
    I toyed around with the idea of "greatest hits" a year ago. Trying to build a single list is difficult because there are essentially two categories of posts: Those that are "interesting". And those that are "useful". Each category appeals to a different audience. The "interesting" posts are likely to satisfy the passerby audience. The "useful" posts satisfy those coming in directly from search engines looking for that specific answer. – Mysticial Nov 6 '14 at 3:03
  • statisticians have discovered the optimal strategy for ranking things that have received upvotes and downvotes. Indeed. So can devs please implement that as an option for answer-sorting on individual questions? That's been a feature request before: as an answer to the FGITW problem (currently with +581 score) and as its own question (strangely with a -3 score). Other sites have implemented this and their signal-to-noise ratio has noticeably improved. – Mr. Bultitude Apr 12 '17 at 12:23
5

My suggestions:

  1. Kill the page, or continue to ignore it. The list can't be re-sorted or filtered by tag... it's just a bad way to present a 1000-long list of questions. Also, the algorithm is rather opaque:

The current algorithm divides the number of page views with the total amount of question and answer feedback received (adding a bonus for high view counts)

(I don't even understand what is divided by what here, let alone why it's divided.)

  1. Instead, add a page about Data Explorer to the Help Center. This is something that is needed regardless: new users often get told "you can get that from DE", and there is no clear way of getting started there. The page should have a brief intro to DE (weekly snapshot, and so on), followed by a list of example queries with explanation of their meaning. This is where Greatest Hits should go. Except they should not be vaguely named "greatest hits" anymore: there should be specific queries such as

    • most viewed questions
    • posts with most negative feedback
    • posts with most positive feedback
    • interesting unanswered questions
    • a few other popular queries from SEDE, like one's top comments.

And a brief explanation of how one might use these data, e.g., "these answers were often marked as unhelpful by visitors: check if they are correct and complete".

Not a huge number of users will go there, and that's fine: we don't want to tell everyone with 2K to go edit high-traffic posts.

  1. An HTML page might still be useful if it is based on data not available from SEDE. One problem is that some questions get inflated high view counts from the Hot Network List, or social networks. This does not guarantee their lasting value. As Mad Scientist, I am more interested in sustained interest over time: the median number of visits per day (say, for the last 3 years, and only for questions that exist at least 3 months.) If you can get median daily viewcount from server logs... that might be an interesting list to see.
4

Is there any real value in trying to determine what a site's greatest hits - or maybe we should say "Google Favorites" - are? And... Is /questions/greatest-hits doing that at all?

There are questions on the network which help a disproportionate amount of people because they are "Google Favorites", but are virtually impossible to discover through normal browsing, yet show up on the greatest-hits lists.

Examples: (look at the views)

Because of the inordinate amount of attention these questions get, at the very least giving the community a way to recognize which questions these are, and make sure they adequately represent the community and its quality standards. There is value in that, but the current list has a lot of noise unfortunately, and probably isn't the best tool for the task anyway.

If so, how can we encourage folks to actually put that information to good use? That is, where should this page be linked to from?

One place this info would be put to good use is in /review/site-eval reviews. Because these questions get so many eyes, they are a good place to see how the site is doing, and to give the community a chance to improve them through the self-evaluation process.

If not, is there any real point in keeping this page around?

As-is, the page itself doesn't seem to serve much use on its own as it is noisy, and doesn't actually lead to any good action.

  • 2
    I don't think there would be much benefit in putting them through evals. For one, those don't even run on graduated sites (largely with good reason - in practice it just turned out to be a weird thing to have enabled on sites that have "made it"). For another, it's the "normal" questions that make up the bulk of the site, not the popular greatest-hits type stuff. Looking at the latter doesn't really give you a great picture of the site's current state at all. – Adam Lear Nov 4 '14 at 2:26
  • 2
    @AnnaLear not as a replacement for current site eval things, but rather as an extra bit of self-evaluation for questions bringing a lot of people and their quality. – jmac Nov 4 '14 at 2:27
  • I didn't figure you meant "replace". I don't think these are useful in that context. They also don't change enough between evals (unless we change up the "greatest hits" criteria). – Adam Lear Nov 4 '14 at 2:30
  • 5
    I read the first example as "How to write a polite reindeer email?" Needless to say, the actual question is far less interesting. – Jon Ericson Nov 4 '14 at 16:50
3

There's another problem too: the page differentiates itself by including views and anonymous feedback in the ranking - but the vast majority of the time, views and anonymous votes correspond pretty closely to lots of actual votes. So the highest-scored questions on the site also tend to be the greatest hits. There are exceptions - this question has a ton of views and anon feedback, but very few votes - but they're just that: very rare exceptions.

Two things, the first of which you hint at above: when anon feedback radically differs from voting, either in a good or bad way, that should generate automatic flags for review. This is probably the #1 finding of greatest hits and it's very important.

The second is hinted at in the answer from Travis; people are only interested in gardening on their greatest hits, so make sure the user page encourages them to do this. The invisible bits of feedback are anon votes, so perhaps the user page should have some kind of "greatest hits" function that folds this in and ties it into the only music you really want to listen to -- your own.

2

You guys must know best how the homepage is used by users that are not logged in, however this seems to be the (second*) best usage of the homepage for new not-yet-registered users. It shows exactly the kind of questions I would have expected on the homepage for a new user that has come to visit the site.

Advantages:

  • Presents the user with interesting questions
  • Presents the user with high quality questions, thus the user is more likely to ask high quality questions as well

Disadvantages:

  • A non-new non-registered user would be relatively bored by the homepage. Then again, I doubt there are a lot of lurkers that actively visit the site (vs arriving via Google or other Search engines)

Either way, this would require some proper analytical tools to see how much sense this would make (or some A/B testing), but to me these questions seem to be even better than the top voted question list.

* First would be my proposal over on meta.SO O:) , but that's just for popular SE sites either way.

  • 2
    My only issue with this is the list never really changes and it is difficult for new items to break through. If it were to be used for something like this I would like to see the rank on the list decay with question age. – Jason C Nov 4 '14 at 3:01
  • 2
    @JasonC: My assumption here is that a user only would see this list once (or close to once) in their life (at that point they either start using the site and register or leave forever), so it never changing would be mostly a non issue. But once again, I know little about how those users truly behave. I know that I have 'shown off' various SE sites to friends by going to the top voted pages however and to me it seems sensible if that's what people would end up with if they search in google for "Skeptics Stack Exchange" after hearing somebody mention it in real life. – David Mulder Nov 4 '14 at 3:05
  • 3
    SE tried this kind of page back in 2013. It was criticized on this meta, hated on Math (and possibly other sites), and soon ditched. – user259867 Nov 4 '14 at 3:05
  • @justabrickinthewall: Ah, that's interesting to hear. Do you (or anybody else) know whether any A/B tests were run (and shared) back then? – David Mulder Nov 4 '14 at 3:08
  • @DavidMulder Hm; that seems like a reasonable assumption to me. I might still be subconsciously clinging to ways to keep it interesting for existing users. – Jason C Nov 4 '14 at 3:10
  • Been a long time now, but I'd be shocked if we didn't A/B test that, @David - however, that doesn't mean we looked at the right results. If your goal is to increase click-through, putting more popular questions in front of folks is extremely likely to do just that - however, this effect might not last, and it completely ignores other results (answers to unanswered questions, time spent elsewhere on the site, etc.) – Shog9 Nov 4 '14 at 3:17
  • @Shog9: Ah, personally I was just thinking of registration rates~, but yeah, I get your point. Been trying to dig up any published reasons for the change back then, but can't seem to dig up anything. – David Mulder Nov 4 '14 at 3:23
  • One other disadvantage is the date asked, on some sites most questions will be years old and it might make the sites look like a ghost town with no new content. Might also encourage new low-quality answers on questions that have already had stacks of attention / answers. – PeterJ Nov 4 '14 at 11:00
  • 1
    Considering the annoying account management on stackexchange, there are quite a few sites in the network I occasionally visit without being logged in. – CodesInChaos Nov 4 '14 at 12:28
2

I happened to like David Mulder's suggestion of using it as the front page for logged out users, but a commenter pointed out that it was tried and failed before.

So, taking the pros and cons of it, maybe it should be combined with the current system.

The homepage would show a mix of the recent/hot questions, plus some of the "greatest hits". The hits could possibly be marked somehow as "this is a great question" - maybe with the yellow background usually used for followed tags?

If this was added, it would also make sense to remove the timestamp, and just show the modifier + their rep.

This way, a visitor to the site would get more flavor - new, unanswered questions; answered questions; currently hot; greatest hits.

1

Overall it is one of those "huh, that's interesting" links that you can go to and browse. Essentially it is the same as clicking to sort on "votes" in the questions tab. But there are some small nuanced differences that really shine in certain scenarios.

What scenarios? Glad you asked. Although this feature of "Greatest Hits" has been removed, you used to be able to use your userId, and there was even a link that is no longer present, in order to get the "Greatest Hits" questions that you participated in.

For example, mine on SO was https://stackoverflow.com/questions/greatest-hits?userId=1026459 which no longer works. Sad face. To me that was the value of this feature because it was interesting and a little fun to review at times. There were some posts that I did not realize were so popular in there.

So, I think that the user profile could link to the greatest hits list that a user has participated in because it is fun, and users like fun features.

If the user query is removed from the greatest hits feature and there is no plan to bring it back then burn the whole thing down in my opinion, because the heart of what made that feature useful is gone.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .