Mitigating publicity-driven vote inflation

Question

Following the Reddit debacle™ back in January, we've had another instance today of a question becoming overwhelmingly popular due to external "advertising".

As soon as Joel tweeted about Eric Lippert's fantastic answer, the viewcount shot up. The OP received three gold badges for the question, which itself has so far garnered 191 upvotes and been favourited 107 times... despite being, frankly, a medium-quality question at best, with a whole slew of pre-existing duplicates.

Fortunately, this scenario comes up fairly rarely, but in my opinion it does demonstrate a fundamental flaw in the Stack Exchange voting system: question/answer score isn't merely a function of upvotes and downvotes, but also of how many masses of people have had the opportunity to vote, however it was that the question became known to them.

Now, I'm almost loathed to suggest the following as it's a pretty hacky "fix", but I want to open the floor to ideas, suggestions and opinions by starting with the following draft proposal:

Question/Answer Score = Ceil((Upvotes - Downvotes) / f(Number_Of_Question_Views))

where f is some weighted, but linear, transformation.

Even if constructed such that questions with an "average" number of page views would not see any significant impact from the addition of the new factor, this would immediately result in heavy rep changes for quite a few people, and for quite a few honestly high-quality questions. This could be a practical drawback, but I reckon the result would be a much fairer distribution of rep.

And it's not just about reputation. I'm almost anticipating an onslaught of "Stack Overflow is not a competition" comments on this suggestion, but reputation is the mechanism for determining, if only partially, the strength of an individual contributor... and the strength of individual questions/answers. It's misleading to have super-highly voted questions that are orders of magnitude lower in quality than some absolutely superb questions that are stuck with a score of, say, 12.

^{The overall question of whether users should benefit from publicity has been posed before; in this question, I'm pre-supposing that the answer is no, and asking what can be done about it.}

^{It was also discussed more generally two years ago. Again, since I'm suggesting a specific "fix", I'm not considering that to be a duplicate.}

^{This question is also related, though it focuses on internal SE factors rather than the effects of massive external publicity.}

The floor is open.

Answer 1

The foundation of the Stack Exchange Q&A model is democratic voting. A Winston Churchill quote comes to mind:

Democracy is the worst form of government, except for all those other forms that have been tried from time to time

While it is true that some freak situations produce voting patterns that defy merit, it is also true that voting is terribly unfair in a multitude of ways, some extremely common. Everywhere you look voting produces poor results:

• Posts in low-view tags rarely get out of the single digits
• Trivial posts are routinely highly upvoted while hard ones nearby get only a few
• Posts are frequently overvoted based on the popularity of the author
• Voters are sometimes tricked by smooth-talking posters
• Once a post gets votes (either way), future voters are often less cautious
• Voting cultures are different in essentially non-overlapping tag areas
• Posts that are pranks or joke answers can become wildly popular

And the list goes on and on: democracy is an extremely dull tool, but it is sharper than all the alternatives.

The good news is that inequities tend to get smoothed out over time and what looks bad under the microscope looks better in the aggregate. Common unfair outcomes can be mitigated with special case rules like the reputation cap. Other caps can be imagined.

In fact the real tragedy is not that a high-view question unfairly rewarded the reputation of the asker and answerers, but that the merits of those particular posts become less meaningful. And this is actually a common outcome where the votes don't match up with the technical merits of the posts.

This just is what it is. It's the system we've chosen for all its flaws. The best we can do is add some checks and balances for the most egregious inequities. I think in this case the checks worked and we should just accept these outliers for what they are: outliers.

Answer 2

(tl;dr: The top 50-100 are already crowded with worse, more popular questions. If a vote limiter is put into place it won't help move better questions to the top, without cleaning up the top first.)

---

I think it is pretty telling when that question, while having received far more credit than it deserves, isn't even in the top 50 overall. Let's be honest, if you forget it is a dupe and look at this question compared to the top 50 overall, it's actually pretty good!

1. It has concise, digestible code.
2. It has applicable output.
3. It has an answerable question.

This is the sort of question--or the older questions that this is a dupe of--that should be in the top 50!

Instead, the top 50 are all "popular" because of votes rather than "technical merit". Only 4 of the top 50 are real, actual problems being solved by the community:

• Flash CS4 refuses to let go: The question is poor, but it has refreshing, actual code.

• Cycles in family tree software: No code, but a problem to be solved, even if it may be a joke.

• RegEx match open tags except XHTML self-contained tags: This poor guy got thrown under the bus for having a "bad" solution in his question, but he has a solvable problem...and code!

• Why doesn't gcc optimize aaaaaa to (aaa)(aaa)?: CODE! RESEARCH! Be still my beating heart....I'm actually shocked people voted for this.

Also in the top 50 are a smattering of interesting questions that show research and likely should exist as fascinomas, but fundamentally the questions with technical merit are not the questions the community deems "useful" if we use community votes as a metric.

Placing a vote cap/ratio won't help questions with technical merit rise to the top because the reality of behavior on SO is the community at large doesn't spend a lot of time voting on questions (and we've known it for years). They even added a badge to help!

Further, a vote cap/ratio won't keep tweeted/famous questions from going to the top, because it can't get there! The top is already crowded with famous, bad questions...

I'm not sold on a better cap, but whatever system (if any) gets put in place needs to include deleting, closing, or moving (archive.stackoverflow.com?) the vast majority of the other popular questions. Otherwise, how will the new better questions move to the top?

Answer 3

This is a community-generated site. It works by getting lots of people to make different sized contributions which, we hope, in the aggregate, collectively produce an artifact that makes the Internet better.

Collectively generating useful artifacts is tricky. Wikipedia was one of the first and most amazing examples of the phenomenon. See, e.g., Clay Shirky, Here Comes Everybody for a book-length treatment of the subject by Clay, who is now a member of Stack Exchange's board of advisors.

There are many principles involved in making successful systems for generating large artifacts from a large number of smaller actions.

The relevant one here is that you need to allow people to contribute at different levels. At the lowest level, the only contribution you ask for is a simple vote, up or down. At a slightly higher level, you ask people to type questions or answers. At an even higher level, you ask people to moderate or contribute on a frequent basis. Each of these levels requires successive amounts of commitment. You can't create a successful site that demands that each visitor immediately exhibit the highest level of commitment. You have to provide some easy things that people can do that, nevertheless, contribute to the awesome artifact that everyone is putting together.

The way in which these sites combine lots of discrete, small actions by many people into a large artifact contains many surprises.

One such surprise is that something which appears to be a terrible question can be the grain of sand that becomes an amazing pearl of knowledge. In this case we got an amazing article by Eric Lippert about memory management out of a question which had already been asked before. Upon closer examination, though, that particular question was, in fact, an exemplary question about the particular misunderstanding that Eric cleared up. You literally could not ask for a more pared-down, simple and clear example of the code necessary to illustrate the question that Eric answered.

That question is valuable because it triggered an amazing artifact and a great explanation of a common misunderstanding. This led to thousands of programmers learning something new. As such, the very idea that it doesn't deserve the upvotes it got is almost nonsensical. Maybe there's some luck involved in the fact that this particular version of the question triggered a great response by Eric, but the truth is there was only one other pure duplicate of this question I could find on the site that had pared it down so concisely, so it's more like a 50%-50% chance rather than a lightening strike.

I think a lot of the reaction I see here ("hey, this question is terrible, why did it get so many votes?") stems from a sort of snobbery because the question was asked by someone who is closer to the beginning of their understanding of C-style languages.

The artifact we're trying to generate here at Stack Overflow is intended to be valuable to learners at all stages of their learning process. As such you need honest questions about basic things and you need amazing answers to those questions.

Answer 4

A few thoughts.

First off, on the specific question and answer:

Regarding upvotes on the answer: holy goodness, what the heck happened there? I just threw together a little analogy about what "unspecified behaviour" really means while I was waiting for the unit tests to finish. I had no idea it would strike such a chord. I don't consider it to be particularly insightful, but apparently a lot of people liked it, so, I guess that's good.

Regarding upvotes on the question: I think that despite being a duplicate, and being a pretty basic question about C/C++ programming, it really is quite a good question. The original poster, like many C novices, was apparently of the honest belief that some tool is supposed to prevent you from doing something dangerous. Disabusing people of this common belief and letting them know how much trouble they can get into with an "enough rope to hang yourself" language like C has value.

Now, to address the more general points raised in this question.

The question suggests tweaking the "scores" of questions (upvotes minus downvotes), answers (same) and participants (reputation) in particular ways. What is not clear to me, Tomalak, is what metric for goodness you believe may be improved appreciably by tweaking the algorithms.

I see a number of benefits of having a scheme of "scores" for questions, answers and users. Benefits include:

• Encouraging desirable behaviours
• Discouraging undesirable behaviours
• Identifying unusally awesome or horrid questions, answers and users

Before tweaking the system I would first ask "is it broken?" I don't care a bit about some abstract notion of "fairness". I care about two things:

• does the system work well in the vast "middle ground"? Most questions, answers and users are in the middle; the system has to serve them well.

I think you are not proposing any change that would be germane to the vast majority of normal boring questions, answers and users. My second concern is:

• in the outlying cases, does the system tend towards undesirable miscategorizations?

That is, in the outlying cases we might be:

1. misidentifying horrid questions/answers/users as awesome
2. misidentifying middle-of-the-road questions/answer/users as awesome
3. misidentifying middle-of-the-road questions/answer/users as horrid
4. misidentifying awesome questions/answer/users as horrid

In the first case, we have stuff that is really toxic that is getting the attention of something that is really good. This is clearly undesirable.

In cases three and four, we have stuff that is acceptable or awesome being voted down as horrible. This is also clearly undesirable. (See my question from yesterday and this post about deleting good answers on bad questions for more thoughts on that.)

But the situation we find ourselves in today is the second case. We've got what's arguably a middle-of-the-road question and a middle-of-the-road answer that have been identified as awesome.

Who really cares? Is that really such a big deal, compared to the other three outlying cases? I would be prioritizing:

• first, get the system working for the vast bulk of middle-of-the-road questions, answers and users.
• second, ensure that good or awesome content/users is not misidentified as undesirable
• third, ensure that bad content/users is not misidentified as desirable
• last, worry about middling content getting more attention than it deserves.

In short, I think you're trying to solve the least important problem. I would be inclined to just live with it. Sometimes mediocre questions are going to get voted way up just out of the "viral" nature of attention on the internet. It's maybe an imperfection of the system, but the system in general works well.

Answer 5

I actually think the rep cap is precisely the mitigation you're looking for. It distringuishes between a contribution that helps people again and again over time, and one that just got a lot of attention at one instant, perhaps with the help of publicity. After 20 votes on a single day, the rewards stop. Sure, people will see that big giant upvote number, and some badges, but Eric didn't suddenly get 10K powers overnight from nothing more than writing that (amazingly good) answer.

I think some of what Eric has written on SO and P.SE in the last few weeks has been astonishing. It can only attract people to these sites, and we are damn lucky that he happened to be having the kind of week that led to these answers being written. I am all for rewarding that, and rewarding askers for happening to trigger it. But I note with pleasant surprise that the rep cap served an interesting and entirely appropriate purpose here. Well done, system!

Answer 6

I definitely think that this is a problem. Reputation and votes should be directly proportional to the value of the question or answer they are awarded for. Not that I'm trying to take anything away from Eric or the questioner here, it's not like it's a bad question or answer, but was Eric's answer worth a thousand upvotes? I don't think so. This is especially since niche questions that actually have difficult answers that require a lot of efforts receive relatively low rep because they apply to less people- C++ TMP questions tend to suffer from this, I've noticed, and some hardware/software-specific questions also.

I can't help but agree- the rep should be proportional to the number of people who viewed. If you only got viewed by 50 people but 25 of them voted you up, that should definitely be worth more than if you got 1500 upvotes from 45k views. I just made that up and don't know the numbers, but the point is the same.

Answer 7

I had this happen to one of my answers on Server Fault. Jeff and/or Joel tweeted it and it made a small splash on Hacker News and both my answer and the question got a lot of votes. I think (well hope) the person who asked the question wouldn't be too unhappy if I said that several of the answers to that question were an example of good answers being the making of an average question.

The upboat was very nice, but regardless of whether not the question or any of the answers "deserved" that much attention, the fact is that posts of either kind that take the imagination of the web at last are a great chance to publicise the sites and build our userbase. That's more people asking the questions you love to answer or helping you solve that tricky problem that's keeping you awake at nights.

I might well be biased but I think that's worth the rep boost anyone associated with the publicised question might get. And the cap should stop it getting totally out of hand.