What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

Following the Reddit debacle™ back in January, we've had another instance today of a question becoming overwhelmingly popular due to external "advertising".

As soon as Joel tweeted about Eric Lippert's fantastic answer, the viewcount shot up. The OP received three gold badges for the question, which itself has so far garnered 191 upvotes and been favourited 107 times... despite being, frankly, a medium-quality question at best, with a whole slew of pre-existing duplicates.

Fortunately, this scenario comes up fairly rarely, but in my opinion it does demonstrate a fundamental flaw in the Stack Exchange voting system: question/answer score isn't merely a function of upvotes and downvotes, but also of how many masses of people have had the opportunity to vote, however it was that the question became known to them.

Now, I'm almost loathed to suggest the following as it's a pretty hacky "fix", but I want to open the floor to ideas, suggestions and opinions by starting with the following draft proposal:

Question/Answer Score = Ceil((Upvotes - Downvotes) / f(Number_Of_Question_Views))

where f is some weighted, but linear, transformation.

Even if constructed such that questions with an "average" number of page views would not see any significant impact from the addition of the new factor, this would immediately result in heavy rep changes for quite a few people, and for quite a few honestly high-quality questions. This could be a practical drawback, but I reckon the result would be a much fairer distribution of rep.

And it's not just about reputation. I'm almost anticipating an onslaught of "Stack Overflow is not a competition" comments on this suggestion, but reputation is the mechanism for determining, if only partially, the strength of an individual contributor... and the strength of individual questions/answers. It's misleading to have super-highly voted questions that are orders of magnitude lower in quality than some absolutely superb questions that are stuck with a score of, say, 12.

The overall question of whether users should benefit from publicity has been posed before; in this question, I'm pre-supposing that the answer is no, and asking what can be done about it.

It was also discussed more generally two years ago. Again, since I'm suggesting a specific "fix", I'm not considering that to be a duplicate.

This question is also related, though it focuses on internal SE factors rather than the effects of massive external publicity.

The floor is open.

share|improve this question
2  
Eric was rep-capped shortly after the question was tweeted. –  Robert Harvey Jun 23 '11 at 23:18
1  
@Robert: The question wasn't, though. –  Lightness Races in Orbit Jun 23 '11 at 23:19
1  
The question? Avrahamshuk was rep-capped also. –  Robert Harvey Jun 23 '11 at 23:21
2  
@Robert What about question/answer score (which I think is more important here)? –  R. Martinho Fernandes Jun 23 '11 at 23:27
    
Yes, what @Martinho said. (And let's not forget those three gold badges.) –  Lightness Races in Orbit Jun 23 '11 at 23:31
14  
@Robert: My gripes isn't so much with the rep. But I think the question is poor, and it definitely is a dupe of many others and should have been closed as such, instead of being upvoted. In fact, it even was closed as a dupe (and I flagged it for merging Eric's answer into one of the others) but Joel reopened it and merged an older question into this one instead. Whether Eric's answer is worth 1k upvotes is debatable, but as a side-effect the question has gained almost two hundred upvotes more than it deserves. And not closing such an obvious dupe sets a bad precedent. –  sbi Jun 23 '11 at 23:32
    
@Martinho What do you mean by "question/answer score"? What is that supposed to measure? –  Anna Lear Jun 23 '11 at 23:33
2  
When I first read this Meta post, I was going to say it wasn't a problem. Then I saw the score on the answer and was like o_O. –  waiwai933 Jun 24 '11 at 3:02
10  
@sbi: Closing and deleting questions that have exemplary answers on them, regardless of the quality of the question, amounts to content destruction. I don't care how obvious the dupe is or how bad the question may seem, closing and deleting questions without considering the answers is just irresponsible. As Jeff points out, what we are really looking for are quality answers, not quality questions, so if someone like Eric Lippert can make a silk purse out of a sow's ear, I say more power to him. –  Robert Harvey Jun 24 '11 at 21:36
2  
@Robert: Yeah, all those irresponsible people that close duplicates, how dare they. Or even worse, merging these duplicates. Is that what you are saying? And who is even talking about deleting? –  sth Jun 25 '11 at 3:02
1  
@Tomalak: True, that. –  Robert Harvey Jun 25 '11 at 4:01
4  
@Robert: Merging was invented to salvage quality answers from bad questions. And, as I wrote in my comment you replied to, "I flagged it for merging Eric's answer into one of the others". Nobody wanted to get rid of Eric's great answer. We wanted to get rid of the bad question it was given to. –  sbi Jun 25 '11 at 5:40
2  
And, really, if it was one of us mere minions who had given such an answer, who'd have hesitated to close a dupe and ditch it? –  sbi Jun 25 '11 at 5:41
1  
1) Merging only works on questions that are exact duplicates. 2) It's not about the person posting the answer, it's about the quality of the answer. –  Robert Harvey Jun 25 '11 at 5:44
2  
@Robert: Being an exact duplicate of an earlier question is bad beyond all means and a perfect reason to close a question; no further reason needed, and the one exception to this rule I would consider valid (this one being a much better question than the ones before it) doesn't apply. And, FWIW, I consider it an abuse of your mod powers that you cut down the list of questions this one was a dupe of to a single one and then prevented everybody else from adding to it. –  sbi Jun 26 '11 at 11:13

7 Answers 7

The foundation of the Stack Exchange Q&A model is democratic voting. A Winston Churchill quote comes to mind:

Democracy is the worst form of government, except for all those other forms that have been tried from time to time

While it is true that some freak situations produce voting patterns that defy merit, it is also true that voting is terribly unfair in a multitude of ways, some extremely common. Everywhere you look voting produces poor results:

  • Posts in low-view tags rarely get out of the single digits
  • Trivial posts are routinely highly upvoted while hard ones nearby get only a few
  • Posts are frequently overvoted based on the popularity of the author
  • Voters are sometimes tricked by smooth-talking posters
  • Once a post gets votes (either way), future voters are often less cautious
  • Voting cultures are different in essentially non-overlapping tag areas
  • Posts that are pranks or joke answers can become wildly popular

And the list goes on and on: democracy is an extremely dull tool, but it is sharper than all the alternatives.

The good news is that inequities tend to get smoothed out over time and what looks bad under the microscope looks better in the aggregate. Common unfair outcomes can be mitigated with special case rules like the reputation cap. Other caps can be imagined.

In fact the real tragedy is not that a high-view question unfairly rewarded the reputation of the asker and answerers, but that the merits of those particular posts become less meaningful. And this is actually a common outcome where the votes don't match up with the technical merits of the posts.

This just is what it is. It's the system we've chosen for all its flaws. The best we can do is add some checks and balances for the most egregious inequities. I think in this case the checks worked and we should just accept these outliers for what they are: outliers.

share|improve this answer
1  
Even I real life a lot of people vote for the wrong party. :-) –  Bo Persson Jun 24 '11 at 7:32
    
There is nothing undemocratic about using "votes per 100 views"... –  Ian Ringrose Jun 24 '11 at 8:20
    
"In fact the real tragedy is not that a high-view question unfairly rewarded the reputation of the asker and answerers, but that the merits of those particular posts becomes less meaningful." That's what I'm saying. –  Lightness Races in Orbit Jun 24 '11 at 8:45
3  
The only comparison to democracy here would be if one party got a billion dollars to spend on advertising on the Internet, TV, etc, and the other could only stand around with megaphones in a single city. –  DeadMG Jun 24 '11 at 12:33
    
@DeadMG: Parties success is at least partially decided by how much money they have to spend on campaigning. The way they appropriate that cash often has absolutely nothing to do with the merits of their proposed policies. I think the analogy is quite apt, because to my mind this is one of the fundamental flaws in a capitalist democracy. Not that there are any particularly fairer systems: perhaps the analogy extends there too. –  Lightness Races in Orbit Jun 27 '11 at 22:00

(tl;dr: The top 50-100 are already crowded with worse, more popular questions. If a vote limiter is put into place it won't help move better questions to the top, without cleaning up the top first.)


I think it is pretty telling when that question, while having received far more credit than it deserves, isn't even in the top 50 overall. Let's be honest, if you forget it is a dupe and look at this question compared to the top 50 overall, it's actually pretty good!

  1. It has concise, digestible code.
  2. It has applicable output.
  3. It has an answerable question.

This is the sort of question--or the older questions that this is a dupe of--that should be in the top 50!

Instead, the top 50 are all "popular" because of votes rather than "technical merit". Only 4 of the top 50 are real, actual problems being solved by the community:

Also in the top 50 are a smattering of interesting questions that show research and likely should exist as fascinomas, but fundamentally the questions with technical merit are not the questions the community deems "useful" if we use community votes as a metric.

Placing a vote cap/ratio won't help questions with technical merit rise to the top because the reality of behavior on SO is the community at large doesn't spend a lot of time voting on questions (and we've known it for years). They even added a badge to help!

Further, a vote cap/ratio won't keep tweeted/famous questions from going to the top, because it can't get there! The top is already crowded with famous, bad questions...

I'm not sold on a better cap, but whatever system (if any) gets put in place needs to include deleting, closing, or moving (archive.stackoverflow.com?) the vast majority of the other popular questions. Otherwise, how will the new better questions move to the top?

share|improve this answer
2  
An answer that actually addresses the question, with RESEARCH! EVIDENCE! +1! –  Josh Caswell Jun 24 '11 at 3:47
    
The gcc optimization actually was mentioned by Reddit as well - it got the same effect that the scope one we're discussing now did. Although, admittedly, half as many votes for the answer. –  Grace Note Jun 24 '11 at 11:47

This is a community-generated site. It works by getting lots of people to make different sized contributions which, we hope, in the aggregate, collectively produce an artifact that makes the Internet better.

Collectively generating useful artifacts is tricky. Wikipedia was one of the first and most amazing examples of the phenomenon. See, e.g., Clay Shirky, Here Comes Everybody for a book-length treatment of the subject by Clay, who is now a member of Stack Exchange's board of advisors.

There are many principles involved in making successful systems for generating large artifacts from a large number of smaller actions.

The relevant one here is that you need to allow people to contribute at different levels. At the lowest level, the only contribution you ask for is a simple vote, up or down. At a slightly higher level, you ask people to type questions or answers. At an even higher level, you ask people to moderate or contribute on a frequent basis. Each of these levels requires successive amounts of commitment. You can't create a successful site that demands that each visitor immediately exhibit the highest level of commitment. You have to provide some easy things that people can do that, nevertheless, contribute to the awesome artifact that everyone is putting together.

The way in which these sites combine lots of discrete, small actions by many people into a large artifact contains many surprises.

One such surprise is that something which appears to be a terrible question can be the grain of sand that becomes an amazing pearl of knowledge. In this case we got an amazing article by Eric Lippert about memory management out of a question which had already been asked before. Upon closer examination, though, that particular question was, in fact, an exemplary question about the particular misunderstanding that Eric cleared up. You literally could not ask for a more pared-down, simple and clear example of the code necessary to illustrate the question that Eric answered.

That question is valuable because it triggered an amazing artifact and a great explanation of a common misunderstanding. This led to thousands of programmers learning something new. As such, the very idea that it doesn't deserve the upvotes it got is almost nonsensical. Maybe there's some luck involved in the fact that this particular version of the question triggered a great response by Eric, but the truth is there was only one other pure duplicate of this question I could find on the site that had pared it down so concisely, so it's more like a 50%-50% chance rather than a lightening strike.

I think a lot of the reaction I see here ("hey, this question is terrible, why did it get so many votes?") stems from a sort of snobbery because the question was asked by someone who is closer to the beginning of their understanding of C-style languages.

The artifact we're trying to generate here at Stack Overflow is intended to be valuable to learners at all stages of their learning process. As such you need honest questions about basic things and you need amazing answers to those questions.

share|improve this answer
5  
All true. But what do you think about the weighted equation the OP posted? Could it be put to some useful purpose? You haven't really addressed the "bikeshed" factor. –  Robert Harvey Jun 24 '11 at 20:36
6  
I don't think it makes any sense. A question that garnered 100,000 views did 100 times more good for the world than a question that garnered 1000 views. If it gets a few more votes it deserves them. If anything the fact that so many reddit/ycombinator/twitter users do not have accounts and can't vote makes me think perhaps such a question deserves a BOOST in votes, if anything. –  Joel Spolsky Jun 24 '11 at 20:48
4  
I think that your assumption of snobbery on the part of those concerned about the upvotes for the question is mistaken. You asserted on the question itself «dupe answer doesn't mean dupe question»; by the exact same token, good (even fantastic) answer doesn't mean good question. –  Josh Caswell Jun 24 '11 at 21:11
1  
@Josh: Mistaken and slightly insulting. –  Lightness Races in Orbit Jun 25 '11 at 17:32
1  
@Joel: So are you suggesting that a comparison of answer scores between different questions has no intrinsic meaning? My main issue was that, despite Eric's answer being great, when you look at the sort of scores that answers usually get, 1548 is clearly on an entirely different scale... and this is only because you happened to tweet it. My suggestion was designed to take us towards a system where scores, as an indication of question or answer quality, can be reasonably interpreted on the same scale, no matter which question you're reading or how many people happen to have seen it. –  Lightness Races in Orbit Jun 25 '11 at 17:36
1  
See, to my mind, an absolute number of votes has no meaning at all if the factor of how many votes could have been cast is ignored from the equation. –  Lightness Races in Orbit Jun 25 '11 at 17:38
    
@Tomalak: but what about all those people who don't have an account? How do you determine how many votes, in which direction, and on which posts they would have cast? That's pure speculation. And you can't just extrapolate from the behaviour of registered users who have taken their time to earn the right to vote. They are, by definition, very different from random passers-by. And if you just ignore the passers-by, then you're right back to square one, because the current system already does just that: every single vote on a popular question ultimately comes from within the community. –  ЯegDwight Jun 27 '11 at 22:30
    
@RegDwight: Mm. Tricky! –  Lightness Races in Orbit Jun 27 '11 at 23:31

A few thoughts.

First off, on the specific question and answer:

Regarding upvotes on the answer: holy goodness, what the heck happened there? I just threw together a little analogy about what "unspecified behaviour" really means while I was waiting for the unit tests to finish. I had no idea it would strike such a chord. I don't consider it to be particularly insightful, but apparently a lot of people liked it, so, I guess that's good.

Regarding upvotes on the question: I think that despite being a duplicate, and being a pretty basic question about C/C++ programming, it really is quite a good question. The original poster, like many C novices, was apparently of the honest belief that some tool is supposed to prevent you from doing something dangerous. Disabusing people of this common belief and letting them know how much trouble they can get into with an "enough rope to hang yourself" language like C has value.

Now, to address the more general points raised in this question.

The question suggests tweaking the "scores" of questions (upvotes minus downvotes), answers (same) and participants (reputation) in particular ways. What is not clear to me, Tomalak, is what metric for goodness you believe may be improved appreciably by tweaking the algorithms.

I see a number of benefits of having a scheme of "scores" for questions, answers and users. Benefits include:

  • Encouraging desirable behaviours
  • Discouraging undesirable behaviours
  • Identifying unusally awesome or horrid questions, answers and users

Before tweaking the system I would first ask "is it broken?" I don't care a bit about some abstract notion of "fairness". I care about two things:

  • does the system work well in the vast "middle ground"? Most questions, answers and users are in the middle; the system has to serve them well.

I think you are not proposing any change that would be germane to the vast majority of normal boring questions, answers and users. My second concern is:

  • in the outlying cases, does the system tend towards undesirable miscategorizations?

That is, in the outlying cases we might be:

  1. misidentifying horrid questions/answers/users as awesome
  2. misidentifying middle-of-the-road questions/answer/users as awesome
  3. misidentifying middle-of-the-road questions/answer/users as horrid
  4. misidentifying awesome questions/answer/users as horrid

In the first case, we have stuff that is really toxic that is getting the attention of something that is really good. This is clearly undesirable.

In cases three and four, we have stuff that is acceptable or awesome being voted down as horrible. This is also clearly undesirable. (See my question from yesterday and this post about deleting good answers on bad questions for more thoughts on that.)

But the situation we find ourselves in today is the second case. We've got what's arguably a middle-of-the-road question and a middle-of-the-road answer that have been identified as awesome.

Who really cares? Is that really such a big deal, compared to the other three outlying cases? I would be prioritizing:

  • first, get the system working for the vast bulk of middle-of-the-road questions, answers and users.
  • second, ensure that good or awesome content/users is not misidentified as undesirable
  • third, ensure that bad content/users is not misidentified as desirable
  • last, worry about middling content getting more attention than it deserves.

In short, I think you're trying to solve the least important problem. I would be inclined to just live with it. Sometimes mediocre questions are going to get voted way up just out of the "viral" nature of attention on the internet. It's maybe an imperfection of the system, but the system in general works well.

share|improve this answer
    
I can't disagree with you on any of these points. But, it's clearly a thing so I thought it worth bringing up, whether we decide in the end that it's important or not. I still think that these edge cases represent a symptom of scenarios that we're just lucky happen so rarely at present; that the voting system doesn't represent "fairness" across different questions is a problem when we define "quality" as score. I honestly don't know how much of a "big deal" it is, but it does make me anxious enough to have opened this discussion. :) –  Lightness Races in Orbit Jun 27 '11 at 21:58
2  
@Tomalak: Oh, absolutely I think it is reasonable to bring this up and discuss it. Better to actually make a rational decision based on consideration of the facts rather than just let it slide and hope for the best. –  Eric Lippert Jun 27 '11 at 22:00
    
That's the hope! –  Lightness Races in Orbit Jun 27 '11 at 22:07

I actually think the rep cap is precisely the mitigation you're looking for. It distringuishes between a contribution that helps people again and again over time, and one that just got a lot of attention at one instant, perhaps with the help of publicity. After 20 votes on a single day, the rewards stop. Sure, people will see that big giant upvote number, and some badges, but Eric didn't suddenly get 10K powers overnight from nothing more than writing that (amazingly good) answer.

I think some of what Eric has written on SO and P.SE in the last few weeks has been astonishing. It can only attract people to these sites, and we are damn lucky that he happened to be having the kind of week that led to these answers being written. I am all for rewarding that, and rewarding askers for happening to trigger it. But I note with pleasant surprise that the rep cap served an interesting and entirely appropriate purpose here. Well done, system!

share|improve this answer
2  
"After 20 votes on a single day, the rewards stop" - the rewards for the person stop, but the rewards for the question/answer do not. You're right, though, that the repcap system did a damn good job on a individual level. –  Lightness Races in Orbit Jun 25 '11 at 17:40
2  
Perhaps what is good for the Goose is good for the Gander. If we cap people, why not questions? They both are "gaining" attention using the same system, right? –  Edwin Buck Nov 30 '11 at 20:33

I definitely think that this is a problem. Reputation and votes should be directly proportional to the value of the question or answer they are awarded for. Not that I'm trying to take anything away from Eric or the questioner here, it's not like it's a bad question or answer, but was Eric's answer worth a thousand upvotes? I don't think so. This is especially since niche questions that actually have difficult answers that require a lot of efforts receive relatively low rep because they apply to less people- C++ TMP questions tend to suffer from this, I've noticed, and some hardware/software-specific questions also.

I can't help but agree- the rep should be proportional to the number of people who viewed. If you only got viewed by 50 people but 25 of them voted you up, that should definitely be worth more than if you got 1500 upvotes from 45k views. I just made that up and don't know the numbers, but the point is the same.

share|improve this answer
4  
Looks like Eric got about 174 rep from that question. So says his reputation page, at any rate. Is there any harm in the answer receiving 1000 upvotes beyond potentially not being "worth it"? –  Anna Lear Jun 23 '11 at 23:28
6  
@Anna: If you're looking for a stellar question to read, you're going to do so on the merits of the voting system. This question breaks that system and pushes truly stellar questions off the list. The question doesn't deserve at all to be rewarded just because one of the answers was marvellous. Similarly, though it's a great answer, the number of upvotes alongside it is simply ridiculous when considered in comparison to other fantastic answers on SO. –  Lightness Races in Orbit Jun 23 '11 at 23:32
    
@DeadMG: Thanks for phrasing my thinking in another way; always handy! –  Lightness Races in Orbit Jun 23 '11 at 23:33
2  
@Tomalak I see your point regarding the question, but I still don't see a problem with the votes on the answer. Does it really matter how many votes it got? Who is it harming? Going back to the question votes for a moment, how many stellar questions are being harmed by this? It may be just my perception, but it seems like there are enough mediocre low-vote type questions that anything with, say, over 10 votes is worth reading. Whether it has 10 or 100 votes isn't that big a deal. –  Anna Lear Jun 23 '11 at 23:35
2  
@Anna: If it doesn't matter how many votes an answer gets, then why do we vote on them at all? –  Lightness Races in Orbit Jun 23 '11 at 23:39
    
@Tomalak I'm not saying voting is pointless. My question is, does it matter if an answer gets 20 votes or 1000? Thousands of people who visited that question could've voted for other answers, but for the most part didn't. I think that speaks to the quality of the answer. –  Anna Lear Jun 23 '11 at 23:40
3  
@Anna: If voting isn't pointless, then a difference between a score of 20 and a score of 1000 inherently does "matter", yes. That the answer is the highest-quality answer and thus has more votes than the other answers is all well and good, but it's become one of the highest-voted answers on SO not through its own merits, but because of some arbitrary external influence on the view count. It's simply not what the answer score is supposed to represent. –  Lightness Races in Orbit Jun 23 '11 at 23:43
5  
@Anna: I think the harm is mostly to people who get there from searches later. At least IMO, much of the legitimate reason for having scores on questions and answers is precisely to guide them to the questions and answers that are likely to be of the greatest help for them. When a rather mediocre question with a fairly good answer out-scores much better questions with at least as good of answers, the scores become not merely useless, but downright harmful (misleading). –  Jerry Coffin Jun 23 '11 at 23:49
9  
@Anna: When votes appear to come primarily for a question/answer being tweeted, there's a very good change (bordering on certainty) that the people giving the up-votes haven't looked at (or even looked for) the other questions or answers. In short, it says a great deal about the number of people who follow Joel's tweets, and next to nothing about the quality of the answer in question. –  Jerry Coffin Jun 23 '11 at 23:52

I had this happen to one of my answers on Server Fault. Jeff and/or Joel tweeted it and it made a small splash on Hacker News and both my answer and the question got a lot of votes. I think (well hope) the person who asked the question wouldn't be too unhappy if I said that several of the answers to that question were an example of good answers being the making of an average question.

The upboat was very nice, but regardless of whether not the question or any of the answers "deserved" that much attention, the fact is that posts of either kind that take the imagination of the web at last are a great chance to publicise the sites and build our userbase. That's more people asking the questions you love to answer or helping you solve that tricky problem that's keeping you awake at nights.

I might well be biased but I think that's worth the rep boost anyone associated with the publicised question might get. And the cap should stop it getting totally out of hand.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .