What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

In science, peer review works a bit like upvotes: person A says that "yes, I think this work doesn't have any major flaws". Over time people who have published lots of work that has got such approval accrue reputation.

So far this is reminiscent of Stack Overflow.

But in science, if someone with a lot of reputation says that a certain work has no flaws, this has a lot of weight, while a crackpot with zero reputation saying the same thing contributes pretty much nothing. Similarly, if a reputable person says "this unsolved question is a good one to answer", that has a lot more weight than if, say, I were to say that.

This seems to be a very sane and sensible system.

Why doesn't Stack Overflow work the same? Do you reckon it wouldn't work for Stack Overflow if something like that were attempted? Why not?

share|improve this question
29  
I'm pretty sure the history of Science is littered with cases where new (correct) ideas were not accepted due to opposition from established figures. –  Martin Smith Sep 18 '10 at 14:17
2  
@romkyns, see Why are people downvoting my question on Meta? (Though I can only assume that applies to the downvotes for this very question too.) –  Arjan Sep 18 '10 at 15:31
    
@Arjan thanks!! –  romkyns Sep 18 '10 at 15:47
1  
Science is too busy working on the flying car and meals in pill form. –  user149432 Sep 18 '10 at 17:50
3  
See: If a user has good rep on a particular tag, shouldn't his votes on that tag weight more ?, Weighted Down-Voting based on Reputation and Indicate How Trustworthy the UpVotes Are (short answer: SO rep doesn't necessarily correspond to real-world reputation, peer respect, or expertise.) –  Shog9 Sep 18 '10 at 17:54
7  
That being said, I have a crapload of rep points here, and just publicly criticized your proposal. Let's see how that works out... –  Shog9 Sep 18 '10 at 17:55
    
@shog seems to me that you argument goes around in a circle. Within the closed frog-pond of SO, rep means something. If it doesn't mean anything that translates into 'votes have more weight', then what does it mean? –  Rosinante Sep 19 '10 at 21:27
4  
That's funny, I always thought that in science, claims were tested and accepted or rejected based on actual evidence, not the reputation of the parties making or criticizing the claims. Are you sure you're not talking about religion? –  Aarobot Sep 19 '10 at 23:07
    
@Rosinante: it means you've been participating. And that's about it, really. –  Shog9 Sep 19 '10 at 23:39
2  
@aaro I've met any number of perfectly credible scientists who have been squashed by political peer review. Or, for big fun, read about Hilbert, Einstein, and Brouwer. –  Rosinante Sep 20 '10 at 0:36
    
@Shog practically, sure. I agree. However, the only justification for my 10K powers is that my rep somehow indicates that I more trusted with power tools than the next person down the scale. If you assume that I have better taste in close votes, why not assume the same thing about all other votes? –  Rosinante Sep 20 '10 at 0:37
    
@Rosinante: you don't get rep for closing. And there's no meta-moderation on SO. So there's no possible way reputation could be accurately connected to your skill in voting to close. The assumption is merely that if you've spent enough time on the site to amass 10K points, you've had the opportunity to see which questions are considered appropriate, to participate in those decisions, and to learn what is and isn't appropriate. Doesn't mean you have - indeed, there are users who rarely if ever use those tools. But at least we're not handing them to someone who knows nothing of the site. –  Shog9 Sep 20 '10 at 1:13
2  
@Aarobot In mathematics you cannot test a proof. How do you know that Vinay Deolalikar's alleged P!=NP proof is not valid? I personally deduce that from the claims of several reputable mathematicians, because that's the only means available to me. And I am extremely glad that I don't need to also consider the votes of thousands of amateurs to "decide for myself" whether the proof holds. –  romkyns Sep 20 '10 at 10:06
    
@Rosinante OK, lets talk about Einstein. He was 26 years old in 1905 when he published ground-breaking papers on relativity and other topics. Within three years he was recognised as a leading scientist and was given his first academic post. Physicists were very open to his astonishing ideas, even though he himself was young and little known. en.wikipedia.org/wiki/Einstein –  MarkJ Sep 20 '10 at 12:56
2  
@Rosinante: There's a pretty clear difference. Participation implies familiarity with the mechanics of the system, not necessarily expertise in any of the subject matter. I'm qualified to decide that a rant against garbage collection is Subjective & Argumentative, but I'm not any more qualified than anybody else here to evaluate an answer about the inner mechanics of a garbage collector. "Weighted" votes would only have meaning if they were tag-weighted, and that system would be a nightmare to develop/maintain while also being prone to abuse. –  Aarobot Sep 20 '10 at 14:25

4 Answers 4

up vote 7 down vote accepted

As someone who has dealt with both the peer review process of traditional science and the Stack Overflow reputation system, I can tell you that they each have their flaws.

Scientific peer review is not a weighted voting system open to anyone, like you state. It is a system where only those who have earned a certain level of reputation can vote yes or no on whether a paper should be published. Once you clear that threshold, your vote as a reviewer counts as much as someone with more reputation. It is as if SO only allowed users with 3000+ reputation to vote.

As you would expect, this encourages a "good old boy" network to form. It is ridiculously hard for new investigators to get started in a scientific discipline. For example, the current average age for first-time NIH grant awardees is 43. Even if you had more of a sliding-scale reputation-based voting system, this would still be a problem.

SO has a stated goal of being welcoming to newcomers, so this approach doesn't fit with that philosophy. However, SO's current voting system does allow for popularity contests, where easily understandable or more visible questions and answers get voted above more technical and / or useful ones. Overall, though, the most useful answers seem to rise to the top over time.

The biggest problem facing SO at the moment is not a lack of quality answers, it is an increasing number of garbage questions. The reputation system does little to prevent those (aside from the negative-vote question slowdown now in place).

share|improve this answer
    
Hmmm...two different peer review processes here. Paper review occurs at the journal/conference level and does favor an in-group, but not so stiffly as the grant proposal system. This is at least in part because proposal review is a government/business/non-profit monetary tracking and due diligence issue. /also working scientist –  dmckee Sep 18 '10 at 20:54
    
@dmckee - True, but both use panels of established researchers to vote up or down papers or proposals. It's not really a weighted voting system as suggested, but one that only lets those above a certain threshold vet these items. This can lead to suppression of new researchers as well as new ideas, and present plenty of opportunities for conflicts of interest. –  Brad Larson Sep 18 '10 at 22:37

While SOs voting and reputation system certainly isn't flawless, neither is the peer review system that's common in science.
Highly reputable people can be wrong too, especially if new developments are concerned that aim to overthrow some major established view.

Additionally there are many different areas of expertise treated at SO - someone who has gained much rep in one area isn't automatically highly knowledgeable in other areas.

See also these questions for similar discussions:

Grabbing some results from those:

  • "Reputation != expertise. It is mostly an indication of how much time you spent on SO."
  • "Reputation does not mean you're an expert, because reputation can be earned by asking masses of low value questions."
  • "Reputation is too broad a measurement; it doesn't indicate that a particular voter is knowledgeable in that particular subject."

In particular devinbs answers here addresses the flaws with weighted voting well.

share|improve this answer
1  
Downvoting because of application of the tu-quoque logical fallacy. Just because the peer review system isn’t flawless, it doesn’t follow that it might not still be heaps better than SO’s system, nor does it follow that SO’s system couldn’t be improved. –  Timwi Sep 18 '10 at 15:16
3  
@Timwi: The question started out with the preposition that the science peer review system is inherently better, which is what i adress in paragraph 1. Paragraph 2 adresses one problem of applying it here directly and seeing how many discussions there already are about weighted voting i don't see why we have to repeat this here from the very beginning... –  Georg Fritzsche Sep 18 '10 at 15:58

Let's see...

  • Each sub discipline of science is a small world. It really is a reputation economy where "reputation" takes the dictionary meaning. Stack Overflow has tens of thousands of "active" users (page 100 of the users shows nearly 3000 rep at this point and page 1000 shows around 140 rep).

  • Science works mostly on new problems. Confirmation is an important part of the process, but you'll never achieve fame that way. On the other hand SOFUE mostly works on old problems and what is valued is timing, clarity, completeness, and the degree to which the question answers the problem.

  • The reputation system on SOFUE is not only (or perhaps even primarily) about how well people think of you: it is about measuring contribution for the purposes of empowering you in the maintenance of the sites. Peer review does not generate a quantitative or centralized understanding of "reputation"; the result is a distributed and vague value which requires endless committee meeting and water cool conversations to translate into control.

...and there a probably other differences.

share|improve this answer
3  
Downvoting this because it betrays a slavish adherence to the theoretical ideal that SO’s reputation system tries to achieve, entirely ignoring reality which shows that it clearly doesn’t work. The reputation system does encourage timing, but at the detriment of all the other qualities, most notably correctness and completeness. A half-arsed and superficial (and wrong in a non-obvious way) answer easily makes it to the top of the votes when it is posted sooner than a correct, complete and competent answer. This holds both for beginner questions as well as difficult expert questions. –  Timwi Sep 18 '10 at 15:14
2  
@Timwi: My experience holds that while getting on the board fast is critical, you can overcome a modest initial deficit by continuing to improve your answer. That is: having gotten an answer up early (which is indeed likely to be a bit superficial), providing a quality answer by iterative improvement is the most certain way to get a lot of votes over time. I almost never write and abandon an answer, rather I write a sign post, do a little more research or test some code and edit. Then proofread and edit; read other answers and edit again to cover common mistakes... –  dmckee Sep 18 '10 at 18:16
1  
Thanks for highlighting the flaw further. A system that encourages users to post anything before it is even close to ready, or before they’ve even done their research, is clearly stupid. –  Timwi Sep 19 '10 at 7:24
1  
@Timwi: You're wrong about the FGITW. That's a feature, not a bug. It is a large part of what makes SO so fast. And it doesn't help to pos incorrect answer the try to fix hem. It only works if you post correct (if incomplete or superficial) info. –  dmckee Sep 19 '10 at 14:50
    
The fact that you’re saying that suggests that you haven’t used StackOverflow much or you haven’t paid any attention while using it. Or you don’t know enough to realise how many answers are wrong. –  Timwi Sep 20 '10 at 4:27

In my view, this proposal is just what the doctor ordered for questions.

Question voting produces far more depressing muppetry that answer voting. Every muppet looks at the new questions and upvotes every bit of subjective, humorous, fluff that floats down the pipe. Far fewer of them look at all the answers.

Anyone can judge an answer if they know something about the topic. Question quality is another story, and it seems to me that only people with some demonstrated rep should be voting.

share|improve this answer
    
+1, but chances that something like this can actually happen are probably zero... :/ –  romkyns Sep 20 '10 at 10:14
    
You seem to be assuming that higher-rep users are more likely to downvote fluff. But I think you'd be disappointed. Don't you remember that rant about a week ago from Developer Art? There is actually a depressing number of 10k users and a good number of 20k and higher users who upvote that garbage and even vote to reopen when it gets closed. After all, how do you think they got that high? Some people made a lot of genuine contributions but many were/are just rep whores. –  Aarobot Sep 20 '10 at 21:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .