Doesn't science have a better reputation system than Stack Overflow?

Question

In science, peer review works a bit like upvotes: person A says that "yes, I think this work doesn't have any major flaws". Over time people who have published lots of work that has got such approval accrue reputation.

So far this is reminiscent of Stack Overflow.

But in science, if someone with a lot of reputation says that a certain work has no flaws, this has a lot of weight, while a crackpot with zero reputation saying the same thing contributes pretty much nothing. Similarly, if a reputable person says "this unsolved question is a good one to answer", that has a lot more weight than if, say, I were to say that.

This seems to be a very sane and sensible system.

Why doesn't Stack Overflow work the same? Do you reckon it wouldn't work for Stack Overflow if something like that were attempted? Why not?

Answer 1

As someone who has dealt with both the peer review process of traditional science and the Stack Overflow reputation system, I can tell you that they each have their flaws.

Scientific peer review is not a weighted voting system open to anyone, like you state. It is a system where only those who have earned a certain level of reputation can vote yes or no on whether a paper should be published. Once you clear that threshold, your vote as a reviewer counts as much as someone with more reputation. It is as if SO only allowed users with 3000+ reputation to vote.

As you would expect, this encourages a "good old boy" network to form. It is ridiculously hard for new investigators to get started in a scientific discipline. For example, the current average age for first-time NIH grant awardees is 43. Even if you had more of a sliding-scale reputation-based voting system, this would still be a problem.

SO has a stated goal of being welcoming to newcomers, so this approach doesn't fit with that philosophy. However, SO's current voting system does allow for popularity contests, where easily understandable or more visible questions and answers get voted above more technical and / or useful ones. Overall, though, the most useful answers seem to rise to the top over time.

The biggest problem facing SO at the moment is not a lack of quality answers, it is an increasing number of garbage questions. The reputation system does little to prevent those (aside from the negative-vote question slowdown now in place).

Answer 2

While SOs voting and reputation system certainly isn't flawless, neither is the peer review system that's common in science.
Highly reputable people can be wrong too, especially if new developments are concerned that aim to overthrow some major established view.

Additionally there are many different areas of expertise treated at SO - someone who has gained much rep in one area isn't automatically highly knowledgeable in other areas.

See also these questions for similar discussions:

• Applying PageRank-like algorithm to Stack Overflow votes
• Indicate How Trustworthy the UpVotes Are
• Weighted Down-Voting based on Reputation

Grabbing some results from those:

• "Reputation != expertise. It is mostly an indication of how much time you spent on SO."
• "Reputation does not mean you're an expert, because reputation can be earned by asking masses of low value questions."
• "Reputation is too broad a measurement; it doesn't indicate that a particular voter is knowledgeable in that particular subject."

In particular devinbs answers here addresses the flaws with weighted voting well.

Answer 3

Let's see...

• Each sub discipline of science is a small world. It really is a reputation economy where "reputation" takes the dictionary meaning. Stack Overflow has tens of thousands of "active" users (page 100 of the users shows nearly 3000 rep at this point and page 1000 shows around 140 rep).

• Science works mostly on new problems. Confirmation is an important part of the process, but you'll never achieve fame that way. On the other hand SOFUE mostly works on old problems and what is valued is timing, clarity, completeness, and the degree to which the question answers the problem.

• The reputation system on SOFUE is not only (or perhaps even primarily) about how well people think of you: it is about measuring contribution for the purposes of empowering you in the maintenance of the sites. Peer review does not generate a quantitative or centralized understanding of "reputation"; the result is a distributed and vague value which requires endless committee meeting and water cool conversations to translate into control.

...and there a probably other differences.

Answer 4

In my view, this proposal is just what the doctor ordered for questions.

Question voting produces far more depressing muppetry that answer voting. Every muppet looks at the new questions and upvotes every bit of subjective, humorous, fluff that floats down the pipe. Far fewer of them look at all the answers.

Anyone can judge an answer if they know something about the topic. Question quality is another story, and it seems to me that only people with some demonstrated rep should be voting.