Up/down votes or useful answers? Weighted score and "agreement index"!

Question

Primary tool used to quickly search for good answers (and often questions too) and corner brick of SO/SE is..voting. Other sites use similar method to highlight featured answers (for example Amazon).

Motivation

One point where I think this system lacks is to highlight answers with high number of downvotes. Let me use an example:

Answer              Votes            Score
             Total   Up   Down
1               10    6      4           2
2               10    2      0           2

They have same final score but their history is pretty different (and probably their quality too). Such controversial answer may be a result of raw voting ("is it +3? should be good, let me up vote!"), wrong down voting or anything else but for sure it's more controversial and reader should take much care (and this is should be highlighted).

Improvements

I can't suggest (and even imagine) to change this mechanism but a better index may be needed to highlight this kind of answers. For example Amazon style would be: 6 persons of 10 did find useful this answer.

A short text like "6/10" may be viable but a better index may be more useful. Let me call this index "agreement index", it should be calculated using the number of up/down votes. Let me use another example:

Vote   Voter's reputation
+1     1000
+1     2000
-1      100
-1      100

Score for this answer is 0 and unweighted index may be 50%.

I would even introduce another index with the score calculated according to reputation of each voter. For example weight of each vote in this index should be calculated using rep of voter. In this case we may have index of agreement 50% and weighted score of 0.875 (1000/3200 + 2000/3200 - 100/3200 - 100/3200) that may be rounded to 1 (see this meta question about this topic). I wouldn't suggest to replace score with a weighted score but to integrate it somehow.

What I suggest is (finally!) to add two simple numbers near the score, like this:

Advantages

Weighted score: browsing answers you can quickly judge them with a weighted score according to voters' reputation. If we judge answers (also) according to user's reputation then why votes shouldn't be judged in the same way? At least if you want to see it...you can. This may be especially helpful for OP and for someone else who lands on that page searching for the same question (very often answer with more up votes is not the best one from a technical point of view but the one more easy to implement or to understand).

Index of agreement: you can easily highlight controversial answers: to read them carefully and to contribute with your vote, to better read them before use, to highlight "anomalies" in the votes for the question.

Disadvatages

Index of agreement: it's easy to expand votes to see how many up and down votes there are so we essentially duplicate information (and it'll make screen more "noisy"). It's true but it's an important information I would highlight more.

Answer 1

One major flaw in your "weighted score" is you are equating expertise to reputation.

• Why should Jon Skeet's vote count more than mine or a new user.
• How do we know that a new user isn't an expert in the technology and his vote is actually meaningful
• Likewise, how do we know the high rep user is an expert in the technology. The high rep could just be because they are experts in in Stack Overflow.

Then factor in a misunderstanding of the question or an accidental vote. If a very high rep user like Jon or anyone into a 6 figure rep make a mistake in voting, the weighted score will need a lot of votes from the other side to counter that 1 misvote.

While this is not a big enough reason to reject it if it were the only negative, you do need to consider voting anonymity. Right now, it is extremely difficult to know who voted for you because there are millions of users who can vote on posts and every vote looks the same. However, now a weighted score might help someone think they know who voted on a post simply because they know a high rep user was active on a post and then the weighted score was skewed in 1 direction. It would be difficult to know for certain, but it would narrow down the field and might lead to more accusations and revenge downvoting.

As for your "index of agreement" idea, I can understand the motivation, especially for users who do not have enough rep to see the vote splits. But I don't necessarily think it offers any information that allowing all users to view vote counts or total votes doesn't offer and is less confusing.

What I mean by the confusion aspect is from the user experience, especially the new user and non-members who find a question via google. Unless you are a member and familiar with the community, "Index of Agreement = 50%" doesn't really explain what it is or what it means. But the definition of "10 votes" or "+5/-5 votes" is more apparent to everyone.