There seem to be two large possible sources of bias, that is, things which could cause different voting on identical answers:

  • we could be more likely to upvote answers for high-rep users, especially ones with familiar names

  • we could be more likely to upvote answers with high scores, and downvote answers with low (negative) scores (even assuming that all answers fit on the screen)

Of course, high-rep users are generally likely to write answers worth upvoting, and high-scoring answers are often worth a vote. But humans are not generally perfectly rational; I assume there is bias. Have there ever been attempts to quantify the effect here? (Please try to avoid answers based on opinion and anecdote.)

Edit: I tried to write this as neutrally as possible. I am not arguing that Stack Exchange is fundamentally broken, or that you personally are a biased voter and therefore a bad person. I am not complaining about any perceived unfair voting. I think high-reputation users deserve their reputation. I like these sites. I'm just curious, within this context, how large of an effect cognitive biases have.

Another edit: I'm curious about the downvoting. I know it's different on meta, and I'm not offended, but since the core idea of the question is "is there any data about this effect" I'm wondering how people disagree - simply because it's hard to measure, or you believe that it would be bad for us to even know the answer? I'm happy to try to improve the question.

share|improve this question
5  
I'm sure this is a duplicate... anyone know offhand what a good candidate is? –  The Unhandled Exception Dec 8 '11 at 22:33
    
@TheUnhandledException: I was sure too, and I couldn't find one quickly. Hopefully someone else is better at searching than me. –  Jefromi Dec 8 '11 at 22:34
2  
    
Basic cognitive biases lead to these behaviors. Unfortunately you're not going to remove them so long as the names and vote counts remain public, and removing those would yield more damage than these biases. –  Ben Brocka Dec 8 '11 at 22:36
2  
@BenBrocka: I know why these things happen, and I'm not trying to fix it; I'm asking if we know how bad it is, and attempted to phrase the question in as neutral a way as possible. –  Jefromi Dec 8 '11 at 22:37
3  
This answer (not mine) is the kind of experiment I'd like to see run. –  ChrisF Dec 8 '11 at 22:38
    
@ChrisF: Yes, I was envisioning the same sort of thing, except say, taking .1% of the viewers of a given question, and serving them a page with modified votes (or usernames/avatars, or reputations). –  Jefromi Dec 8 '11 at 22:41
8  
"Please try to avoid answers based on opinion and anecdote." Given that no one but the team has access to attributed voting histories except their own there are very nearly no other answers available. –  dmckee Dec 8 '11 at 22:41
1  
@ChrisF while A/B tests are fun the answer seems fairly obvious, and we have no reasonable action to take regardless of the outcome of the test. The two solutions to the two potential problems (A: Users are more likely to upvote the first question they read, B: Users are more likely to upvote the question of high rep users) would be to remove the sort-by-upvote and remove the rep-count, which IMO are both opposed to the intent of the site's mission of promoting authoritative answers. –  Ben Brocka Dec 8 '11 at 22:42
    
@BenBrocka - true, but unless you know if reputation (or indeed identity) affects voting behaviour and by how much you can't make a proper decision. –  ChrisF Dec 8 '11 at 22:45
    
@dmckee: Well, maybe someone extremely enterprising out there has been scraping pages and doing their own study. Failing that, I'm sure the team has considered it, and figured perhaps might be willing to at least say "we don't think it's a large enough problem to worry about", and even data about size of bias in similar situations could be informative. –  Jefromi Dec 8 '11 at 22:46
    
Sure there is bias. –  Pëkka Dec 8 '11 at 22:50
2  
Jefromi - but how could an enterprising screen scraper find out any more about this than we can with access to the data explorer? Voting data isn't available; hence we can't know for sure how much bias there is. Hell, we couldn't even know if we had the voting data. Maybe high-rep users get so many votes solely because their answers are so great? (I don't believe that, but it's a possibility.) The only way to find out would be an experiment like @Chris pointed to –  Pëkka Dec 8 '11 at 22:53
2  
It is worth examining the discussions around Is there an actual “pity” or “sympathy” upvote problem? to get an idea how hard and contentious it can be to work this stuff out. –  dmckee Dec 9 '11 at 0:09
2  
There's a very detailed analysis about this topic on the Cross Validated (statistics StackExchange) blog that may interest you as well. –  jadarnel27 Dec 9 '11 at 19:13
show 11 more comments

4 Answers

Cross Validated (the statistics StackExchange site) has a great blog post on this topic:

Does Jon Skeet have mental powers that make us upvote his answers? (The effect of reputation on upvotes).

There is a lot of detail in that post (much of which I do not fully understand), but the tl;dr version is this

But it appears on average these high rep users always had a high score per answer, even before they gathered a high reputation.

As that guy with the palindromic name answered, there is always human, mental / emotional bias that goes into voting (probably unconcscious in many cases); but, according to the numbers, it is not statistically significant.

share|improve this answer
    
+1: I much appreciate the link to the analysis, though I'm not sure I agree with the conclusions. (Score per view, or per post age, might show very different results.) –  Jefromi Dec 9 '11 at 19:52
    
I just figured out another way to test this. Have Jon Skeet open another account, lets call it Teeks Noj, and see if simultaneously answer a question with both accounts, randomly deciding which one gets momentarily posted first. Also keep the wording the same. Then see which get upvoted more. –  puk Dec 9 '11 at 20:01
    
@puk That would just result in Jon Skeet getting suspended for using sock-puppet accounts ;-) And then where would we be! –  jadarnel27 Dec 9 '11 at 20:57
    
@jadarnel27 Jon Skeet is just as likely to be suspended as the president is to get a ticket for J-walking –  puk Dec 9 '11 at 21:25
add comment

Okay, here's a plot. I am quite aware that this is not a perfect measurement, but it is so strikingly different from the plots linked to in jadarnel27's answer that I feel it is worth seeing.

I took Jon Skeet's answers, calculated score per question view, and averaged over all posts from each day to reduce scatter.

jon-skeet-votes-per-view

So making the reasonable assumption that Jon Skeet mostly gets upvotes, then we can conclude that people have been far more likely to upvote Jon Skeet's newer answers than his older ones. I don't want to leap to any conclusions about how much of this effect is reputation-based - I'm sure his post quality and speed have improved too - but we certainly can't say that the data demonstrates that this is a statistically insignificant effect, as claimed on Cross Validated. (Side note: that's quite an upturn earlier this year.)

For good measure, here's a similar plot, with score per post age in days instead:

enter image description here

share|improve this answer
1  
Interesting data; but I don't think it's meaningful without some solid stats showing the overall data, too. That is: How do we know that people in general aren't voting more often per post view/post age today than two years ago? –  Andrew's a Unitato Dec 9 '11 at 20:47
    
@AndrewBarber: +1, yes, that's one of the limitations. I posted this with two primary goals: demonstrating that the Cross Validated analysis wasn't really measuring the right thing, and providing a bar for further answers to hopefully exceed. I've never used the data explorer before, and this took very little time. I'm sure someone with more spare time and more experience could do better, and inform us all. –  Jefromi Dec 9 '11 at 20:52
    
I would be interested to see the same graphs for my answers: I answer posts which are usually more than 10 minutes old, on less popular tags, for users who generally don't even have enough rep to vote(!). –  VonC Dec 10 '11 at 7:34
1  
Jon is, well, Jon. This is a nice approach, but it might be more convincing if you did it for users of, say, 50k (maybe a subset as that will be a lot of data). –  dmckee Dec 10 '11 at 16:36
add comment

A LOT

There is a lot of bias.

People assume that sinse I have almost 30K rep on http://stackoverflow.com that I must know what I am doing and upvote my answers and ask me questions in chat.

When I do get stuff wrong they say things like:

How did you get all that rep if you know nothing?

Rep has nothing to do with a user's knowledge of any subject, it is just how much time they have used this (or any other) site, or their ability to google faster than other users.

Rep !== knowledge;

In conclusion:

Yes voting is bias. But it is not always the best way to go.

share|improve this answer
    
I have >30k rep on SO, and I'm sure there's bias in upvoting my answers. But I have no idea how much, and that's the point of the question. An anecdotal "a lot", while probably true, isn't terribly informative. –  Jefromi Dec 9 '11 at 19:45
    
Rep doesn't equal knowledge but it typically correlates with one's ability to share useful knowledge and it is inherently a measure of authority. –  Ben Brocka Dec 9 '11 at 19:46
    
I sometimes wonder whether people are just geniuses or whether they google the stuff. For example, I'll ask which Michelin tire I should use on my vespa, and then 30 seconds later someone has a detailed response describing the chemical composition of Michelin tire TX-3450 right down to the string theory, and why tire TX-3550 is actually not optimal. –  puk Dec 9 '11 at 19:48
4  
Funny, when I see similar/identical answers from both high and low rep users, I usually +1 the "little guy" –  Adam Rackis Dec 9 '11 at 20:13
add comment

Personally, I don't vote for people based on their reputation. However, I have found that I am more inclined to upvote an answer that has a dozen or more upvotes than one with zero or one. This might mean that it is a great question. It might also mean that it was the first question =P

A good test would be to go through a couple dozen answers and see how well votes correlate with post order.

share|improve this answer
    
Instead of question, don't you mean answer? –  Daniel δ Dec 9 '11 at 19:27
    
@Danielδ yes, you are right. I edited the post to reflect this. –  puk Dec 9 '11 at 19:43
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .