There seem to be two large possible sources of bias, that is, things which could cause different voting on identical answers:

  • we could be more likely to upvote answers for high-rep users, especially ones with familiar names

  • we could be more likely to upvote answers with high scores, and downvote answers with low (negative) scores (even assuming that all answers fit on the screen)

Of course, high-rep users are generally likely to write answers worth upvoting, and high-scoring answers are often worth a vote. But humans are not generally perfectly rational; I assume there is bias. Have there ever been attempts to quantify the effect here? (Please try to avoid answers based on opinion and anecdote.)

Edit: I tried to write this as neutrally as possible. I am not arguing that Stack Exchange is fundamentally broken, or that you personally are a biased voter and therefore a bad person. I am not complaining about any perceived unfair voting. I think high-reputation users deserve their reputation. I like these sites. I'm just curious, within this context, how large of an effect cognitive biases have.

Another edit: I'm curious about the downvoting. I know it's different on meta, and I'm not offended, but since the core idea of the question is "is there any data about this effect" I'm wondering how people disagree - simply because it's hard to measure, or you believe that it would be bad for us to even know the answer? I'm happy to try to improve the question.

  • 5
    I'm sure this is a duplicate... anyone know offhand what a good candidate is? Dec 8, 2011 at 22:33
  • @TheUnhandledException: I was sure too, and I couldn't find one quickly. Hopefully someone else is better at searching than me.
    – Cascabel
    Dec 8, 2011 at 22:34
  • 2
  • Basic cognitive biases lead to these behaviors. Unfortunately you're not going to remove them so long as the names and vote counts remain public, and removing those would yield more damage than these biases.
    – Ben Brocka
    Dec 8, 2011 at 22:36
  • 2
    @BenBrocka: I know why these things happen, and I'm not trying to fix it; I'm asking if we know how bad it is, and attempted to phrase the question in as neutral a way as possible.
    – Cascabel
    Dec 8, 2011 at 22:37
  • 3
    This answer (not mine) is the kind of experiment I'd like to see run.
    – ChrisF Mod
    Dec 8, 2011 at 22:38
  • @ChrisF: Yes, I was envisioning the same sort of thing, except say, taking .1% of the viewers of a given question, and serving them a page with modified votes (or usernames/avatars, or reputations).
    – Cascabel
    Dec 8, 2011 at 22:41
  • 9
    "Please try to avoid answers based on opinion and anecdote." Given that no one but the team has access to attributed voting histories except their own there are very nearly no other answers available. Dec 8, 2011 at 22:41
  • 1
    @ChrisF while A/B tests are fun the answer seems fairly obvious, and we have no reasonable action to take regardless of the outcome of the test. The two solutions to the two potential problems (A: Users are more likely to upvote the first question they read, B: Users are more likely to upvote the question of high rep users) would be to remove the sort-by-upvote and remove the rep-count, which IMO are both opposed to the intent of the site's mission of promoting authoritative answers.
    – Ben Brocka
    Dec 8, 2011 at 22:42
  • @BenBrocka - true, but unless you know if reputation (or indeed identity) affects voting behaviour and by how much you can't make a proper decision.
    – ChrisF Mod
    Dec 8, 2011 at 22:45
  • @dmckee: Well, maybe someone extremely enterprising out there has been scraping pages and doing their own study. Failing that, I'm sure the team has considered it, and figured perhaps might be willing to at least say "we don't think it's a large enough problem to worry about", and even data about size of bias in similar situations could be informative.
    – Cascabel
    Dec 8, 2011 at 22:46
  • Sure there is bias.
    – Pekka
    Dec 8, 2011 at 22:50
  • 2
    Jefromi - but how could an enterprising screen scraper find out any more about this than we can with access to the data explorer? Voting data isn't available; hence we can't know for sure how much bias there is. Hell, we couldn't even know if we had the voting data. Maybe high-rep users get so many votes solely because their answers are so great? (I don't believe that, but it's a possibility.) The only way to find out would be an experiment like @Chris pointed to
    – Pekka
    Dec 8, 2011 at 22:53
  • 2
    It is worth examining the discussions around Is there an actual “pity” or “sympathy” upvote problem? to get an idea how hard and contentious it can be to work this stuff out. Dec 9, 2011 at 0:09
  • 2
    There's a very detailed analysis about this topic on the Cross Validated (statistics StackExchange) blog that may interest you as well. Dec 9, 2011 at 19:13

4 Answers 4


Cross Validated (the statistics StackExchange site) has a great blog post on this topic:

Does Jon Skeet have mental powers that make us upvote his answers? (The effect of reputation on upvotes).

There is a lot of detail in that post (much of which I do not fully understand), but the tl;dr version is this

But it appears on average these high rep users always had a high score per answer, even before they gathered a high reputation.

As that guy with the palindromic name answered, there is always human, mental / emotional bias that goes into voting (probably unconcscious in many cases); but, according to the numbers, it is not statistically significant.

  • 1
    +1: I much appreciate the link to the analysis, though I'm not sure I agree with the conclusions. (Score per view, or per post age, might show very different results.)
    – Cascabel
    Dec 9, 2011 at 19:52
  • I just figured out another way to test this. Have Jon Skeet open another account, lets call it Teeks Noj, and see if simultaneously answer a question with both accounts, randomly deciding which one gets momentarily posted first. Also keep the wording the same. Then see which get upvoted more.
    – puk
    Dec 9, 2011 at 20:01
  • @puk That would just result in Jon Skeet getting suspended for using sock-puppet accounts ;-) And then where would we be! Dec 9, 2011 at 20:57
  • @jadarnel27 Jon Skeet is just as likely to be suspended as the president is to get a ticket for J-walking
    – puk
    Dec 9, 2011 at 21:25

Okay, here's a plot. I am quite aware that this is not a perfect measurement, but it is so strikingly different from the plots linked to in jadarnel27's answer that I feel it is worth seeing.

I took Jon Skeet's answers, calculated score per question view, and averaged over all posts from each day to reduce scatter.


So making the reasonable assumption that Jon Skeet mostly gets upvotes, then we can conclude that people have been far more likely to upvote Jon Skeet's newer answers than his older ones. I don't want to leap to any conclusions about how much of this effect is reputation-based - I'm sure his post quality and speed have improved too - but we certainly can't say that the data demonstrates that this is a statistically insignificant effect, as claimed on Cross Validated. (Side note: that's quite an upturn earlier this year.)

For good measure, here's a similar plot, with score per post age in days instead:

enter image description here

  • 2
    Interesting data; but I don't think it's meaningful without some solid stats showing the overall data, too. That is: How do we know that people in general aren't voting more often per post view/post age today than two years ago? Dec 9, 2011 at 20:47
  • @AndrewBarber: +1, yes, that's one of the limitations. I posted this with two primary goals: demonstrating that the Cross Validated analysis wasn't really measuring the right thing, and providing a bar for further answers to hopefully exceed. I've never used the data explorer before, and this took very little time. I'm sure someone with more spare time and more experience could do better, and inform us all.
    – Cascabel
    Dec 9, 2011 at 20:52
  • 1
    I would be interested to see the same graphs for my answers: I answer posts which are usually more than 10 minutes old, on less popular tags, for users who generally don't even have enough rep to vote(!).
    – VonC
    Dec 10, 2011 at 7:34
  • 1
    Jon is, well, Jon. This is a nice approach, but it might be more convincing if you did it for users of, say, 50k (maybe a subset as that will be a lot of data). Dec 10, 2011 at 16:36
  • is it possible to regenerate those plots today in 2019?
    – KolA
    Aug 14, 2019 at 6:04

Personally, I don't vote for people based on their reputation. However, I have found that I am more inclined to upvote an answer that has a dozen or more upvotes than one with zero or one. This might mean that it is a great question. It might also mean that it was the first question =P

A good test would be to go through a couple dozen answers and see how well votes correlate with post order.



There is a lot of bias.

People assume that since I have almost 30,000 reputation on Stack Overflow that I must know what I am doing and they upvote my answers and ask me questions in chat.

When I do get stuff wrong, they say things like:

How did you get all that rep if you know nothing?

Reputation has nothing to do with a user's knowledge of any subject, it is just how much time they have used this (or any other) site, or their ability to google faster than other users.

Reputation != knowledge;

In conclusion:

Yes, voting is biased. But it is not always the best thing to base votes on.

  • 1
    I have >30k rep on SO, and I'm sure there's bias in upvoting my answers. But I have no idea how much, and that's the point of the question. An anecdotal "a lot", while probably true, isn't terribly informative.
    – Cascabel
    Dec 9, 2011 at 19:45
  • Rep doesn't equal knowledge but it typically correlates with one's ability to share useful knowledge and it is inherently a measure of authority.
    – Ben Brocka
    Dec 9, 2011 at 19:46
  • I sometimes wonder whether people are just geniuses or whether they google the stuff. For example, I'll ask which Michelin tire I should use on my vespa, and then 30 seconds later someone has a detailed response describing the chemical composition of Michelin tire TX-3450 right down to the string theory, and why tire TX-3550 is actually not optimal.
    – puk
    Dec 9, 2011 at 19:48
  • 4
    Funny, when I see similar/identical answers from both high and low rep users, I usually +1 the "little guy" Dec 9, 2011 at 20:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .