The problem with reputation: does high reputation attract too many upvotes? [duplicate]

Question

It's not that I feel bitter when somebody else's answer is upvoted above my own, but it's when a user with a high rep answers a question and their answer seems to attract an immediate barrage of upvotes. Now I'm not saying that these votes are always unwarranted, but by simply observing this behavior, I think the system seems flawed.

The concept of reputation (or at least one of the concepts) is to give users a sense of trust in the reputed user's answers. But this should be trust, not blind faith. I know that this problem doesn't exist for every question asked on SO, but it's definitely present.

I'm sure this is a contentious question, as it might seem overly critical, but does anybody else agree with me and, if so, does anybody have any thoughts on how this problem could be alleviated? It's a great shame when a new or less active user's answer is stifled by the reputation-rich user's answer.

Answer 1

I operated a second account for some time because I was curious to find out whether low-reputation users are really treated much differently than high-reputation users. (Don't worry, the sock puppet never even glanced at anything contributed by my main account.)

My - of course subjective - impressions from that were:

• low-rep users get more scrutiny on their contributions. That scrutiny isn't always there for high-rep users - I have seen instances where blatantly incorrect contributions (either by me or other high-rep users) were blindly upvoted by the community.

• if there are two identical contributions from a new and a veteran user, that of the veteran user is more likely to get upvoted.

However, I also found that

• if you make mistakes, they usually get pointed out. Whether you're a high or a low reputation user doesn't matter. The only difference is that with a high rep user, people may check twice before they say something.

• If you provide good content, you will gain upvotes, no matter how much reputation you have. Apart from the fact that high-rep users are more trusted, and viewed somewhat less critically, by the community, there is no discrimination against lower-reputation users.

I dare say that overall, the voting system works fairly. If you are competent and able to express your knowledge in an understandable way, there are fine chances of succeeding on Stack Overflow, no matter how big the "establishment" of veteran high-rep users is.

Answer 2

I didn't say I upvoted them, I just said that I assumed they were correct. :) – Robert Harvey Jan 21 at 23:28

There is a difference in trust in an answer (because you have the knowledge to say something about it), and trust in an answer (because it is explained well, and the author is trustworthy). The StackExchange reputation mechanism does not distinguish between the two, and therefore it cannot be told what prevails and how a certain reputation is constituted. I do not think it is a big problem, but it is quite interesting what happens when for instance a user does not see the reputation of a person (who gives an answer) until he votes for an answer. That is one way to test it.

Another way to test this is by sending 3 groups of 50 persons an email with a question and two answers. For all groups, the question and answers are the same. Each person must choose the better answer.

• Group 1 gets a question and two answers. The author of answer A has a high (artificial) reputation. The author of answer B a low reputation.
• Group 2 idem, but other way around.
• Group C idem, but without the reputation of the authors included.

The persons have to be familiar with the reputation system, otherwise the reputation does not make sense to them. There are different ways to extend such an experiment: showing/not showing the votes for an answer, looking at the reputation of the persons and see if there are correlations between own reputation and trust in another reputation, etc.

I have also thought of another, similar approach to testing the influence of accumulated reputation and the trust it seems to convey. Show a video to the 3 groups and ask them to rate the video. The first group is not shown a rating, the second group is shown an average high rating of the video, and the third group is shown an average low rating in the video. What you try to decipher here, is how much an existing reputation/rating influences choice, trust, and perceived value. A video is not the same as a person, clearly, but I think the same internal mechanics play a role here (I’m not a psychologist.. anyone?).

The underlying assumption is of course that people are guided by other people, preferably exemplary people (or icons), and that the trust in SO-metrics allow for ‘not having to think myself’, hence consuming less energy and is therefore a preferred strategy for many people in such communities. There is of course nothing wrong with this, but it of course puts some requirements on the reputation system: the reputation people trust in must not be able to predict the behaviour of an reputed agent, at least when the agent has a high reputation. It means: I value my high reputation > I have been willing, and will be willing to put effort in finding answers that are valuable > therefore my intention is right. Even though intention does not equal quality of the answer, it comes quite close, because wrong/bad answers will degrade your reputation.

Answer 3

Does High Reputation Attract Too Many ‘Up-Votes’?

Correlation does not imply causation.

I think you are simply looking at things in reverse:

The correct question is

How did they get such a high rep in the first place?

You'll find that those with high rep answer quickly, legibly, understand the problem, explain themselves well, and in general provide very, very good answers.

So while you might think that two answers are the same from your perspective, the one with a slightly more accessible phrasing has an immediate edge over the one that while technically correct isn't worded as well.

But if you provide some examples of answers where the reputation was the only difference, then we might be able to quantify the effect better. In the past (this is not the first time this question has come up) most examples could readily be torn apart and shown that the votes really did fall to the more deserving answers, apart from the person posting.

Answer 4

Maybe I'm just a nerd, but the title of this question immediately brought to mind Gravity Wells. It seems almost inevitable that if you get a lot of reputation, you will get even more upvotes. And those upvotes will increase with the increase of reputation.

That being said, we don't operate the same way the Universe does. Some users will downvote reputable users out of envy, or spite. So while I think there is a valid point in suggesting users upvote other users because of their impressive reputation, I think it's equally valid to suggest that those with large reputation experience the exact opposite too.

Jon Skeet, personal testimonies?

Answer 5

I've experienced this a few times. One example that comes to mind is: Message queue system

Wyatt, Joel and I say essentially the same thing. In fact Joel and my post are nearly identical but Joel has +13/-0 while I have +4/-1.

Why is Joel's answer 10 (9) points more valuable than mine?

Why is Joel's answer 6 points more valuable than Wyatt's when he provides several options?

Answer 6

Maybe there's another way to look at this. Maybe the "problem" isn't the upvoting itself, maybe it's the idea that all upvotes are identical.

If reputation is a measure of the community's trust in someone, then the value of an endorsement of an answer should, logically, scale with reputation. Or to put it more simply, Jon Skeet upvoting someone has more meaning than if I do.

To take this to a more fully-fleshed system, you could argue that an upvote from someone with approximately the same rep as you is worth 10 points, as it is today. An upvote from someone with 1 rep might be worth a single point, while a Jon Skeet upvote could be worth significantly more (20? 50? 100? I hear that an upvote from Jon Skeet actually changes the correct answer to match what was posted...)

This may not change the actual upvoting, but may change the impact of it - presumably the drive-bys would be from more casual people, and so would have little impact on a reasonably-repped user. However, even a single upvote from a well-respected user could have significant impact.

Similarly, upvoting something is an endorsement of the answer. Perhaps future votes on that answer should reflect on the rep of the up/down-voter? So if you upvote something that then proceeds to get 100 downvotes, including the Mighty Downvote of Jon Skeet, then that's probably a sign of a lack of judgement on your part, and a rep hit may be appropriate. This may also encourage people to annotate answers in comments rather than writing a new, but basically identical answer that just modifies one small bit. This idea is (obviously) ripe with abuse potential, and would need to be thought out more fully.

Answer 7

First of all I have seen many many cases where a low reputation user got many upvotes for his great answer and often in the same question where the high rep users are sitting in the dust.

Secondly I kinda agree, but in a different sense. What seems to happen is when an answer gets 1-2 upvotes an is at the top of the list. The rest of the upvotes for that question seem to go for that answer in blind faith.

I have seen of course many many cases where a question has many upvoted answers and a late well done answer comes along and blows the others away.

BTW I did a quick search for questions where this occurs (low rep users winning with good answers) And I found these in under about 2 minutes of searching.

551289 / 918865 / 1053705 / 1048643 / 934937

Answer 8

I wonder what the ratio of such occasions are? The problem is, that there is no absolute 'oracle' person who could vote the good and medium and bad answers. Everything is relativized to the community, but the built in mechanisms to 'guide' the users to the optimum behavior seems to be inadequate. The SO site is a great place for socio-scientific research though.

There are tons of options available: make a bit harder to vote (e.g. move the vote option to the bottom-right of the answer), use floating point value between (-1 and 1) for voting. Use weighting system to compensate for high-rep/low-rep answers, randomize answer listing by default, move the accepted answer away from the top, etc.

With proper scientific analysis, things could be improved.

Answer 9

I think the problem is that people are too ready to upvote - full stop. An experienced SO user will typically get a well-presented (though maybe not technically correct) answer in before a less experienced user, and despite the proprietor's "coddle the newbs" attitude, these will tend to get the first few drive-by upvotes, which then escalate. I'm tempted to suggest a penalty for upvoting - maybe 1 point to upvote versus 5 to downvote?

Answer 10

I'd be interested in the following aspect of this question: does a high-reputation user attract traffic to a given question. I've seen questions of low importance with correct low-rep answers and hardly a vote at all. And I've seen similar questions with a correct high-rep answer and votes pouring in seconds afterwards. So it looks to me as if the fact that there was a high-rep answer attracted visitors and votes to the question. I wonder:

Do people prefer to read questions which had recent activity by a high-rep user?

If so, how would they know? One way would be looking at the page of recent answers in the user profile of e.g. Jon Skeet. If there are actually people who do so, it should be possible to obtain that information from the server logs: see which users followed a link from that page to a question and afterwards voted on one or more answers of that question.

Another way would be the attribution of activity in the listing of questions. I consider this the more likely route for users to take, and also harder to track. It would be interesting to omit that information for a day and see how that impacts votes for high-rep users, but I'm not sure you're willing to tamper with the system in this way just for the sake of gaining more insight here.

One mor way to tackle this might be some correlation between visits, votes and reputation. If people do not care about rep when reading questions (i.e. “No” to my statement above) but do prefer to up-vote high-rep answers (i.e. “Yes” to the statement from the question title), then the ratio of votes to visits should strongly correspond to the reputation. If, on the other hand, people were more interested in questions with high-rep activity, but still casting their votes on merit without considering the rep, then the effect should be far less dramatic, and only due to the better quality of high-rep answers. Not sure what numbers I'd expect for either case, though, so I'm not really happy with this approach.

Answer 11

There is a combination of factors here.

Remember, high rep users had to start at 1 like everyone else. They got there through consistently providing answers that are timely, accurate, well-written, and with follow-up. All five factors (consistency, speed, accuracy, prose, follow-up) are important. There are a lot of users that are good at two or three or even four of those, but I think you'll find that most first-pagers tend to do all five.

Accumulating a high rep makes surprising little difference in this system. You still post lots of answers, many of which are in early and of sufficiently high quality to earn a few upvotes regardless of who wrote them or when they arrived. Whenever this happens you have both a time and votes advantage, so no matter what sort option is used people see a good answer at the top of the list with already a few votes.

In this circumstance I freely acknowledge a "piling-on" effect, but I want to point two things:

1. It's not necessarily bad any more. It's a good thing when one answer stands out from the crowd in terms of communicating to the asker and future readers what the solution to that problem is, and bad when you have several answers suggesting different approaches that all have about the same score. This is especially important when you have questions like "Should I do A or B?" and correct answer is "Neither, do C" or "How do I do A?" and the correct answer is "A is evil, never do that. Do B instead." Such answers need the credibility earned from a lot of votes. Since users have limited votes per day, it's common to only vote for one answer of a particular type, and so going with the most popular here is not a bad thing to do. That said, I consciously try to avoid voting for answers that seem to have attracted this effect.
2. Being a high rep user has little to do with it. It's still about getting a good answer in fast. You may see high rep users benefit from this more often, but as I mentioned earlier it's mainly because they're good at those points. I've had plenty of answers ignored that I thought were as good or better because I was a little too slow on the keyboard. This is okay; one of Stack Overflow's best features is the speed with which you can normally attract a response to a question. It happens enough that I can now often recognize that situation and just delete mine in favor of the earlier response. There is a certain trust factor, though. High rep users might benefit from a few extra votes for their answers now and then when they've earned the trust of other viewers to a question. However, at such times this is both 1) trust that is first earned and 2) consistent with the way reputation is designed for Stack Overflow, that is, reputation is intended to gauge trust.

One final point is that for the really high rep users — those in the first column of the first page — high votes to an answer don't usually translate into high rep for answer. The user has either already hit the rep cap for the day or would soon anyway, even if the answer earned no votes at all. For I time I was in that group, and it was not uncommon to get enough votes to end that day at 400 or more if there had been no cap in place, even if my actual final score for the day was somewhere between 198 and 230.

Answer 12

I don't have a solid proof that this happens on SO regularly (but my experience tells me it's true); but I most certainly noticed this pattern on other SE sites.

On SFF (where I'm one of top users) there were definitely times when I noticed that my answer of OK quality got more upvotes than a competing - and much better - answer from a lower rep user. Many times the answers were posted in roughly same time, so that's as clean experiment as possible.

In some cases, I had to openly post a comment to question poster requesting that they accept a better answer and not mine (to their credit, 100% of time it happened, the poster changed their accept after my recommendation).

P.S. As far as SO, I am aware of some single examples, even if I can't prove a trend that I suspect is true:

• Alan Kay got gazillion upvotes for basically posting an answer that amounted to "I don't remember".

• Joel's Turtle LOGO question - leaving aside the quality and desirability of it, its votes were CLEALY super-inflated considering the low overall popularity of LOGO tag

Answer 13

I don't look at rep at all when I upvote, but I do confess that if Jon Skeet or Marc Gravell (really, anyone in the top ten) answer the question, I generally assume that their answer is correct without checking.

That said, the idea that people are upvoting high-rep users because they have high rep is highly speculative. Do you have proof that this is occurring? Maybe the high-rep users have high rep because they give better answers. I have nearly 30,000 rep on SO, and I can tell you that I'm pretty sure that I'm not favored at all when it comes to voting.

What is favored are the kinds of popular questions that get lots of views, where the answers are easily understandable. But I think that kind of low-hanging fruit is getting harder to find, what with duplicates, strict Community-Wiki policy and so forth.

Answer 14

In my experience, high reputation doesn't particularly attract upvotes. High reputation means lots of posts, not lots of votes per post.

You can look at some numbers on the data explorer. Let's look in particular at List of top users ranked by highest "Average Rep earned per Answer" and Top users by average answer score. The people with the highest-rated answers on average are not the people with the most reputation. These queries are not perfect (for example they use rep now, not rep when each vote was cast), but I think they do show that there isn't a strong bias towards upvoting high-reputation answers.

So the phenomenon you describe is nonexistent or marginal.

By the way, do not mistake better-worded answers with the same technical solution with identical answers. (I'm not saying you're making this mistake, but I have observed it; Cory Charlton is definitely making it.) Maybe the answerers who get more upvotes than you are equally good problem solvers but better writers. Or maybe they were just faster (especially in high-volume tags on Stack Overflow, speed is a big factor in getting upvotes).