One of the weaknesses of the current reputation points system is that is it heavily biased towards people participating in "hot" topics.

E.g. a super-expert on "Turkey" language that has 10 users who are interested in the language on SO and who answered EVERY one of 10 questions asked about "Turkey" with everyone up-voting it, would only get 1000 reputation, despite being an acknowledged guru for that tag (e.g. 100% view/up-vote rating, and 100% accept his answers rating).

Whereas everyone who spent >5 minutes on SO can point to numerous examples of people who earn much higher rep by virtue of not bothering to Google for a bunch of simple newbie answers and get up-voted for a bunch of truly irrelevant questions (this post is not meant as a whine about the latter pattern, as there are many such whines already on Meta).

What I would propose to combat such bias would be an ability to view - either per-tag or globally - the user's reputation normalized by the amount of views for the question that the rep was gained on.

E.g., if you answer 100 Java questions with so-so answers just good enough to get voted +1 but aren't even close to being the best answers, your normalized Java score would not be the same 1000 of the above-mentioned "Turkey" language guru, but some % value much lower.

I'm uncertain of the exact formula to use, but perhaps dividing the rep for a question by either linear or logarithm of the number of views would do that.

I would like to know whether this is considered a good idea - either from site's usability standpoint, or from difficulty/cost of implementation standpoint, for either per-tag or global numbers.

  • 1
    Nobody's stopping you to make a Greasemonkey script to do this ;-) – Ivo Flipse Sep 21 '09 at 20:53
  • I asked a similar question: meta.stackexchange.com/questions/4460 – Rich Seller Sep 21 '09 at 20:55
  • Both these comments refer to the info from standpoint of a single user computing it for itself. I was referring to displaying this info by SO for ALL the users - automatically. Not as a special action you need to do per user. So if this was downvoted as a "duplicate", please kindly remove that downvote as it is not. – DVK Sep 21 '09 at 21:22

No I don't think so. And this is why.

  • A question tag changes from edit to edit. So normalized rep would fluctuate quickly
  • A question within a popular tag could be very difficult to answer
  • Same in reverse, a question in an unpopular tag could be very simply answered
  • Point #1 well taken. I was assuming that the main tags at this point are somewhat stable - could be wrong. I agree with validity of your other 2 points as examples but for large datasets they should average out – DVK Sep 21 '09 at 21:24
  • The issue is that with tags that have few items, averaging out is something you cant easily do. – Ólafur Waage Sep 21 '09 at 21:30
  • As long as all the questions are tagged (or re-tagged - which I personally help a lot with and I know other people do) with at least 1 or 2 correct main tags, thos tags can be used for accurate averaging. – DVK Sep 22 '09 at 1:13

I do not think this is a good idea.

*Votes would have different worth based on your environment**. My votes in the C# are worth 'less' than my votes in the 'r' tag. Which is, of course, not true, I didn't do more thinking about my vote in the 'r' tag, so why should it be worth more?

Very Easily Gamed. All you have to do is answer some questions in an obscure tag, and then have a sock puppet upvote a few of them. Not all of them, or you'll run afoul the 'sketchy vote detector'. But it is possible.

# of views is not a fair indicator of quality. There are many reasons why a question or tag may have low views. It may be very ill-formed, and therefore gets low views. On the other hand, it may be very interesting, be very obscure and get a high number of views, but no one qualified to vote on the one answer. So, a thorough detailed answer might not end up with that many votes.

# of votes per view varies across tags. I'm sure the statistics will bear me out on this assumption, but I'll bet there are higher votes per view percentage on some of the smaller, more obscure languages. This is because most users are looking into C# (the most popular tag) and not voting very often. As they branch out into new tags, they'll tend to vote more in general.

And, as Olafur noted, tags change.

  • @devinb - (1). yes, easily gamed by a few people dedicated enough to bother with this. But so are other ways of learning points (e.g. posting a scattershot of simplistic "the answer is the first hit on Google" questions). And not many people would bother with such a weird gaming, I hope. – DVK Sep 21 '09 at 21:27
  • (2) I completely agree that # of view is not an indicator of quality. However, the RATIO of how many peolpe viewed vs. # of up-votes probably is a decent enough indicator. What my point was that this ratio is a BETTEr indicator than just the # of up-votes straight up. – DVK Sep 21 '09 at 21:29
  • (3) I'm too green on SO to know how to get such statistics, but I know my own usage patterns - I vote up any and all great answers, namely ones that either teach me something useful accurately or answer the question accurately (the two do not always match). But per-tag ratios are definitely the same. you could of course be right and i'm an aberration :) – DVK Sep 21 '09 at 21:31
  • @DVK - (1) The point there (I articulated badly) was not necessarily that gaming would take place, but that a few votes in a few questions (legitimate or not) would wildly skew the results. It means that an extra two votes in the 'r' could rocket you from the bottom (untrustworthy) to the top (knows more than the language architect), whereas it is impossible to do that in the larger tags. – devinb Sep 21 '09 at 21:48
  • (2) There could be a really badly written question with low views. So a users tries to answer it. Everyone thinks it is a great answer, even though it isn't. So it gets upvoted 10 times, because it seems right. On the other hand, in an 'r' related question, you won't find that kind of mad upvoting. The basic point here is that the number of views (or the ratio) does not necessarily indicate the intrinsic quality of any particular answer. – devinb Sep 21 '09 at 21:53
  • (3) Your usage pattern is excellent. However, by and large, most people upvote things that sound interesting or useful. Rather than upvoting things that they have verified as useful. Who has that kind of time? For this reason, people with high reputations have had incorrect answers marked as correct. Even the OP isn't checking the answers. The point being: people vote based on their whims. – devinb Sep 21 '09 at 21:55
  • (1) While it is plausible that some really sparse tag would be skewed by just two votes, I trust that enough people would take care to rectify the situation. That is the whole point of community-maintiained information after all :) I know I put MORE effort in cleaning up sparse tags I stumble upon specifically for that reason, since there are less likely cleaners and I as a cleaner can have more positive impact. – DVK Sep 22 '09 at 1:17
  • (2) I agree that a high ratio on specific question is not 100% correlation with quality. But on the average, the correlation should be OK – DVK Sep 22 '09 at 2:57

Is reputation normalized by question views a relevant piece of info?

No. The questions with more views will get a proportionate increase in votes - no more. The 'normalized' values should map very well with the regular values.

The only case where this may not be true are for very polarizing topics, and they account for only a very, very small (almost insignificant) portion of the system.

It's not worthwhile to tune the system for such a small problem.

Further, reputation is only meant to increase participation, and give a simple way to decide when to trust users with greater system access.

The underlying mechanism is set up such that no amount of normalization by any other available metric is going to tell you anything different than rep alone will.

To get something different you're going to have to fundamentally change rep.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .