Contribution quality indicator

Question

There are several significant benefits in being able to see the quality of contributions of a user to SO (for example, reading answers of a high quality contributor is great for learning; providing privileges to the site should ideally take contribution quality into account; etc.).

For a variety of reasons, reputation is a very noisy indicator of contribution quality. It's a much better indicator of the contribution quantity and of the topic popularity.

What measures might work better as an indicator of quality? I'm not suggesting that such metric should be added to SO; for now, I just wanted to explore what options are available.

For example, I was thinking about one indicator, call it "QualityRank" (QR), defined as follows:

1. Everyone starts with QR = 100
2. An author of an upvoted question / answer is rewarded by 0.1% / 1% of the upvoter's QR, rounded down to the nearest integer
3. An author of a downvoted question / answer is penalized by 0.5% / 5% of the downvoter's QR, rounded down to the nearest integer
4. Steps 2 and 3 are repeated again as many times as necessary, to reflect the updates in the voter QR.

Only a few iterations of steps 2-3 should suffice because most of the times voter QR changes have no impact (only when voter QR cross a whole 100, does it have any impact on the users whose posts they voted on). If there's any problem with convergence, the downvotes can be ignored, or some better heuristic could be used (convergence would greatly improve if QR can only go up).

In step 3, I set penalty from downvoting much higher than reward from upvoting because people are generally unwilling to downvote (out of common courtesy, out of sympathy, out of fear of being wrong, etc.), so each downvote carries a lot more information than an upvote.

Do you think this or a similar measure might be useful to judge the quality (rather than quantity or popularity) of contributions?

Answer 1

I think it's an interesting idea, although it's relatively complex, and I'm not it scales correctly, since it increments through fractions. That kind of metric can quickly become asymptotic, since it accelerates over time.

In the interests of discussion, I'll describe a couple of informal metrics that I use when evaluating other people's quality level.

1. Fraction of accepted answers - This gives me an idea of how well this person solves the actual question posed by someone else. It's a combined indicator of focus, quality and expertise, since average answers are much less likely to be accepted. To me someone with a high fraction is like the quiet guy in the meeting, who doesn't ever say anything, but when he does, everyone sits up and pays attention, because it's likely to really matter.
2. Relative badge fractions per unit rep - I've found that the fraction of gold and silver badges, normalised by a user's reputation, is a pretty good indicator of quality. Someone with 10K who got there through hundreds of scattershot questions is not going to have Enlightened, Guru, Good Question, etc. If you look at the top users on any Beta site, I'll bet you can distinguish between the quality of two users of similar rep, based on this metric alone.

Obviously these metrics have their limitations. Some users deliberately post additional answers to provide extra information, or another angle, even after another has already been accepted. This is positive, since it helps round out the question for future visitors. You won't spot them with metric 1. And sometimes you're just unlucky, and don't get accepted, regardless of how well you solve the problem. So there's some noise in the metric. And users that fire off lots and lots of answers have a different kind of value to the system too.

Spotting the quality people in low-traffic tags is a special problem. An idea based on yours might work. Another way might be to normalise a tag-specific number based on the tag volume (some function of number of questions or votes).

Let me also say that what a metric tells us, and whether it is useful or not, is one question. Whether it should be implemented for general consumption is quite another! Making metrics public adds all sorts of complications, because now you have to consider the effect of that metric on user behaviour, whether the metric can be gamed, and judge whether the evoked behaviour is what you want. For that reason I'd be very wary of adding any new metric or statistic to the system without careful thought.