There are several significant benefits in being able to see the quality of contributions of a user to SO (for example, reading answers of a high quality contributor is great for learning; providing privileges to the site should ideally take contribution quality into account; etc.).
For a variety of reasons, reputation is a very noisy indicator of contribution quality. It's a much better indicator of the contribution quantity and of the topic popularity.
What measures might work better as an indicator of quality? I'm not suggesting that such metric should be added to SO; for now, I just wanted to explore what options are available.
For example, I was thinking about one indicator, call it "QualityRank" (QR), defined as follows:
- Everyone starts with QR = 100
- An author of an upvoted question / answer is rewarded by 0.1% / 1% of the upvoter's QR, rounded down to the nearest integer
- An author of a downvoted question / answer is penalized by 0.5% / 5% of the downvoter's QR, rounded down to the nearest integer
- Steps 2 and 3 are repeated again as many times as necessary, to reflect the updates in the voter QR.
Only a few iterations of steps 2-3 should suffice because most of the times voter QR changes have no impact (only when voter QR cross a whole 100, does it have any impact on the users whose posts they voted on). If there's any problem with convergence, the downvotes can be ignored, or some better heuristic could be used (convergence would greatly improve if QR can only go up).
In step 3, I set penalty from downvoting much higher than reward from upvoting because people are generally unwilling to downvote (out of common courtesy, out of sympathy, out of fear of being wrong, etc.), so each downvote carries a lot more information than an upvote.
Do you think this or a similar measure might be useful to judge the quality (rather than quantity or popularity) of contributions?