There are several significant benefits in being able to see the quality of contributions of a user to SO (for example, reading answers of a high quality contributor is great for learning; providing privileges to the site should ideally take contribution quality into account; etc.).

For a variety of reasons, reputation is a very noisy indicator of contribution quality. It's a much better indicator of the contribution quantity and of the topic popularity.

What measures might work better as an indicator of quality? I'm not suggesting that such metric should be added to SO; for now, I just wanted to explore what options are available.

For example, I was thinking about one indicator, call it "QualityRank" (QR), defined as follows:

  1. Everyone starts with QR = 100
  2. An author of an upvoted question / answer is rewarded by 0.1% / 1% of the upvoter's QR, rounded down to the nearest integer
  3. An author of a downvoted question / answer is penalized by 0.5% / 5% of the downvoter's QR, rounded down to the nearest integer
  4. Steps 2 and 3 are repeated again as many times as necessary, to reflect the updates in the voter QR.

Only a few iterations of steps 2-3 should suffice because most of the times voter QR changes have no impact (only when voter QR cross a whole 100, does it have any impact on the users whose posts they voted on). If there's any problem with convergence, the downvotes can be ignored, or some better heuristic could be used (convergence would greatly improve if QR can only go up).

In step 3, I set penalty from downvoting much higher than reward from upvoting because people are generally unwilling to downvote (out of common courtesy, out of sympathy, out of fear of being wrong, etc.), so each downvote carries a lot more information than an upvote.

Do you think this or a similar measure might be useful to judge the quality (rather than quantity or popularity) of contributions?

  • A related question asks whether it would be useful to identify high quality answers (it assumes if it's done, it would be by hand). – max Sep 26 '12 at 5:13
  • 6
    You want to add another metric that users can obsess over? – David Robinson Sep 26 '12 at 5:21
  • 1
    :) But seriously, I want to find out if a useful metric can be designed. If it can, it's far from certain that it should ever be introduced! – max Sep 26 '12 at 5:23
  • What evidence do you have to support your case that "many users reach 10k+ by asking a bunch of generic questions"? – jmort253 Sep 26 '12 at 5:35
  • @jmort253: I feel it's mean to name users who didn't do anything bad on purpose. If you like, you can look at my rep - it will get to 10k before long, with nearly no answers. If you feel my statement is unfounded, I would rather remove it. – max Sep 26 '12 at 5:36
  • @max - Might be a good idea just to avoid any offensive retribution or naysayers. If you focus on just how the metric benefits us, you'll likely have better response. Good luck! – jmort253 Sep 26 '12 at 5:37
  • @jmort253 - good advice, I'll remove that statement. – max Sep 26 '12 at 5:38
  • @YannisRizos: I thought I wouldn't ever be able to see who voted up or down, even through API/Data Explorer. If this data is exposed, and I missed it, I can certainly do it! – max Sep 26 '12 at 5:40
  • @max Ah, yes I missed that part, an author of a voted post gets a percentage of the voter's QR, sorry. Well, then your only option would be to convince someone with access to the database to test run your algorithm (and that's going to be tough). – yannis Sep 26 '12 at 5:46
  • I'm beginning to think that my approach won't work for a technical reason. If the first 100 points are given for free, the first 200-point users will be the authors of "popular" rather than "high quality" questions/answers. And they would be the ones whose opinion gets higher weight. And so on. Thus, my measure would simply measure the popularity of a post among authors of popular posts rather than among the general population. I don't know if this is any better at measuring "quality" than the current rep is. Still, I'd be interested in testing this out... – max Sep 27 '12 at 11:42

I think it's an interesting idea, although it's relatively complex, and I'm not it scales correctly, since it increments through fractions. That kind of metric can quickly become asymptotic, since it accelerates over time.

In the interests of discussion, I'll describe a couple of informal metrics that I use when evaluating other people's quality level.

  1. Fraction of accepted answers - This gives me an idea of how well this person solves the actual question posed by someone else. It's a combined indicator of focus, quality and expertise, since average answers are much less likely to be accepted. To me someone with a high fraction is like the quiet guy in the meeting, who doesn't ever say anything, but when he does, everyone sits up and pays attention, because it's likely to really matter.
  2. Relative badge fractions per unit rep - I've found that the fraction of gold and silver badges, normalised by a user's reputation, is a pretty good indicator of quality. Someone with 10K who got there through hundreds of scattershot questions is not going to have Enlightened, Guru, Good Question, etc. If you look at the top users on any Beta site, I'll bet you can distinguish between the quality of two users of similar rep, based on this metric alone.

Obviously these metrics have their limitations. Some users deliberately post additional answers to provide extra information, or another angle, even after another has already been accepted. This is positive, since it helps round out the question for future visitors. You won't spot them with metric 1. And sometimes you're just unlucky, and don't get accepted, regardless of how well you solve the problem. So there's some noise in the metric. And users that fire off lots and lots of answers have a different kind of value to the system too.

Spotting the quality people in low-traffic tags is a special problem. An idea based on yours might work. Another way might be to normalise a tag-specific number based on the tag volume (some function of number of questions or votes).

Let me also say that what a metric tells us, and whether it is useful or not, is one question. Whether it should be implemented for general consumption is quite another! Making metrics public adds all sorts of complications, because now you have to consider the effect of that metric on user behaviour, whether the metric can be gamed, and judge whether the evoked behaviour is what you want. For that reason I'd be very wary of adding any new metric or statistic to the system without careful thought.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .