36

Has there been any attempt to objectively estimate Stack Overflow users skill? If so, was any correlation between skill and reputation found?

To make the question more precise, let's assume skill to be defined as "ability to provide answers that end up on top".

Disclaimer: Self answered question. Just wanted to share my findings since I think it could be of interest to the community. I've tried hard not to formulate this Q/A as a plug for the stackapp. Let me know if I failed, and how to improve it. (I'm also genuinely interested in more answers than my own.)

Improve this question

1 Answer 1

Reset to default
38

Has there been any attempt to objectively estimate Stack Overflow users skill?

Yes, this has been done in a recently released StackApp called StackRating. Each question was seen as a "game", the responders were seen as "players" and the outcome of each game was determined by the votes. The players skill was then estimated using the Elo rating system.

In a nutshell it worked as follows: If you answer a question and get 20 upvotes, and I answer the same question and get 2 upvotes, then you have "won" and I have "lost". This will result in a positive rating update for you, and a negative rating update for me. The magnitude of the updates will depend on our current ratings; If you were higher rated than me, the result was expected, and our updates would be small. If on the other hand I had a higher rating than you, the result was unexpected and our updates would be large.

The key property of this approach to rating is that the ratings converge (as opposed to reputation which just grows and grows).

cnicutar's rating has a very characteristic graph which illustrates this:

A very characteristic graph

For the details of how the scoring was implemented and what K value was used, see the about page.

If so, was any correlation between skill and reputation found?

Well, high rep users tend to be skilled in the sense that they are good at writing answers that end up on top (big surprise). Whether this corresponds to being a good programmer is up for debate, but arguments can be made that this is indeed the case.

The opposite is not true though. A low rep user can be an unskilled user or a skilled user (shouldn't come as a surprise either).

Perhaps someone who is more experienced with statistics can come to a more interesting conclusion. Here's the reputation vs rating plot:

Reputation vs rating plot

Jon Skeet is the blue dot at the top, and Eric Lippert is the dot furthest to the right.

The full list of users can be found at http://stackrating.com.

Improve this answer
25
  • 14
    I love how that looks like a candle, fluttering in a mild breeze. That mild breeze, of course, being the constant stream of farting from new and incompetent users. :)
    – Lightness Races in Orbit
    May 3, 2015 at 16:00
  • Haha, well, new users (users with fewer than 100 answers) barely affect rating. (See how the K value is defined.)
    – aioobe
    May 3, 2015 at 16:03
  • 4
    Interesting. How do you count (if at all) questions with only one answer? Did you investigate whether not counting accepts changes things?
    – Mat
    May 3, 2015 at 16:25
  • 2
    Questions with only 1 answer are not considered, and no, an accept is just treated as one extra upvote.
    – aioobe
    May 3, 2015 at 16:32
  • 4
    It strikes me that someone who is the only person capable of answering some given question is likely very skilled! But I see why you can't include them.
    – Lightness Races in Orbit
    May 3, 2015 at 17:36
  • +1 on question and answer, but I don't like your results for me :-) I often add answers (that may be should be comments sometimes) that either add to an existing answer or only answer part of the question. These seem to often get large negative deltas. Of course, there's also the problem of working in the "niche" VB.NET.
    – Mark Hurd
    May 8, 2015 at 1:18
  • And some of my better answers are Necromancers :-(
    – Mark Hurd
    May 8, 2015 at 4:39
  • 1
    @MarkHurd, regarding your first comment: This is unfortunate, but there's obviously no way to tell the difference between an answer that adds useful and insightful additions to an otherwise complete and well written answer, and a stand-alone substandard answer. This is inherent to the voting system which both rating and reputation builds upon. As for your second comment, there's a huge back-story that hasn't actually been told yet... See my next comment.
    – aioobe
    May 8, 2015 at 5:30
  • 1
    The reason for not taking answers that were posted >90 days after the question into consideration when computing the rating is due to many things. One reason is that new technology may have emerged and newer questions have an unfair advantage (i.e. upvotes don't relate well to skill). The other reason is that it's not obvious what initial rating to use. If you post an answer today, to a question that was posted last year, then you should reasonably be competing with the rating you have today... However, your "todays rating" is not known when judging games that were posted a year ago...
    – aioobe
    May 8, 2015 at 5:35
  • 2
    ...since I can't solve all rating equations in one gigantic equation system. The games need to be linearized in time. And if one of your opponent on that question also posts a necromancer answer, then there is a cycle in order in which games must be judged. For this reason, the games are linearized in the order according to the post time of the question. This means that you're "changing history" by posting a necromancers answer. The 90 day cutoff limits the impact of this.
    – aioobe
    May 8, 2015 at 5:36
  • 1
    This is a very interesting analysis, and I accept the conclusion that many of my answers are highly voted up. But I'm not convinced that there is that much of a correlation between being able to convincingly say "a bowl of apples is not a bowl of fruit", and being able to actually program computers.
    – Eric Lippert
    May 15, 2015 at 16:00
  • 1
    I also note that in any analysis of "skill" where I am at the top and supercat is fourth from the bottom, something must be uncorrelated with reality. Supercat clearly has a deep knowledge of programming language design and semantics, and consistently gives thought-provoking comments and answers.
    – Eric Lippert
    May 15, 2015 at 16:26
  • Thanks for your comments. I feel like I could respond with an essay, but I'll try to keep it short. My desire and initial goal was to measure programming skill. I might have been able to pull that off if I had based the analysis on results from standardized programming tests or programming competition results. The Stack Overflow data reflects something else though. In my experience a good sense of didactics for example typically beats deep knowledge when it comes to upvotes, so if you ask me I'd say rating probably reflects something closer to pedagogical skill rather than...
    – aioobe
    May 15, 2015 at 22:59
  • ...programming skill. (The bowl of fruit answer is an example of pedagogical brilliance if you ask me.) That being said I do think being knowledgable in an area is a prerequisit for being a good teacher in the area. In other words, if you're a good programming teacher, chances are you're a good programmer. As I see it the converse is not necessarily true; Being a good programmer does not imply that you're a good teacher...
    – aioobe
    May 15, 2015 at 22:59
  • 4
    @aioobe: I think your conclusion -- that you have found a metric for pedagogy -- is pretty reasonable. Something just struck me though -- this may also be a metric for knowing how to "pick your battles". In chess tournaments you don't get much choice about who you play, but I do get choice about which questions I choose to answer and which I leave alone, and that is influenced by who has already left answers.
    – Eric Lippert
    May 17, 2015 at 13:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .