15

I've been a member of StackOverflow for almost 2 years. I only became active about a year ago.

Today my rep is 6382.

https://stackoverflow.com/users/1212341/dave-a

I earned that ranking because of my skills in JQuery, ASP.NET-MVC, etc. I have a very high accept rate among my answers too.

But while I seem to give reliable and useful answers (in my realm of expertise), I am a mediocre (or worse) reviewer. My audit history is poor. I've been banned multiple times for failing audits. I've improved by failing, but have also accepted that I'm not great and come to avoid it.

So I wonder: Does having a high rep prove that I am a good reviewer?

In my case, nothing about the way I earned my rep had anything to do with reviewing. I became a better answer poster from trial, error, and some critique. Hands on experience with my own projects helped.

I began to do review tasks after my rep was almost 4K, and I my audit history shows I was sloppy. I think I learned by doing and by failing. At this point I suspect I'm only OK as a reviewer. Far from reliable.

I would bet there are many users with 500 or below rep who would be far better reviewers than me. In fact, the only real test of a user's reviewing reliability is doing reviews (or audits).

I suspect that if users at the 200 level began doing reviews, and the community broadened their scope and entitlements / powers of reviewing proportionately to their audit scores, we would have more reliable reviewers than if we empowered users by their rep.

I may not be representative. I may be the sloppy outlier. But my experience with good developers and good administrators leads me to suspect there may be a negative correlation.

All I can say with certainty is that

  • answering questions and earning StackOverflow rep did NOT prove I was worthy of being a reviewer.

  • earning StackOverflow rep did not prepare me to do quality reviews.

  • being shamed and instructed by failed audits and bans helped me to learn.

  • a track record for good reviewing seems to be a more reliable measurement of how reliable a reviewer is.

Improve this question
7
  • 1
    The review queues are a dreaded place because of the robo-approvers who often have reputation in the lower ends (my personal experience) of the allowed spectrum (< 4k). Obviously the pool of low-rep users is bigger than the +50k one, but I think we can assume that the proportion between good reviewers and bad reviewers are more negative in lower reputation regions than higher ones. Therefore extending this privilege to 200 rep (which any person can get in just one day) would, if the trend can be continued, result in a worse review queue.
    – Jeroen Vannevel
    Feb 13, 2014 at 15:41
  • @JeroenVannevel, I agree with that statement. But I fear you may have missed the point in the forest of random example numbers thrown out. The question I pose is more like "if we allowed users to start reviewing at lower levels than currently set and enforced more frequent audits, or perhaps even had a test of 50 audit questions as a barrier to getting review privileges, would we find that we have more reliable and maybe even more enthusiastic reviewers?"
    – Dave Alperovich
    Feb 13, 2014 at 16:04
  • I'm NOT proposing specific levels (eg 200, 500) for opening enrollment. only posing philosophically that better reviewers may exist even at the lowest levels of imaginable rep. That arguably, at a certain point (maybe 500, 1K, 1.5K, etc), rep stop proving understanding of SO and track record of passing audits proves more about reliability to perform reviews.
    – Dave Alperovich
    Feb 13, 2014 at 16:08
  • high reputation is an indicator, but it's not in any way proof.
    – Sam I am says Reinstate Monica
    Feb 13, 2014 at 16:32
  • @SamIam, at what point does rep hit a point of diminishing return, or even a wall? Does audit history prove when rep goes beyond a certain score (500, 1K, 2K, 3 fish, blue fish)?
    – Dave Alperovich
    Feb 13, 2014 at 16:37
  • See also Introduce a better way to identify trusted users.
    – user163250
    Apr 2, 2014 at 10:27
  • Reputation is really not reliable. I have some reputation too and I'm a total freak. I just got one lucky answer.
    – Tomáš Zato
    Nov 24, 2014 at 18:35

6 Answers 6

Reset to default
22

I don't think the reputation requirement is because reputation proves that you'll be a good reviewer; rather, it's an indication of overall activity on the site, which suggests you are more likely to be a good reviewer.

But it's certainly not guaranteed, hence the audits and other systems you have encountered.

Kudos, by the way, for honestly admitting (to yourself; not us!) that reviews just aren't for you. Personally, I tend to steer clear of reviewing Tag Wiki edits, beyond really obvious things, because I just can't quite wrap my head around what's appropriate there.

Improve this answer
7
  • 1
    thanks. But I just wonder if there are low rep users who may or may not be technical heavy-weights but would have better review sense if given the chance.
    – Dave Alperovich
    Feb 12, 2014 at 20:17
  • 1
    +1 for admitting about tag wikis!
    – Dave Alperovich
    Feb 12, 2014 at 20:20
  • 1
    @DaveA Perhaps; but without some experience with the network itself, the chances are not high.
    – Andrew Barber
    Feb 12, 2014 at 20:20
  • 2
    agreed. I wonder if there are 1K rep users who would be better close and wiki-edit reviewers than 3K users. I suspect at some point rep hits a wall of proof and from there only audit history is important.
    – Dave Alperovich
    Feb 12, 2014 at 20:23
  • 2
    Oh, I'm positive there are 1k users who would be better reviewers than many 3K'ers. Or even many 5-digit users. But I suppose it's a matter of trying to find the right level with the best 'probabilities', @DaveA
    – Andrew Barber
    Feb 13, 2014 at 16:49
  • Maybe this is wishfull thinking, but if a 1K user took an test of audit questions (50, 100, who knows) and scored very high, they would "prove dedication". I imagine a user couldnt be allowed to retake the test for at least a month to minimize gaming. And questions would have to be protected from double checking their actual result. Though a "good score" on such a test would be far from conclusive, I suspect such a user would perform better than our current crop of reviewers. (esp if they were held to a higher standard and higher ban penalty for failing future audits than higher reps)
    – Dave Alperovich
    Feb 13, 2014 at 17:29
  • In fact, I wonder if all users should have to take such a test to qualify for reviewing...
    – Dave Alperovich
    Feb 13, 2014 at 17:32
9

Does having a high rep prove that I am a good reviewer?

No. There is considerable evidence that this is not the case.

However, it is the best metric that we have available to us, however imperfect it is.

Having more reputation increases the odds that you'll be a good reviewer, and also makes it more likely that you'll know where to go to find out what to do when you are unsure.

If you can think of some other metric that is likely to allow more qualified reviewers to review sooner, or prevent more low quality reviewers from being able to review, then by all means, propose them here on meta. We'd love to evaluate them and see if they can help us get more good reviewers in and more bad reviewers out. It's a hard problem, one that many smart people haven't been able to solve, but progress has been made in no small part due to constant community participation in trying to solve the problem.

Improve this answer
24
  • I think a user's audit history is the best indicator of reliability. if a 1K user has a high audit passing history, then I believe that user should be allowed to vote on closing, tag wikis, etc
    – Dave Alperovich
    Feb 12, 2014 at 20:21
  • @DaveA Each type of review action is radically different, and being more experienced at one doesn't really make you more experienced at another. Knowing what types of edits should be suggested doesn't mean I know what types of questions should be closed. They each have an entirely separate pool of knowledge. The only real way to know if someone is qualified to review close votes is to let them review close votes and see how well they do, and of course automated tests, such as audits, have problems doing that effectively as well.
    – Servy
    Feb 12, 2014 at 20:23
  • Servy, I agree on that point too. But here again, I suspect that each user's history of audits in a particular review type is more indicative of their ability to review than their rep (once they hit beyond some number which I would ball park at 1K)
    – Dave Alperovich
    Feb 12, 2014 at 20:28
  • @DaveA So you're suggesting that users must review some certain number of suggested edits before they should be allowed to, for example, review close votes? If that's your suggestion, then by all means, propose it, although given that the former doesn't really do much to prepare the user for the latter, in combination with the fact that audits are only there to remove the very worst of the worst of reviewers, not to ensure top notch review quality, and are thus easily defeated by poor reviewers, I don't see it being particularly helpful and wouldn't support it.
    – Servy
    Feb 12, 2014 at 20:40
  • Servy, I don't have a particular suggestion yet (thus the discussion). But I do suspect that practice with audits makes users better reviewers. And yes, I do believe that anyone should be allowed to learn against audit questions. I understand why you disagree.
    – Dave Alperovich
    Feb 12, 2014 at 20:43
  • @DaveA Audits are there to catch people who aren't even reading the content, and who are selecting an option literally without looking at anything. If you took the time to so much as look at anything, you have surpassed the bar for audits. That doesn't make you a good reviewer. It doesn't even make you an acceptable reviewer. It just means you're not a malicious reviewer. Personally I'm not willing to set the bar that low on who I want to have access to the review system.
    – Servy
    Feb 12, 2014 at 20:46
  • Servy, I agree with your point about the intention of audits. But I ask: if a 1K rep user performed 100 closer review audits and was correct every time (this is admittedly forced scenario), would you trust that user to perform close reviews?
    – Dave Alperovich
    Feb 12, 2014 at 20:48
  • 1
    @DaveA No. It's simply too easy to game that system. All it takes to know whether a post is an audit, before submitting an answer, is to click the link to go to the post. This give you 100% certainty as to what the answer is, given how audits currently work.
    – Servy
    Feb 12, 2014 at 20:50
  • O, good point. Then the audit system would have to NOT have a real link to the original question. And that involves new development. Which is always time consuming.
    – Dave Alperovich
    Feb 12, 2014 at 20:53
  • 1
    @DaveA On top of that, people out there just looking to get a gold badge are likely willing to take the time to sit through 100 reviews just to have access to the queue. Someone just interested in helping out the site and making a meaningful contribution is likely uninterested in wasting a ton of time doing 100 audits, and is instead going to just find some other way (of which there are many) where they can meaningfully contribute. You're encouraging the undesirable users and discouraging the desirable ones.
    – Servy
    Feb 12, 2014 at 20:53
  • Thats why I dislike the badges. I get that theyre incentive to do what people would otherwise not do, but I think they promote the wrong behavior.
    – Dave Alperovich
    Feb 12, 2014 at 20:55
  • @DaveA Even that is still easily gamed. I can just google the text of the question and find the real question in an instant. It's a very mild deterrent you've created. Creating a true automated test that can accurately evaluate whether someone is a qualified reviewer is going to be excessively hard.
    – Servy
    Feb 12, 2014 at 20:55
  • That's the best point of all. That an authentically good testing system is too much work.
    – Dave Alperovich
    Feb 12, 2014 at 20:55
  • 1
    @DaveA Yes, I've very strongly protested the introduction of those badges from the very start, and continually ever since, but they're here, and sadly it doesn't look like they're going anywhere, so you need to be prepared to deal with them in your proposed solutions to review-queue related problems.
    – Servy
    Feb 12, 2014 at 20:56
  • 1
    @DaveA And doing that properly is hugely involved, likely in ways you're not even aware. You need to be very careful about what types of posts you use as input, how you verify that you don't have "incorrect" answers, while also not re-using questions much (as then people just post an "answer key") you need to prevent people from finding the real posts (which is super hard, if even possible) to see what actions others took, you need to have questions that "test" different types of common problems, as well as some of the less common problems, you need to minimize manual intervention, etc.
    – Servy
    Feb 12, 2014 at 21:04
5

So I wonder: Does having a high rep prove that I am a good reviewer?

Of course not but having high rep means that the person has a better understanding of what questions/answers should look like. It shows the person has been involved in many questions and/or answers. Many people with low rep are still working on understanding how posts should look. So if you are suggesting lowering that rep minimum then I don't think it's a good idea.

answering questions and earning StackOverflow rep did NOT prove I was worthy of being a reviewer.

No but it showed that you have seen many posts and hopefully understand what good posts look like.

earning StackOverflow rep did not prepare me to do quality reviews.

Maybe not for you but it certainly does help the majority of users to understand what makes a good post.

a track record for good reviewing seems to be a more reliable measurement of how reliable a reviewer is.

I certainly agree but there needs to be some requirement before a person is able to start reviewing. With needing only 3 votes for closings, think about how bad it could be with new members reviewing.

Improve this answer
13
  • I agree that practice posting helps the learning process. But is there a point of diminishing returns? are 3K users likely to be more trust-worthy to do close reviews than 1K?
    – Dave Alperovich
    Feb 12, 2014 at 20:18
  • "point of diminishing returns" I don't think so. I think in general they are more trust-worthy. I know I understand a lot better now of what is expected than when I had 1k, 3k, etc... At 200 I was still learning about what made a good, useful answer. Useful in that it taught something instead of just posting a code-only answer.
    – codeMagic
    Feb 12, 2014 at 20:23
  • That's an interesting point. I think it's the inverse with me (though again, being a proven unreliable reviewer I may not be representative). I suspect that if after 1K rep I was more interested in performing review-audits (I know theres no option to just do that), I would have done more to improve as a reviewer.
    – Dave Alperovich
    Feb 12, 2014 at 20:26
  • @DaveA I do agree that audit history probably proves more than rep once you get to a certain point. But there has to be that "certain point" somewhere in there and it should be relatively high if it allows users to close posts and such.
    – codeMagic
    Feb 12, 2014 at 20:32
  • codeMagic, I'm not sure I agree or disagree. I'd only counter my suspicion that a 1K user with an impecable history with reviewing in other areas would probably be a great reviewer of closing questions. And with that queue so high, we could use a lot of help by opening that door.
    – Dave Alperovich
    Feb 12, 2014 at 20:34
  • I highly disagree that someone might be good in one area just because they are in another area, especially when talking about closing a question. I would say raising the cap on daily tasks would be a better way to help out the queue
    – codeMagic
    Feb 12, 2014 at 20:37
  • I like the idea of raising that limit. And I admit that I'm speculating. In fact, the only way to be sure is to allow lower ranked users to test themselves against audits.
    – Dave Alperovich
    Feb 12, 2014 at 20:39
  • I see your point and its an interesting thought, though I'm not sure how well it would pan out. How would you go about doing that? With the different types of reviews you would need a lot of testing to have a good idea and I'm not sure many users would want to go through that.
    – codeMagic
    Feb 12, 2014 at 20:58
  • My feeling is that the users who WOULD want to do that to be empowered to review would make better reviewers than us rep people
    – Dave Alperovich
    Feb 12, 2014 at 20:59
  • Hmmm...that's a fair point! However, I still think a good amount of rep is needed because I can take and retake tests and finally see some pattern to passing the tests. But having practical, hands-on with the material (such as commenting and answering on posts) is the best way to understand what is actually expected.
    – codeMagic
    Feb 12, 2014 at 21:01
  • agreed. But the ideal answer probably falls somewhere in between.
    – Dave Alperovich
    Feb 12, 2014 at 21:02
  • 1
    I agree to that point. Damn, nothing left to argue about...I'll think of something :P
    – codeMagic
    Feb 12, 2014 at 21:03
  • 1
    LOL! it's meta, there will always be something to argue about.
    – Dave Alperovich
    Feb 12, 2014 at 21:04
5

I have seen my share of 10K+ users make poor judgements and vehemently ignore the guidelines in the help center. IMHO rep is an indicator of how much work you have done and how much others have liked your work, not a direct indicator as to the kind of person you are. For that reason I have always wished I could vote up or down a person's account, and then be able to sort posts or answers based on the person's profile score, but I know that is a pipe dream and will probably never be a part of SO. Just my opinions on the subject. Cheers.

Improve this answer
8
  • 1
    couldn't agree more. Not sure if audit tests, more aggressive audits, or peer review and scoring of review history would be superior, but all seem like plausibe improvement.
    – Dave Alperovich
    Feb 13, 2014 at 16:30
  • 1
    Succinctly said, and I agree ;-) pffft! Well, except on the ability to vote on accounts rather than users.
    – Andrew Barber
    Feb 13, 2014 at 16:51
  • Thanks @DaveA. I've been review banned twice due to audits. Mea culpa. I want to believe those incidents have improved my ability to perform reviews, but I am wary of that because of the amount of subjectivity displayed on this site. I've flagged questions that should be closed, only to have them declined. I've seen posts that go against the guidelines, yet remain open because they are popular. If we are to expect our community to work as a team, we have to do as we say and stop this sort of behavior and lead by example to set a precedent for the quality we expect from everyone.
    – Anil
    Feb 13, 2014 at 16:53
  • 1
    @AndrewBarber, thanks! But are you sure you're responding to me this time. ;)
    – Anil
    Feb 13, 2014 at 16:54
  • @SlyRaskal, you hit a bulls-eye on a few points. That many audit questions miss their mark or are misleading is well known. The best defense has been that making good audits is tuff. You're right that many senior SO users are sloppy or apathetic in their review activities (often chasing silly things like badges!). But be aware that the close question queue is consistantly over 100K. It's one of the reasons I've argued for lowering the bar for involvement and finding new standards (such as audit reliability).
    – Dave Alperovich
    Feb 13, 2014 at 17:04
  • I suspect, with your responsible attitude, you would make a better reviewer than many of us "qualified" to perform those tasks.
    – Dave Alperovich
    Feb 13, 2014 at 17:07
  • 1
    @SlyRaskal Actually, I meant to post that to my own answer, but then realized "succinct" doesn't apply to much of anything I do. Including my Tweets!
    – Andrew Barber
    Feb 13, 2014 at 17:25
  • 1
    @AndrewBarber, thank you for the laugh! I needed that.
    – Anil
    Feb 13, 2014 at 17:38
3

Having a reputation of a few thousand is a good sign that you might be a good reviewer. By then you will have seen enough of the mechanics of the site to see how things work.

The audit checks are there to help assert that you are or are becoming a good reviewer. They are there to help you learn; as I am still doing.

Improve this answer
7
  • that may be true. But suppose a good technical person reached 10K and never performed a single review? What has he proven as a reviewer?
    – Dave Alperovich
    Feb 12, 2014 at 20:16
  • 1
    @DaveA: Nothing. (Did you expect a different answer than that?)
    – Keith Thompson
    Feb 12, 2014 at 21:21
  • @KeithThompson, a rhetorical question isn;t expecting a serious answer.
    – Dave Alperovich
    Feb 12, 2014 at 21:22
  • Succinctly said, and I agree. :)
    – Andrew Barber
    Feb 13, 2014 at 14:09
  • @AndrewBarber, Was your comment in regards to my comment or Bathsheba's answer? Just wanted to be clear before I made a fool of myself and responded. Cheers. ;)
    – Anil
    Feb 13, 2014 at 16:25
  • @SlyRaskal Eek! I was commenting to Bathsheba's answer. I often forget to be clear about that sort of thing! :D
    – Andrew Barber
    Feb 13, 2014 at 16:47
  • @AndrewBarber No worries at all, my suspicion was that you were commenting on Bathsheba's answer, but just wanted to be certain. Cheers.
    – Anil
    Feb 13, 2014 at 16:50
1

Stackoverflow is a big, user moderated community. It means, beyond core developers and moderators, site is moderated by users.

In that case, there is a ladder-like mechanism that grand users privilages to moderate. Because, as I mentioned above, users expected to help moderation. Like

  • Editing questions to fit them better shape, or suggest edits. Or update them if informatin ithin it was deprecated for some reasons or became less useful.
  • Inspecting questions and closing ones that are duplicate or do not fit here

etc.

As you probably know, all things about the site and moderation are argued in Meta, and applied if it gets related support from the users.

With increasing reputation, community gives you privilages so you can fix things when you come-across problematic questions/answers. Community needs you because there are much more user input then the moderators can watch and control. User counter is now 3.3M! It needs a very big team for moderation, or here will become a mess.

Community thrusts you because you learn how to ask and how to answer while you earn that reputation. You know what is welcome in here and what is not.

You can make mistakes, everybody make mistakes. But there are auditions and other high-rep users that can correct your mistakes and warn you so you learn how you can moderate better. If you fail many audits, you got review ban. but if you delete a lot of questions that belongs to you, then you may get a question ban too. Everything needs some time to learn and some care while doing in here. It is a good thing to have.

Improve this answer
5
  • Agreed on all points. But I suspect that once you get beyond a certain level of experience (say 1K) audit history is more indicative of your reviewing reliability than rep.
    – Dave Alperovich
    Feb 12, 2014 at 20:30
  • Of course, but the basic idea is we need people to help moderate. Then who you can trust? you can not select 10.000 users for such things because it is neary impossible to test all thier skills in moderation. So reputation is a tool to choose ones to give those privilages. And also, to have aan audit history, a user must first be able to do reviews.
    – FallenAngel
    Feb 12, 2014 at 20:34
  • Also true. And rep IS a useful benchmark. upto a point. At some point, if we just let users test themselves on audits only, we would discover hidden talent. And I suspect such people would be more motivated too.
    – Dave Alperovich
    Feb 12, 2014 at 20:36
  • Also you can check my last edit that eddied a few more sentences to the end.
    – FallenAngel
    Feb 12, 2014 at 20:38
  • Just saw them. I hear you.
    – Dave Alperovich
    Feb 12, 2014 at 20:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .