When reviewing, it would be nice to see whether your Accept/Reject decisions are in line with the community's decisions. Specifically, I think the stats page should indicate the four signal detection theory quadrants:

  • How many reviews did you mark Accept that were accepted? (hit)
  • How many reviews did you mark Accept that were rejected? (false alarm)
  • How many reviews did you mark Reject that were rejected? (correct rejection)
  • How many reviews did you mark Reject that were accepted? (miss)

Perhaps clicking on the number might bring you to a list of those questions (e.g. for the second one, a list of questions that I accepted that others rejected).

Assuming that the community is correct, on average, then one could use this information to calculate d-prime (sensitivity) and beta (response bias). These are good metric to compare the quality of reviewers.

As is, you can review the history of your reviews, but tabulating this information would require clicking on every single link.

  • 12
    "Assuming that the community is correct, on average" - On average, they're not... – animuson Dec 3 '12 at 1:56
  • 2
    i think >50% of accepted reviews are legitimate. i'd like to see evidence otherwise. but regardless, it doesn't affect my proposal... – Jeff Dec 3 '12 at 1:59
  • 4
    It'd be a nice feature for those that care about their reviews, but unnecessary noise to those that only care for the badge. I care about my reviews, so I'd love to see this implemented. – Makoto Dec 3 '12 at 3:50
  • 2
    @Makoto Is there a badge for caring about your reviews? – Asad Saeeduddin Dec 3 '12 at 21:41
  • 1
    @Jeff >50% of reviews may be correct, but I can be sure that somewhere in the ballpark of 95% of my reviews in category 4 really should have been rejected, rather than that I should have accepted it, and the poor reviewers will have way too few items in category #2, despite accepting lots of things that they shouldn't. This makes the two useful stats meaningless. – Servy Dec 3 '12 at 21:48
  • @Servy the problem is that serial-approvers have no "correct rejections". these users would have an extreme bias towards approve, far from optimal beta. the 50% comment was in response to animuson: if >50% of reviews are correct, then on average, the community is correct... – Jeff Dec 3 '12 at 22:09
  • please see, e.g. Stanislaw & Todorov, 1999, Wickens, 2001 – Jeff Dec 3 '12 at 22:12
  • @Jeff You don't need this to notice serial approvers. You can already see number of approve/reject votes. Next, I would assert that the number of low quality reviewers is actually dangerously close to 50%; it certainly exceeds 50% over certain local periods of time. Now, most poor reviewers do nothing but approve, and there is a slightly higher percentage of suggestions that should be approved than should be rejected, thus a large percentage are "correct by accident" when the coincidentally approve content that should be approved. Thus the percent of correct decisions is >50% due to that. – Servy Dec 3 '12 at 22:13
  • @Jeff But it looks so boring :( – Asad Saeeduddin Dec 3 '12 at 22:15
  • @Servy reviews get rejected all the time. The problem with serial approvers is noticeable every time it happens, but from my experience, it affects less than 20% of reviews. Another factor is that you personally might have different acceptance criteria than the average reviewer. which would cause the percentage of approved reviews that you disagree with to increase – Sam I am says Reinstate Monica Dec 3 '12 at 22:18
  • @Asad haha, i'll give you that, but let me remind you that we're currently talking on a forum about a forum about programming. i'm not sure who's the pot and who's the kettle, but they both need to pipe down a bit – Jeff Dec 3 '12 at 22:21
  • 1
    @SamIam You also need to differentiate number of bad reviewers from number of bad reviews. Bad reviewers can, as a result of the greatly reduced time spent reviewing, end up making more reviews that people who work to do them well. That also skews the results. – Servy Dec 4 '12 at 1:38
  • I think this is a good idea to check whether you (the reviewer) is in line with the other reviewers or if you have a minority opinion. Feedback is always welcome, IMO. – GUI Junkie Jul 21 '14 at 12:29
  • Taking into account the other comments, I'd say that somebody who is a serial-approver would see the error in his/her way and take reviewing more seriously. – GUI Junkie Jul 21 '14 at 12:31

You must log in to answer this question.

Browse other questions tagged .