Lacking confidence in my reviewing abilities, I usually avoid the review queue. But, I've been asking a decent amount of questions lately and haven't found many questions I could answer (so as to maintain karmic neutrality or whatever). So, I've been reviewing more lately. Wanting to improve my reviewer chops, I've been looking around for ways of getting feedback, but not finding much. Sure, sometimes the review system gives you a fake question, but those're pretty obviously good/bad.
Wanting to spur some good discussion, let's lay out some basics.
Here are some hypothetical examples:
(A) 85% of the questions I marked as "Looks OK" ended up being considered genuinely OK.
(B) 40% of the questions I flagged for "Requires Editing" ended up being closed for needing more detail.
(C) Typically when I flag a late answer to a python question, it's subsequently unanimously marked as OK.
Note that, in each of these cases, I'm learning something specific and actionable about my reviewing approach. Also note that this information is nuanced: it's easy to see whether or not my "Looks OK" reviews match the consensus with a check or X UI element (as suggested here), and a percentage agreement with the consensus (as suggested here) would provide a more quantitative metric. But, neither would provide the more targeted feedback shown in hypotheticals (B) and (C). Equally importantly, the consensus isn't always right, so we shouldn't give this impression.
Not Outsmarting Ourselves
For several reasons (humans are imperfect, reviews are limited, etc.), we shouldn't push everyone to review exactly the same way. So, an important question in any such improvement is how to allow for some wiggle room? To me, this is the keystone to doing this right. Indicating that the consensus answer is always right/wrong is a problematic precedent. Perhaps the feedback system should compare how often my reviews deviated from the norm to how often others' reviews do so, i.e. normalizing the variance. What about a system that provides notifications when we're making the same mistakes? E.g. the next time I flag a late python answer, a pop-up tells me that reviewers don't usually agree with me when I do this. In this way, a nudge is provided against my worst behaviors right as I'm making them.
As it stands, I've had a really hard time figuring out how to be a better reviewer, and it continues to be a deterrent to me reviewing items. This feels bad for the community, and it feels fixable. But, I'm far from a specialist in these matters, and don't have a good sense of what is doable. So, how could automated reviewer feedback be better conveyed? Do any of the ideas above hold water?
The zeroth-order solution would be this feature request or a slightly improved version in this one, though, as laid out in this question, I don't think it'd be hard to provide much more powerful feedback that also doesn't push people too hard toward reviewing the same way.
Reorganized, updated throughout to juxtapose more strongly to the two related questions (for which this question is currently marked a duplicate), and to emphasize how this question differs.