Most of the time I'm active on StackOverflow is scrolling through the Review Queues I have access to. These review queues are important to SO's operation, and going through them is my menial way of giving back to a service and community that has benefitted me immensely over the years.
But the system seems fundamentally flawed to me.
StackOverflow does an excellent job of crowd-sourcing QA effort, except on the QA effort itself. This answer outlines the issue very clearly:
We incentivize asking questions. We incentivize answering them. We incentivize making questions better by editing them (for people with lower rep). However, all of these incentive processes have review.
You only get rep from questions when people upvote them. You only get rep from answers when people upvote them. You only get rep from edits when people approve them. In every case, some human being has to look at what you did and say, "Good".
However, despite outlining the problem excellently, I believe it has reached the incorrect conclusion; the problem is not that reviews are incentivized, but rather that reviews are not reviewed.
Of course, there should not be review queues of review queue entries. But consider the concern mentioned in this post:
I went to a Late Answers review queue that had four items in it and started reviewing the first one by editing it into shape rather heavily.
By the time I was done, someone else had reviewed the same post simply by upvoting it, so I only got a "Next" button, not an "I'm done" button. My edit still counted as an edit; but no longer as a review.
Worse still, the other three items were gone from the review queue as well. Curious, I checked the review tab in the profile of the user who had snatched the first review out from under me. Sure enough, he also "reviewed" the other three items, all within a minute, without even fixing obvious typos, simply by upvoting. I had to go through his review list and review every single post again.
As soon as you add any sort of incentive to a task (even if it's just a silver medal), people are going to try and find the easiest possible method of completing that task. Without any way of performing quality assurance on reviews, the situation outlined here will keep happening.
And as important as this concept is from the perspective of pruning out bad reviews, it is equally important to incentivize good reviews. In the situation outlined above, the writer had clearly put significant effort into improve a late answer, but clearly take issue with the incentive disparity for doing that task properly vs. shoddily. I would wager that a large majority of users ignore the review queue because it's not a good means for rep gains (anecdotally, my rep gains have certainly slowed since I started browsing it).
You want to attract high-quality feedback in the review queues, and the current system (including the automated auditing) does not.
As a discussion point:
- Is there a good reason not to incentivize high-quality review queue feedback?
As a feature request:
- Allow visibility of reviews performed, and voting (both up and down) on them. StackOverflow has proven that crowd-sourced QA works; let it work for performing QA on the QA.