Audits are a terrible metric. They have little significance within the scope of reviews, and say very little about the reviewer. Using an algorithm to select audits doesn't work particularly well, and there will always be exceptions which aren't covered by the algorithm. This means that the value of auditing is limited, at best.
Anyone looking through a user's history will simply look through their entire history; audits serve no purpose other than to instate bans. Specifically, for each audit type, we see these complaints with significant frequency:
- Close Vote audits are often incorrect, both because of old questions and because people sometimes miss things
- Suggested Edit audits are blatantly obvious (more so than the others), and anyone with eyes would see it (most of the time). This makes SE audits hard to fail, and doesn't actually gather significant data about the quality of reviewing.
- First Posts audits are very easy to detect, and also very easy to game. Several methods for doing so have been pointed out in the past, though I'm not going to list them here.
Each of the audit types has its flaws, each can be easily bypassed, all of them have frequent errors, and as a result, none of them truly distinguish the quality of a reviewer. There have been many pushes on Meta in the past to see audit statistics, as well as questioning the effectiveness of the audit system. Piles are designated towards pointing out bad audits. Take a look at the related questions list to the right of this question.
Proposed is an entirely new system for tracking reviews and reviewers' effectiveness.
Instead of maintaining an audits system which can easily be gamed and hacked around, simply audit based on quorum. This would work in a similar way to the banning system; once you reach a low enough internal score, you're suspended for two days, and given a link to your review history.
- If a user picks an option on an audit which is different than the decided action, then their audit score decreases.
- If a user matches the quorum for an audit, then bump their audit score up a bit.
- If that action leans more towards doing nothing (e.g. "No Action Needed"), penalize the user significantly more than if they took action.
- More contentious reviews would be weighed less (e.g. 3 accepts, 2 rejects).
- Less contentious reviews would be weighed more (e.g. 1 accept, 4 rejects).
This sets up a system where, if the user takes no action continuously, they're suspended for two days. If they, for instance, reject edits continuously, they're also going to be suspended and asked to review their history. However, if a user has a couple discrepancies compared to what was decided, nothing serious will happen, and the instances will be overlooked.
Properly tuned formulas for weighing the significance of their review would negate any suspensions which would arise from a few differing opinions. However, if a user's opinion is consistently different than the majority, then they may be suspended - they should look to see what they're doing wrong. The formulas would be very easy to tweak, since calibration could be done in the background while review audits are still active. The formulas could be perfected before they were ever implemented.
This would need to go hand-in-hand with requiring more reviewers for First Posts.
What are your thoughts? What problems do you foresee with this system, and how would you change it to fix them?
To address issues concerning immediate feedback, here are several options:
- First, we could keep selected audits for users under 250-300 reputation.
- We could notify users when their reviews consistently don't match quorum
- We could show users an introduction page to each review, similar to the "How to Ask" page. Some argue that this page would be totally ineffective, however, its purpose would be to let the user know that they're being held to standards. If they're reviewing posts, they're aware of this fact to begin with.
Since we should probably be finding ways to give users better review feedback, this is an opportunity to do so.
To address issues concerning the self-perpetuation cycle of the review queues, and the possibility that bad reviewers will trump good ones, here are my thoughts:
I don't this will happen, actually. I have confidence that most of the reviewers out there will take some form of appropriate action. Still, it's a valid concern, so:
A possible solution is to give moderators significantly more weight than regular users, and ask them to spend a few days focusing down the review queue. I think that, once the status quo has changed, it will stay that way by its own self-perpetuating cycle. Moderators are there to help guide users, so if it needs a small push every now and again, a moderator can step into the queue.
These are several ideas, and are issues which would need to be addressed, so please post your suggestions below.
