First and foremoest: this topic has nothing to do with the existing audit system. It detects people blindly voting, it does that well, nobody disputes that. This topic is about detecting, warning and correcting moderators that continuously and structurally cast votes that disagree with final consensus on the review in question.
Been active for about a month now, got my 3k reputation so I can see all the review queues, and been busy at work taking my responsibility to do the community part of moderation. While at this, I've been stumped a few times already at where I saw ridiculous edits being approved, and good questions being closed without real good reason, for weird reasons just because the post had been flagged, and the flag blindly followed.
I decided to dive into this further when I just noticed the handling of this particular suggested edit. Notably, the moderation system actually worked fine in this case — it's a prime example of a bad edit, adding more information in a link that without a doubt should have been a comment, and it got rejected for exactly that reason. More notably however, 2 out of 5 reviewers actually approved this edit that is so obviously out of line. And most notably, one of the 2 approvers "has approved 337 edit suggestions and rejected 8 edit suggestions". And that's where I suspect we're having a problem that isn't fixing itself.
I've dived into this before, and noticed that most people that actually do a proper job reviewing have a reject rate of between 25% and 50%, like myself. The specific person I'm talking about has rejected 8 out of 345 suggested edits, or about 2.5%. Now that should not only raise some red flags automatically in the system methinks, it should've done that 250 edits ago - even 8 out of 100 would've been an exceptionally low ratio.
Looking up relevant discussion history here on meta found me this topic on forcing some kind of balance between approvals and rejections, and this one on robo-approvers killing review quality. The first has as accepted answer that the 'gotchas' in the queues should be able to fix this, for which I've seen too many examples already to be sure they don't (apparently they read just well enough to catch the gotchas, and approve nearly everything else), and the second boils down to a different problem that was fixed, but doesn't fix this.
I consider this to be a real problem — it's a serious defect in the review system if people can keep this up, and multiply because they aren't being corrected, since at some point there will be enough robo-approvers to become statistically relevant, and they will accidentally start hitting enough posts together to allow a lot of bad moderation decisions to slip through. The sample post I linked demonstrates this — it boiled down to two that didn't actually read the edit the properly, vs. three that did. That's very close to being the other way around.
So, I definitely think the system should analyze reviewing statistics, and keep track of when people:
- Have an approve/reject ratio that is way outside the site-wide normalized range
- Make a statistically relevant number of decisions that do not agree with the eventual outcome of the review
In both cases, if a flag is raised, the user should be put on review alert (with notification of this) where moderators (and/or for example 10k+ rep users) should review (some of) their decisions, until it is either decided that the user will not learn and have his reviewing privileges revoked or downscaled, or he has improved his moderation quality sufficiently to remove the flag.
Let it be said that I'm all for differing opinions, that's what makes a democratic moderation system work, but if someone disagrees with the rest of the system 90+% of the time one has to wonder whether he should be part of the system and be allowed to continue 'sabotaging' it.