Summary
Implement a system that automatically informs and potentially review-bans reviewers (with optional adjustments) based on moderators marking individual reviews as bad. This makes moderating reviews less tedious and allows for a gentler handling and guidance of bad reviewers.
Proposal
Give moderators the option to mark individual review decisions as bad at appropriate places – at least when viewing review items.
Inform reviewers when and which review has been marked as bad, e.g., via their inbox or before they want to review again. Provide them with guidance where they can learn about reviewing and where they can ask why their review was bad.
Implement an escalating automatic ladder of review bans based on bad reviews, e.g., three days for the second bad review within a hundred reviews, a week for the fourth, etc.
When a moderator marks a review as bad, provide them with an interface that allows them to adjust a potential ban and provide a short explanation why a review was bad.
I am not very familiar with how automatic bans due to audits work, but it probably makes sense to count a failed audit as one or more bad reviews in the above sense.
The Underlying Problem
This is not about blatant robo reviewers, for which review audits exist. I am rather talking about reviewers that:
- are very lazy, e.g., selecting no action needed in the first-post queue 90% of the time or always choose the same canned comment in the low-quality queue every time (even though it only fits in 5% of the cases),
- review quick enough to avoid blatant mistakes but too quick for anything beyond that.
- ignore established policies, e.g., by voting to close every single question of a type that the rest of the community agrees should not be closed,
- make overly positive or negative decisions.
These reviewers cannot be detected automatically and only barely with statistics. Some of them reviewers just need a gentle reminder; some need a ban to learn. However, for some, a ban may already be too demotivating – in particular if imposed by a human. This can be more convincing, if you can point to a few clearly bad reviews, but these may be tedious to find. Finally, the review system can handle the occasional mistake, and of course moderators make the occasional mistake as well.
A system that forwards warnings for each bad review and escalates in a predescribed (but adjustable) manner solves most of these problems: Reviewers get a neutral, mostly automatic reminder that they did something wrong and can learn from it without being put off too much. If this was a one-time incident or the review decision is not that clear-cut bad, no big harm is done. Should the reviewer accumulate bad reviews nonetheless, they get small bans, which again are mostly automatic.
From a moderator’s point of view, they only have to decide that a review was bad, but not that it was bad enough to justify a ban. While this may not look like it in the first place, this can is a huge difference, given that bad reviews are much more frequent than abysmal reviews.
“Why do the current tools not suffice?”
Technically, moderators could keep book of bad reviews and inform users via several communication systems about their bad reviews. However, the former is very tedious and all methods for the latter come with their problem:
- Dropping a quick note to the reviewer via a comment or in chat has the problem of public shaming.
- Opening a private chat is a tedious procedure.
- Sending a moderator message has the problem that those are usually known to be a very heavy tool by users. Also, you would still need to collect links to bad reviews, etc.
- The review ban notice only works if you wish to ban in the first place and moreover a reviewer may miss it if they do not attempt to review while banned. (Also, there is this bug.)
Thus, even for a heartless pedant like me moderating reviews is often too tedious to be worth the effort.
There is a userscript that automatises part of the process and allows moderators to issue review bans directly from the review view, but comes with all the disadvantages of the review-ban notices listed above. It also does not feature a default escalation system and does not have the place for elaborate guidance, both of which increase the chance of acceptance by the reviewer.
