18

Summary

Implement a system that automatically informs and potentially review-bans reviewers (with optional adjustments) based on moderators marking individual reviews as bad. This makes moderating reviews less tedious and allows for a gentler handling and guidance of bad reviewers.

Proposal

  • Give moderators the option to mark individual review decisions as bad at appropriate places – at least when viewing review items.

  • Inform reviewers when and which review has been marked as bad, e.g., via their inbox or before they want to review again. Provide them with guidance where they can learn about reviewing and where they can ask why their review was bad.

  • Implement an escalating automatic ladder of review bans based on bad reviews, e.g., three days for the second bad review within a hundred reviews, a week for the fourth, etc.

  • When a moderator marks a review as bad, provide them with an interface that allows them to adjust a potential ban and provide a short explanation why a review was bad.

I am not very familiar with how automatic bans due to audits work, but it probably makes sense to count a failed audit as one or more bad reviews in the above sense.

The Underlying Problem

This is not about blatant robo reviewers, for which review audits exist. I am rather talking about reviewers that:

  • are very lazy, e.g., selecting no action needed in the first-post queue 90% of the time or always choose the same canned comment in the low-quality queue every time (even though it only fits in 5% of the cases),
  • review quick enough to avoid blatant mistakes but too quick for anything beyond that.
  • ignore established policies, e.g., by voting to close every single question of a type that the rest of the community agrees should not be closed,
  • make overly positive or negative decisions.

These reviewers cannot be detected automatically and only barely with statistics. Some of them reviewers just need a gentle reminder; some need a ban to learn. However, for some, a ban may already be too demotivating – in particular if imposed by a human. This can be more convincing, if you can point to a few clearly bad reviews, but these may be tedious to find. Finally, the review system can handle the occasional mistake, and of course moderators make the occasional mistake as well.

A system that forwards warnings for each bad review and escalates in a predescribed (but adjustable) manner solves most of these problems: Reviewers get a neutral, mostly automatic reminder that they did something wrong and can learn from it without being put off too much. If this was a one-time incident or the review decision is not that clear-cut bad, no big harm is done. Should the reviewer accumulate bad reviews nonetheless, they get small bans, which again are mostly automatic.

From a moderator’s point of view, they only have to decide that a review was bad, but not that it was bad enough to justify a ban. While this may not look like it in the first place, this can is a huge difference, given that bad reviews are much more frequent than abysmal reviews.

“Why do the current tools not suffice?”

Technically, moderators could keep book of bad reviews and inform users via several communication systems about their bad reviews. However, the former is very tedious and all methods for the latter come with their problem:

  • Dropping a quick note to the reviewer via a comment or in chat has the problem of public shaming.
  • Opening a private chat is a tedious procedure.
  • Sending a moderator message has the problem that those are usually known to be a very heavy tool by users. Also, you would still need to collect links to bad reviews, etc.
  • The review ban notice only works if you wish to ban in the first place and moreover a reviewer may miss it if they do not attempt to review while banned. (Also, there is this bug.)

Thus, even for a heartless pedant like me moderating reviews is often too tedious to be worth the effort.

There is a userscript that automatises part of the process and allows moderators to issue review bans directly from the review view, but comes with all the disadvantages of the review-ban notices listed above. It also does not feature a default escalation system and does not have the place for elaborate guidance, both of which increase the chance of acceptance by the reviewer.

  • "Implement an escalating automatic ladder of review bans based on bad reviews" No need for anything special there, just count it as a failed audit, letting whatever review ban, if any, would result from an additional failed audit for that user. – Servy Jan 29 at 23:05
  • 9
    @Servy most sites don't have audits. – Catija Jan 29 at 23:06
  • 1
    @Catija So then for those sites this would be the only way of getting "failed audits". One would assume that all the code to check if a user is review banned based on their audit history is still there, even on sites where users aren't being shown audits (the count of failed bans currently just doesn't ever go above zero). – Servy Jan 29 at 23:07
  • @Servy: I am not familiar with how review bans due to failed audits escalate and I cannot find information on it, but yes it makes sense to handle those in the same way. (Also see my edit.) – Wrzlprmft Jan 29 at 23:10
  • @Wrzlprmft meta.stackexchange.com/a/287296/186381 – Servy Jan 29 at 23:11
  • It would be great if this worked like a failed audit in retrospective. – Magisch Jan 30 at 7:44
  • Hopefully, more then a single moderator will be required before a review ban is issued. In a similar fashion that multiple reviewers must vote the same way. – Ramhound Jan 30 at 15:27
  • interesting feature, though IMO only after this list is reduced significantly ... – Pierre.Vriens Jan 30 at 18:57
  • 1
    @Ramhound: Moderators can issue review bans unilaterally already – like the vast majority of moderator actions, many of which have a higher impact that disallowing a user to review. Requiring multiple moderators to agree on this would make things unnecessarily slow and tedious. Also, it’s not that there is not way to appeal bad decisions. – Wrzlprmft Jan 30 at 19:32

You must log in to answer this question.

Browse other questions tagged .