Invalidate reviews by known robo-reviewers

Question

Today, I saw a known robo-reviewer get out of a well-deserved review suspension, only to continue rubber-stamping every review in sight again.

Some robo-reviewers get review-banned 10 times or more, and still continue reviewing.

The audit system catches some of them. Others manage to skip or pass the audits, but happily rubber-stamp everything else.

However, between two review suspensions, these robo-reviewers can do serious damage.

My proposal is that reviews that are done by known robo-reviewers get invalidated, and placed back in the queue.

We could first do this in the Suggested Edits queue, where the audits are obvious.

So, here's the process:

• A user has been suspended from review, by failing audits in the Suggested Edits queue, for at least 3 times in the last 6 months. (We can tweak the numbers as we see fit.)

• This user fails another audit in the Suggested Edits queue.

• Now every edit Approval by this user since their last review suspension (up to, let's say, 3 days) is invalidated. The edit suggestions are placed back in the queue, with the other reviews still valid. So if it was Approved by 3 and Rejected by 2, it would now need one more review.

• If the edit gets Approved by the new reviewer, great — nothing to do here.

• If the edit gets Rejected by the new reviewer, it is automatically rolled back, and the editor loses their +2 rep. Progress towards badges (Copy Editor for the editor, Steward for the reviewer) is lost.

Clearly, the exact algorithm needs some thinking. For example, to prevent a review item from being shuffled in and out of the queue repeatedly, maybe this should only be done once per item.

Alternatively, maybe the review should only be invalidated if an item still is in the queue, as this is probably easier to implement than a system that puts review items back in the queue.

Either way, it would contain the damage done by robo-reviewing.

Answer 1

This is an idea moderators have kicked around internally:

It's also a real pain to go back through a reviewer's history and delete mis-approved spam, so for more aggressive cases I might be useful to invalidate reviews from certain reviewers and have items go back into review in bulk.

but I'm concerned about the complexity and side-effects of any system-triggered process for this. Edits would be particularly troubling, as the edit reviews involve several people, and once they're approved they change a post. Do you roll back the edit? What if there are edits after that?

The cases where I envisioned review invalidation were in the First Posts and Late Answers review queues. In both of those, it just takes a single person to approve a post to kick it out of review. Bad reviewers can (and regularly do) approve multiple instances of spam, non-answers, or trolling before hitting their first audit. Invalidating reviews like this would simply involve placing them back in the review queue for someone else to look at, as if someone had cast a "very low quality" flag on them.

Even there, I thought about this as more of a manually-triggered action by moderators once we came across truly bad reviewers (and not just people voting "Looks OK" on a borderline question), possibly with the ability to set a time range for review invalidation. That would still be a complex new bit of UI that would take developer time to implement.

bluefeet recently queried review ban statistics on Stack Overflow, and 7301 reviewers have been banned via audits at any point. 4292 have been banned two or more times. 370 have been banned 10 or more times. Compared to the total number of reviewers on the site, that's a small fraction of bad actors, but they do cause damage.

There might be simpler actions that could be taken first to reduce this damage. Extending ban durations would reduce the number of repeat offenders active in the review queues. Requiring more than one review for First Posts or Late Answers would help catch instances of spam, non-answers, or trolling that slipped through a single bad reviewer (and to identify those reviewers earlier). Audits in Documentation will largely deal with the current review problems there.

It's a good suggestion, I just worry about how difficult it would be to implement. Maybe there are simpler things that could be done first.