Today, I saw a known robo-reviewer get out of a well-deserved review suspension, only to continue rubber-stamping every review in sight again.
Some robo-reviewers get review-banned 10 times or more, and still continue reviewing.
The audit system catches some of them. Others manage to skip or pass the audits, but happily rubber-stamp everything else.
However, between two review suspensions, these robo-reviewers can do serious damage.
My proposal is that reviews that are done by known robo-reviewers get invalidated, and placed back in the queue.
We could first do this in the Suggested Edits queue, where the audits are obvious.
So, here's the process:
A user has been suspended from review, by failing audits in the Suggested Edits queue, for at least 3 times in the last 6 months. (We can tweak the numbers as we see fit.)
This user fails another audit in the Suggested Edits queue.
Now every edit Approval by this user since their last review suspension (up to, let's say, 3 days) is invalidated. The edit suggestions are placed back in the queue, with the other reviews still valid. So if it was Approved by 3 and Rejected by 2, it would now need one more review.
If the edit gets Approved by the new reviewer, great — nothing to do here.
If the edit gets Rejected by the new reviewer, it is automatically rolled back, and the editor loses their +2 rep. Progress towards badges (Copy Editor for the editor, Steward for the reviewer) is lost.
Clearly, the exact algorithm needs some thinking. For example, to prevent a review item from being shuffled in and out of the queue repeatedly, maybe this should only be done once per item.
Alternatively, maybe the review should only be invalidated if an item still is in the queue, as this is probably easier to implement than a system that puts review items back in the queue.
Either way, it would contain the damage done by robo-reviewing.