A lot of the proposed solutions to bad review audits require extra work, are too ambiguous, don't work in certain review types, or are impossible to implement at scale.
I have a solution I believe should fill those holes reasonably well.
For argument, let's assume:
- 20% of all review audits are failed
- The good review audits are failed closer to 10% of the time
- Bad audits are less common than good audits, but are failed substantially more. Probably close to 50%.
A lot can be gleaned from this information. Namely, audit quality. Over a certain threshold, like 30%, it's probably a bad audit, or at least "bad enough."
I propose removing the post from the audit pool and reversing any failed audits when failure exceeds that threshold (and raw number of reviews is large enough to reduce noise). It might be beneficial to also consider the experience of the reviewers in determining their weight in this calculation.
There may be a SEDE query that could help to justify this feature, but it feels intuitive enough without.
This on its own will do nothing to address complaints, since it's all behind the scenes. What could help address complaints might be changing the Stop! text to mention the automated review of audits, so users are encouraged to forge ahead knowing there may be redemption for their perceived transgression.