At first, this may look like a duplicate, but the other one was downvoted, it seems, because people disagree for my reasoning on why I failed. Please keep your votes centered around the request.
I just failed an audit on a question that should be closed. Why? Because it has 6 upvotes and no close votes:
Look at the question. It's obvious it's one that the community likes, but is that a valid reason to put it as an audit? Maybe not, but because it has 6 upvotes and no close votes it should be put in the queue!? NO.
This question is obviously opinion based, and something needs to be done about it. What I've been thinking is something along the lines of a audit invalidator.
We would need to start by showing multiple people the same audit. We then compare their results, and decide to invalidate an audit if enough people fail it. Now, I know you people like numbers (and unicorns), so here is an example scenario:
Let's say we show an audit to 6 people. 5 fail, but only 1 passed. It's pretty obvious it was a bad audit. What do we do? We invalidate the audit. Ultimately, it is still counted as a review, but it works like this: The people that failed lose that "mark" in the system, while the person that "passed" has no effect on him.
I know what you're asking: What do we do with the people who "pass" a lot of bad audits? Simple. Remember that person who "passed" in the last paragraph? Yeah, that one. We show him the audit again. If he chooses what would have been a "fail" in the first one (which is now a pass), it appears he's learned his lesson. So we let him go. If he fails this (now reversed) audit, we warn him that he should read the messages, then disable the "I understand" button for a longer period of time.
Now, the second part of this proposal (showing the audit another time) is just me brainstorming. Nothing required. However, the first part (invalidating audits), is the main part of this proposal. Please don't downvote if you disagree with the second part. Please focus your votes on the main part - the first part.