Tim Post's FAQ on review audits includes instructions to post on the per-site meta and use the disputed-review-audits tag. That suggests that at least at one time, there was a provision for culling invalid audit posts. Stack Overflow Meta still has that tag, Super User Meta doesn't (if it ever did). Invalid audits still exist. I don't know whether use of this tag on SO Meta actually results in action of some kind. On SU, it appears that a decision was made at some point that close enough is good enough, and human resources aren't justified to clean up bad audit posts. Perhaps the need for cleanup was greater when the audit system was first rolled out, then it was eventually tweaked to an acceptable level.
That may well be a proper economic decision. Invalid audits are usually uncommon exceptions, although there are some logical flaws in the design that can lead to frequent incorrect audit failures in some queues.
While this may be good from an economic perspective, it has a downside that hurts the review process. Most reviewers are "rank and file" site members who don't frequent SE Meta, and don't routinely research issues that may have been discussed here. They often lack the perspective provided in Tim Post's FAQ thread. Failing an audit can be a somewhat jarring experience (by design), and it isn't perceived as gamification.
Even more-experienced users, who understand the audit process, have a limited tolerance for invalid audit failures. I had that experience myself due to a logic flaw, and posted about it on SU Meta: Design flaw in the review audit system. I got tired of constantly (and incorrectly) failing audits and stopped reviewing in the First Posts queue for a number of years.
The review queues rely on users volunteering their time. Invalid audit failures on more than rare occasions discourage users from participating in the reviews. Without sufficient reviewers, some of the queues get backlogged, and feedback to posters, and action on posts, isn't timely.
"How close is close enough" should be considered from the perspective of the reviewers, not just the site designers. The system needs some mechanism to allow problematic audit questions and logic flaws to be reviewed.
