8

Apparently, poor questions which were quickly closed with no answers are used as audits even after such questions are edited, reopened and answered. Since audits present the latest revision of the question, they become confusing since the latest revision doesn't have the problems which made the question close-worthy initially.

I think the easiest solution is to exclude questions which have been reopened from potential audits. The alternative is to present the revision of the question at the time it was closed, but I suppose that's harder to implement.

5

Tim Post's FAQ on review audits includes instructions to post on the per-site meta and use the tag. That suggests that at least at one time, there was a provision for culling invalid audit posts. Stack Overflow Meta still has that tag, Super User Meta doesn't (if it ever did). Invalid audits still exist. I don't know whether use of this tag on SO Meta actually results in action of some kind. On SU, it appears that a decision was made at some point that close enough is good enough, and human resources aren't justified to clean up bad audit posts. Perhaps the need for cleanup was greater when the audit system was first rolled out, then it was eventually tweaked to an acceptable level.

That may well be a proper economic decision. Invalid audits are usually uncommon exceptions, although there are some logical flaws in the design that can lead to frequent incorrect audit failures in some queues.

While this may be good from an economic perspective, it has a downside that hurts the review process. Most reviewers are "rank and file" site members who don't frequent SE Meta, and don't routinely research issues that may have been discussed here. They often lack the perspective provided in Tim Post's FAQ thread. Failing an audit can be a somewhat jarring experience (by design), and it isn't perceived as gamification.

Even more-experienced users, who understand the audit process, have a limited tolerance for invalid audit failures. I had that experience myself due to a logic flaw, and posted about it on SU Meta: Design flaw in the review audit system. I got tired of constantly (and incorrectly) failing audits and stopped reviewing in the First Posts queue for a number of years.

The review queues rely on users volunteering their time. Invalid audit failures on more than rare occasions discourage users from participating in the reviews. Without sufficient reviewers, some of the queues get backlogged, and feedback to posters, and action on posts, isn't timely.

"How close is close enough" should be considered from the perspective of the reviewers, not just the site designers. The system needs some mechanism to allow problematic audit questions and logic flaws to be reviewed.

  • My understanding is that disputed-review-audits is there to complain about a single instance, which of course is quite useless as a feedback for developers. Here (and in your other question too) you have found specific criteria which lead to unreasonable audit outcomes. I don't see why this could not be taken into account by the developers at some point. Even more exotic and mostly harmless corner cases like bounties on a presumably closed question have been dealt with in the past. I don't see why we should ignore more harmful issues. – Dmitry Grigoryev Aug 15 at 7:04
  • 1
    @DmitryGrigoryev, it looks like SO Meta still has that tag, but not SU Meta (if it ever did). But yeah, I agree that bad audit questions are fundamentally different from a design flaw in the system, which would warrant a different type of feedback. Over the years, people have suggested automated ways to clean up the audit post pool. The mechanism for reporting logic flaws (and bad audit posts) is apparently meta site posts. But if appended status tags are the indication, I don't recall ever seeing any of these kinds of posts acted on. – fixer1234 Aug 15 at 7:45
  • For one, that question about excluding bountied questions from audits has status-completed. – Dmitry Grigoryev Aug 15 at 7:49
  • 1
    @DmitryGrigoryev, I stand corrected. I've now seen one. :-) – fixer1234 Aug 15 at 7:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .