Is it time to rethink how audits are chosen? (take 2)

Question

^{At first, this may look like a duplicate, but the other one was downvoted, it seems, because people disagree for my reasoning on why I failed. Please keep your votes centered around the request.}

---

I just failed an audit on a question that should be closed. Why? Because it has 6 upvotes and no close votes:

Look at the question. It's obvious it's one that the community likes, but is that a valid reason to put it as an audit? Maybe not, but because it has 6 upvotes and no close votes it should be put in the queue!? NO.

This question is obviously opinion based, and something needs to be done about it. What I've been thinking is something along the lines of a audit invalidator.

We would need to start by showing multiple people the same audit. We then compare their results, and decide to invalidate an audit if enough people fail it. Now, I know you people like numbers (and unicorns), so here is an example scenario:

Let's say we show an audit to 6 people. 5 fail, but only 1 passed. It's pretty obvious it was a bad audit. What do we do? We invalidate the audit. Ultimately, it is still counted as a review, but it works like this: The people that failed lose that "mark" in the system, while the person that "passed" has no effect on him.

I know what you're asking: What do we do with the people who "pass" a lot of bad audits? Simple. Remember that person who "passed" in the last paragraph? Yeah, that one. We show him the audit again. If he chooses what would have been a "fail" in the first one (which is now a pass), it appears he's learned his lesson. So we let him go. If he fails this (now reversed) audit, we warn him that he should read the messages, then disable the "I understand" button for a longer period of time.

Now, the second part of this proposal (showing the audit another time) is just me brainstorming. Nothing required. However, the first part (invalidating audits), is the main part of this proposal. Please don't downvote if you disagree with the second part. Please focus your votes on the main part - the first part.

Answer 1

You keep making the same mistake with these proposals: you fail an audit and assume you did so because the review audit selection process is flawed... Then you build your case for changing it on this assumption.

Now, there are most certainly flaws in the review audit selection process. But picking questions you think should be closed doesn't quite capture their essential nature; after all, if no one failed them, they'd be useless! I'll elaborate a bit more on this later, but for now let's look at your rationale for this specific question:

This question is obviously opinion based, and something needs to be done about it.

Why? Why does anything need to be done about it? A huge number of questions on SO are "opinion-based" in some fashion; that's not usually a problem. When it is a problem, closing isn't necessarily even the best option.

Let's refer to the help section for the "primarily opinion-based" close reason:

primarily opinion-based - discussions focused on diverse opinions are great, but they just don't fit our format well.

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than on facts, references, or specific expertise.

Ok; this leads off with a link to a list of specific patterns to be avoided (none of which apply to the question you saw), but we don't even have to go that far - the close reason's description itself contains a simple test that "C# Event handling comparison" fails: "answers to this question will tend to be almost entirely based on opinions".

See, this isn't Tabs vs. Spaces - there are objective differences between the two approaches he outlines. The two approaches can potentially behave differently - therefore, an answer need only outline the scenarios and behaviors in which they will differ; no opinions necessary.

Something must be done: edits

That's not to say there weren't a couple of issues with that question: the code had some formatting problems, and the title wasn't particularly descriptive. I've edited to address both; you could have done the same. However, neither problem was severe enough to warrant closing the question.

Actual problems with the close/reopen audit selection

This question actually makes for a great audit, since it's neither particularly flashy or awful. It's the sort of meat & potatoes question that you'll find all over the place on SO - and therefore, the sort of thing you should learn to recognize when reviewing questions in these queues, since folks who aren't reading carefully might inadvertently vote to close them based on a couple of keywords.

Unfortunately, the current audit selection criteria don't guarantee these sorts of questions. You can also end up with stuff like this or this, topics which somewhat niche, are reasonably popular in their niche but are also off-topic as-asked. We've made a few tweaks to minimize this, but haven't prevented it entirely.