Review Audits on Reopen votes is too subjective

Question

I was just asked to review the following question:

Is there a conventional way of returning error statuses from JSON web services?

I voted no, it shouldn't be reopened. It's subjective and open to opinions rather than facts. This is my personal view as a reviewer.

I was presented with:

"You didn't pass. There are no major problems with this question. You should click Reopen..."

I don't know if there is a better way of "testing" reopens, but this method strikes me as odd. The important thing for me is it puts me off reviewing as I get nervous that I'm doing it wrong when I am really trying to help.

I think maybe a set of questions should be selected by a moderator as "good review questions" because going on the number of votes a question has doesn't work. In fact the questions with the highest number of votes on this site tend to be closed as too broad or subjective!

EDIT

I think actually, having looked at the question alone, I can see the point that it is borderline on topic. But I'm only really able to come to that conclusion by seeing that the answers posted are, as a rule, sensible and factual.

Perhaps then, what it needs is for you to be able to see the question and the top voted answer when making your judgement.

Answer 1

The last sentence certainly sounds like a request for opinions:

So, is this a perfectly acceptable way of returning an error status to the AJAX caller, or is there a "proper" way of formatting a JSON error response?

At least one answerer interpreted it that way as well:

My 2 cents:

However, the OP does ask this:

Is there an accepted, universal convention (or - even - specification) for returning error states to JSON-based AJAX calls,

and the top-voted answer addresses that question directly and appears to provide a detailed analysis and relevant information.

As far as I know, requests for specifications and conventions are on-topic (if borderline). So, while a quick reading of the post may make it look like an opinion request, I don't think it really is.

That said, if you feel that it is a bad question and should not be used as an audit again, you can downvote it. A single downvote makes the question ineligble for use as an audit.

Answer 2

If this question were asking for the "best" way, then it would definitely fall into an off-topic category. But it isn't - it's asking if there is a universal way. That question can, and was, answered objectively.

This author could have written this a little better. For example, asking for documentation would have put this more solidly into the "good subjective" kind of question. (Though, in my opinion, this question only appears subjective - it is not, at its core, subjective.)

As-is, it is very similar in format and style to an off-topic question, but, in my opinion, a careful reading shows it to be acceptable.

I agree with your confusion, though, and I think it provides further evidence for why an automated review audit has flaws. That said, failing a single audit will not hurt you at all. You would need to fail several in a row in a short period of time to feel any impact. See https://meta.stackexchange.com/a/157172/191410:

Repeated failures, especially in short succession can lead to your review privileges being suspended for a week or longer, depending on the frequency of the failures. Stack Exchange is continuing to analyze the data to determine the best action to take in these cases.

---

To your second (edited) point: Questions stand alone. They are either on-topic or not. A great answer doesn't make a bad question good. Displaying the top-voted/accepted answer is irrelevant.

A better approach might be to have the option to flag the audit question as "borderline", "confusing" or "bad example" and then give the mods an opportunity to exclude it from future audits, since audits are supposed to be obvious.