29

Post that has been presented as a "known good" audit item is one I voted down and close about two weeks ago (close vote expired).

Is the algorithm to pick such items designed to allow posts having down and close votes (in particular, those cast by reviewer) or this was a bug?

I've seen many known good audits but so far these seemed to be picked from posts without down and close votes.


To avoid misunderstanding, I don't complain about failure because I had a fair chance to skip or pass this audit. Prior to acting from review I opened the post in a separate tab (as I usually do) and it made me curious, what kind audit it was because voting wise it looked quite different from items I saw before, good and bad ones. I abstained of skipping because I wanted to learn more for future reviews, and I didn't want to pick an action for known-good audit because in case of audit failure, it would make me look really dumb ("hey this guy correctly identified bad thing at first but failed to recognize it in audit, bwahaha"), so I picked to repeat my original action on the post fully knowing that it can fail.

4
  • 3
    +1 from me, if it was honestly voted close, especially if it happened from outside the review queue (dunno if anyone can tell that), then punishing reviewer for the same kind of vote is not quite right. Single downvotes on the other hand should not, in my opinion, be considered. If post has highly positive score, it means it seems to meet community standards, even if one or three community members thinks different.
    – Mołot
    Commented Oct 31, 2013 at 8:31
  • 2
    Side note - Posts you've already acted upon (voted / answered / commented) appearing as an first-post audit seems flawed and (generally) easy enough to avoid. Commented Oct 31, 2013 at 9:00
  • 1
    I usually don't ask for the reason for downvotes, but in this case I'm rather puzzled about the reason. I want to better understand what kind posts are considered good enough to teach reviewers via audits. I'd appreciate if someone would explain why this would be a bad idea.
    – gnat
    Commented Oct 31, 2013 at 9:41
  • 1
    @gnat I think we'll need Shog to answer this conclusively, but he remarked once in a comment to an answer of mine that the criteria for VLQ, FP, & LA is significantly simpler than the close/reopen audits. To that end, I'm going to guess that past close votes and downvotes are not included. Commented Oct 31, 2013 at 11:12

1 Answer 1

4

In the absence of an authoritative explanation, I'll try to reconstruct the reasoning that led to audit I observed.

I assume that ways to justify the failure for known good and bad audits when these contradict original voting of reviewer "mirror" each other.

Reviewer who fails "known bad" audit despite their prior positive voting on the item is probably supposed to reason as follows:

It doesn't matter that your original voting for this item was opposite, because there is sufficient evidence that it was wrong. Accept the audit failure, learn that your original assessment was wrong and move on.

Those failed "known good" audit despite their prior negative voting are likely expected to reason the same way, exactly as written in above statement.

This "symmetry", in turn, should probably be based on the assumption that automated selection of items for known good audits is equally reliable as that for known bad ones - note how sufficient evidence is used to justify selection against original reviewer decision in both cases.

Voting close vs leave open, as well as voting up vs down used to select audit items for both kinds of items at first sight feels close enough to assume equal reliability of the selection.

But actually, there is a bit of disbalance related to the fact that voting down requires higher reputation than voting up, suggesting that on average voters down (who feed into known-bad audits) can be expected to have stronger understanding of site quality norms.

Worth keeping in mind that voting to close questions requires even higher reputation and that voting answer down carries a rep penalty, as opposed to totally free upvoting.

Above differences in "voting weight" used to select different kinds audits are probably ignored for the sake of simplicity.


It is worth noting that per my observations, selection of known bad audits actually involves more than close and down votes. Per my recollection, known bad audits I've seen were all picked from deleted posts.

Given that in order to be deleted, post has to go through quite stringent process, this would make a much more reliable selection than that based solely on close/down votes.

Actually, if "known bad" items are selected from posts deleted by moderators / 10K users, one can even argue that these audits passed through human verification before being presented to reviewer. If this is the case, it would be very important to take into account that selection of "known good" audits doesn't even come close to anything like that.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .