29

We've identified a problem on SO with "known good" audits:

  1. Question is asked and quickly upvoted
  2. Question is selected as a "known good" audit candidate
  3. Question is closed (often downvoted below the threshold)
  4. The question becomes ineligible for selection as an audit, but caching means it's not removed from the pool yet; it gets selected as an audit again in between
  5. Reviewers (seeing that the question is closed) select "Close" and fail the audit

Per the comments, the same thing also happens for reopen review audits: an open, upvoted question gets selected as a known-good audit that should be reopened, but it gets closed and thus becomes ineligible for appearing as an audit there, but the question still gets selected because the cache hasn't updated yet.

There are two reports on MSO:

The simple thing here would be to perform a check when a question is closed and immediately remove it from the audit pool. Eventually the system's cache does catch up, but these are frustrating to reviewers, and understandably so.

Improve this question

2 Answers 2

Reset to default
12

I dug into this and implemented a fix that I think should resolve the issue. Basically I added in a check when a question is closed: is this question a current audit and is it a "known good post"? If so, invalidate it. That way, any status change like that will remove that type of potentially ambiguous audit from being used. It should be live today or tomorrow, so just let us know if you see any issues like this from next week on. (Going to go resolve the other similar reports as well.) Thank you for bringing this up!

Improve this answer
10

I'd go further: any action that would invalidate the audit should remove it from the audit pool, whether that's a downvote or a close vote.

People often attempt to fix bad audits by opening the post outside the queue and casting such a vote. Sometimes, they may skip the audit rather than completing it (especially given unpredictable results like those, or these). In such cases, the audit is handed to some other unlucky reviewer, who may be tripped up by it in the way that the question describes.

Improve this answer
4
  • I think the close vote is key here, a downvote could mean anything.
    – bad_coder
    Oct 16, 2021 at 2:47
  • 1
    @bad_coder While that's true, I think as long as "no downvotes" is part of the known-good audit selection criteria (as I understand it to be for the Close queue), then it should be consistent about whether the criteria are applied to existing audits, rather than merely preventing new ones being created. Though I agree that at a minimum, close votes should definitely invalidate any existing known-good audits (across all queues).
    – Ryan M
    Oct 16, 2021 at 2:50
  • I don't know about the DV for audit selection, but I'd elaborate a good post might have a DV or CV pending initial interactions in the comments and some editing. Posts sometimes are not well received in the first few moments until an SME takes interest, saying so in the comments, and starts interacting with the OP. I'd say the CV should invalidate only if cast after the post becomes an audit.
    – bad_coder
    Oct 16, 2021 at 2:53
  • @bad_coder the point of the audits is to show something that is clearly good in order to test if the reviewer recognises that. However, the problem is that the system cannot actually know what is good or not (or we wouldn't need users voting on stuff). So, picking something that is unanimously upvoted is usually a safe bet to assume as "good". Something that is +100/-1 can be ambiguous because it's no longer unanimously upvoted.
    – VLAZ
    Oct 18, 2021 at 7:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .