We've identified a problem on SO with "known good" audits:
- Question is asked and quickly upvoted
- Question is selected as a "known good" audit candidate
- Question is closed (often downvoted below the threshold)
- The question becomes ineligible for selection as an audit, but caching means it's not removed from the pool yet; it gets selected as an audit again in between
- Reviewers (seeing that the question is closed) select "Close" and fail the audit
Per the comments, the same thing also happens for reopen review audits: an open, upvoted question gets selected as a known-good audit that should be reopened, but it gets closed and thus becomes ineligible for appearing as an audit there, but the question still gets selected because the cache hasn't updated yet.
There are two reports on MSO:
- Highly upvoted question selected for audit gets closed. Selecting close for audit results in failure (marked status-review)
- Failed a review audit by voting to close a question that is already closed
The simple thing here would be to perform a check when a question is closed and immediately remove it from the audit pool. Eventually the system's cache does catch up, but these are frustrating to reviewers, and understandably so.