Number of failed audits per number of reviews

Question

Reviewers with 3 out of 3 failed audits who reviewed a dozen of posts in total are treated the same as reviewers with 3 out of 30 failed audits who reviewed thousands of posts. And while automatically generated audits for suggested edits are trivial to notice and are merely annoying, the automatic choice of questions for close/reopen audits is sometimes controversial; some reasons for closing are very subjective to begin with; and there're cases of weird inconsistent behavior punishing curiosity. Futhermore, all reviews are voted, so single mistakes are almost always filtered.

The number of successful audits in a period of time, the total number of reviews and maybe other metrics should be consereed when auto-banning users from reviewing.

Active users who flag a lot of posts receive more flags to use (1 flag per 10 successful flaggings, iirc). Active reviewers should be treated similarly. With almost 100 000 posts in the close vote queue, punishing active reviewers does more harm than good, I think.

Answer 1

I completely agree, since I failed some audits the other day (one legitimately) and haven't been able go through the review queues for two days. After thousands of successful edits/closures and many passed audits, I think a sliding scale would work better. Just an audit every now and then to keep you on your toes.

One thing they are right about:

You have failed too many recent review audits – looks like you might need a break.

Answer 2

Reviewers with 3 out of 3 failed audits who reviewed a dozen of posts in total are treated the same as reviewers with 3 out of 30 failed audits who reviewed thousands of posts.

No, they're not. Not necessarily...

What happens if I fail?

For most, nothing. You'll be given some guidance on why the post was inappropriate and what you could have done instead, and then allowed to continue reviewing.

Repeated failures, especially in short succession can lead to your review privileges being suspended for a week or longer, depending on the frequency of the failures.

I've been a bit cagey about the exact criteria the system uses to instigate automatic review-bans, but it's a bit more interesting than "fail three audits". Some folks fail dozens of audits without ever seeing an automatic ban, while others get banned relatively quickly. Of course, bans instituted by moderators can be enacted after zero failed audits if they notice you're really just not paying attention...

I'm marking this status-completed, because the system in place is already more nuanced than what you suppose it is, and in some ways more cautious even than the one you're suggesting replace it. That said, there are some additional changes we could make that would make audits less annoying for very active reviewers.

Stop, look and listen

The truth is, not very many people get banned from review, and most who do aren't banned for very long. The message you see upon being banned suggests that perhaps you need a break, and that really is the intent - to give someone who has perhaps been pushing themselves too hard and gotten careless a strong encouragement to step away and do something else for a while.

Indeed, the primary purpose of audits is education not punishment. It's easy to make mistakes, especially after you've been doing a fairly monotonous task for a good while - sometimes, it's helpful to have a sanity check. I fail audits occasionally myself, and always take it as a sign that I've perhaps gotten in too much of a hurry and am in danger of becoming careless; the audit (and countdown timer that goes with it) is a welcome reminder to look up and take a breath before going any further.

Answer 3

Yeah, I agree that doesn't make sense. While I haven't been suspended from the review queue, I could easily be for a couple more bad audits. It makes sense that as more reviews you do, the number of failed audits should also grow, albeit logarithmically, since you'll get better at recognizing bad ones.

However, the number of ones that you mis-click or accidentally approve/reject won't really change that much, so a metric that changes as a percentage of reviews would be better.

Perhaps something like:

For users with under 5, audits, no suspension (it wouldn't make sense for someone who had 2 audits, and a 50% pass rate to be suspended

For users with a bigger number of audits, perhaps a suspension after x failures in the last 100 audits, or something like that?