20

From time to time I fail a review audit. That's another story, but after not having run in to an audit (and certainly not failing one) for over a week, and suddenly failing one, I got banned from reviewing for two days.

Is the limit really that low? And should it be? Most audits are fair and those are rarely failed, and some are plain stupid and are easy to miss.

I'm pretty certain a fair amount of people fail audits from time to time, even reviewers that really contribute and put an effort into reviewing. Is the correct approach to ban those users for two days after failing a single audit? If there's really need for that, why not raise the rep-limit for reviewing instead of creating audits that confuse, are plain out bad and easy to slip?

This is a feature request to raise the allowed amount of failed audits per week, alternatively raising the rep limit for reviewing.

It is also a support question asking about how the review audit system works per today – what is the failed audit limit, does it ever reset?

Improve this question
2
  • 2
    Oh... And you're getting WAAAAY too many audits. Gonna find out why that is.
    – Shog9
    Commented Feb 18, 2013 at 23:55
  • I'm a frequent reviewer, and a fast one as well. Gotta work on that.
    – Emil
    Commented Feb 19, 2013 at 0:03

1 Answer 1

Reset to default
27

Short answer: you're unlucky.

I agree that the past few audits you've encountered have been borderline. The first one which we discussed was (if memory serves) an answer to a closed question - such answers are no longer used as audits. The second probably shouldn't have been deleted, at least not for the reason that it was, but I'm not sure we can alter the criteria to avoid such posts - ultimately, it comes down to improving the feedback loop for moderators. If it makes you feel any better, I think this will eventually improve not just reviewing but moderation as well.

I've lifted the quarantine.

As for your feature request: the automatic review-quarantine takes both the number of failures and the time spent on reviews into account. You don't have a particularly high failure rate, but you do complete review audits very quickly, averaging under 4 seconds each for First Posts audits. In general, time-to-review has been a pretty good heuristic; that said, I'm open to the possibility that it not be properly balanced against overall success rate.

Update

Ok - so the logic is supposed to give you an audit every 20 reviews or so; it looks at reviews in any queue though. So you're getting a lot of audits in LQ because you're doing a lot of reviews in Suggested Edits. That's... Kinda bogus. We're tweaking the logic here: 5% chance of being audited starting 5 reviews after your last audit. You probably won't see any fewer audits, but you should see more of them in the queues you're actually most active in. And your chances of being quarantined will depend more heavily on past behavior in the queue you're being audited in, so that should help too.

Still TODO: make this more heavily dependent on your recent failure rate.

Improve this answer
7
  • 2
    Good that it takes time taken for reviews into account. Would be interesting to see the data held in this regard for your own account. When you see the average time spent on reviews, it may give you a wake up call :) Wondering what mine is now :O
    – Ren
    Commented Feb 18, 2013 at 16:50
  • 2
    I kinda suspect that'd encourage the wrong behavior, @Ren. That said, you're at ~100 seconds for First Posts and ~72 seconds for Late Answers over the past month.
    – Shog9
    Commented Feb 18, 2013 at 16:57
  • I suppose people would try to stay above a certain threshold and game the system if it were introduced yeah. Thanks for the times! Curiosity sated :)
    – Ren
    Commented Feb 18, 2013 at 17:03
  • 5
    ...or worse, try to compete for minimum average time and fail to pay attention to what they're reviewing - something already happening in too many cases.
    – Shog9
    Commented Feb 18, 2013 at 17:05
  • Never thought on that but sounds about right unfortunately.
    – Ren
    Commented Feb 18, 2013 at 17:06
  • 2
    Thank you, @Shog9! Finally some real feedback and information on the audit system. In the future, it would probably be best if audits were hand picked by moderators to make sure they aren't bad or borderline. The second part I was not aware of. That's pretty interesting to see statistic-wise. That's probably something I'll have to work on personally.
    – Emil
    Commented Feb 19, 2013 at 0:06
  • Same here with a 2-day ban after 1 failure (as of today). Why do people do a lot of "Suggested Edits" reviews? Because this is simpler to review than, say Low Quality answers or First Posts. And also the number of SE is usually higher than the others, so when someone clicks on Review Queues, selects the first item from list. I admit, that my ban was reasonable. Yes, people are lazy and human factor matters, as one doesn't feel big responsibility for failure. Ban hummer is here for that.
    – Aleksey Potapov
    Commented Jun 19, 2020 at 8:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .