What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

From time to time I fail a review audit. That's another story, but after not having run in to an audit (and certainly not failing one) for over a week, and suddenly failing one, I got banned from reviewing for two days.

Is the limit really that low? And should it be? Most audits are fair and those are rarely failed, and some are plain stupid and are easy to miss.

I'm pretty certain a fair amount of people fail audits from time to time, even reviewers that really contribute and put an effort into reviewing. Is the correct approach to ban those users for two days after failing a single audit? If there's really need for that, why not raise the rep-limit for reviewing instead of creating audits that confuse, are plain out bad and easy to slip?

This is a feature request to raise the allowed amount of failed audits per week, alternatively raising the rep limit for reviewing.

It is also a support question asking about how the review audit system works per today – what is the failed audit limit, does it ever reset?

share|improve this question
2  
Oh... And you're getting WAAAAY too many audits. Gonna find out why that is. –  Shog9 Feb 18 '13 at 23:55
    
I'm a frequent reviewer, and a fast one as well. Gotta work on that. –  Emil Feb 19 '13 at 0:03

1 Answer 1

up vote 23 down vote accepted

Short answer: you're unlucky.

I agree that the past few audits you've encountered have been borderline. The first one which we discussed was (if memory serves) an answer to a closed question - such answers are no longer used as audits. The second probably shouldn't have been deleted, at least not for the reason that it was, but I'm not sure we can alter the criteria to avoid such posts - ultimately, it comes down to improving the feedback loop for moderators. If it makes you feel any better, I think this will eventually improve not just reviewing but moderation as well.

I've lifted the quarantine.

As for your feature request: the automatic review-quarantine takes both the number of failures and the time spent on reviews into account. You don't have a particularly high failure rate, but you do complete review audits very quickly, averaging under 4 seconds each for First Posts audits. In general, time-to-review has been a pretty good heuristic; that said, I'm open to the possibility that it not be properly balanced against overall success rate.

Update

Ok - so the logic is supposed to give you an audit every 20 reviews or so; it looks at reviews in any queue though. So you're getting a lot of audits in LQ because you're doing a lot of reviews in Suggested Edits. That's... Kinda bogus. We're tweaking the logic here: 5% chance of being audited starting 5 reviews after your last audit. You probably won't see any fewer audits, but you should see more of them in the queues you're actually most active in. And your chances of being quarantined will depend more heavily on past behavior in the queue you're being audited in, so that should help too.

Still TODO: make this more heavily dependent on your recent failure rate.

share|improve this answer
2  
Good that it takes time taken for reviews into account. Would be interesting to see the data held in this regard for your own account. When you see the average time spent on reviews, it may give you a wake up call :) Wondering what mine is now :O –  Ren Feb 18 '13 at 16:50
2  
I kinda suspect that'd encourage the wrong behavior, @Ren. That said, you're at ~100 seconds for First Posts and ~72 seconds for Late Answers over the past month. –  Shog9 Feb 18 '13 at 16:57
    
I suppose people would try to stay above a certain threshold and game the system if it were introduced yeah. Thanks for the times! Curiosity sated :) –  Ren Feb 18 '13 at 17:03
4  
...or worse, try to compete for minimum average time and fail to pay attention to what they're reviewing - something already happening in too many cases. –  Shog9 Feb 18 '13 at 17:05
    
Never thought on that but sounds about right unfortunately. –  Ren Feb 18 '13 at 17:06
2  
Thank you, @Shog9! Finally some real feedback and information on the audit system. In the future, it would probably be best if audits were hand picked by moderators to make sure they aren't bad or borderline. The second part I was not aware of. That's pretty interesting to see statistic-wise. That's probably something I'll have to work on personally. –  Emil Feb 19 '13 at 0:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .