What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 129 Stack Exchange communities.

I review a lot on SO, typically on my phone throughout the day when I have down time. I mentioned in my previous question, “STOP! Look and Listen” audit tricked me, that the audit system fails because it does not give me an opportunity to undo my action immediately.

Oh, I accidently clicked Up Vote when trying to Down Vote while using my phone (happens to me EVERY day)? Too bad, I failed.

So a few days ago I received this message after a few false failures:

You have failed too many recent review audits – looks like you might need a break. Come back in 2 days to continue reviewing.

The problem now though is that if I make one mistake (or the audit is a bad one) then I instantly get blocked from reviewing for 2 more days (just happened).

This seems like a problem because

  1. The audit system is already bad. It doesn't allow me to quickly undo my action.
  2. This is very unforgiving. Even for a user who makes a real mistake, they shouldn't be blocked again for making a single "mistake".
  3. Based on my high number of successful reviews and audits, the ratio that is required to be blocked doesn't seem appropriate.

EDIT: A lot of people are discussing #1 so I should bring up the previous responses from a diamond found in the comments of this answer from another question:

Waiting until all actions are performed for an audit task wastes valuable time that could be spent reviewing an "actual" review. Audit tasks are only concerned with the first impulse action taken on a post

However, it seems he backed this up with the fact that nothing was being done with the audits. At that time there was no punishment for failing. To quote the diamond again:

audit tasks are currently working in providing the right guidance to those who make an impulsive call on a post based on the data we have collected thus far. If you don't happen to select the correct action initially on an audit task - don't worry! We aren't doing anything with this data. The thing we are trying to combat at the moment is mindlessly upvoting posts in review to do nothing more than increase review stats which are harmful to review system.

But I do have to worry now.

UPDATE: I think some of the negative responses to this post are from a miscommunication. I simply requested for audit blocks to be more forgiving because a diamond rejected my request back in November to make the interface better. That request can be found in the comments of “STOP! Look and Listen” audit tricked me

share|improve this question
17  
Sorry but I disagree. It's not like you're banned for life from reviewing. There are plenty more reviewers, so no harm is done to the community. Maybe don't review from your phone? Took the team long enough to finally handle the robo reviewers, while you are innocent victim of this I think the system works just fine. –  Shadow Wizard Jan 31 '13 at 9:00
10  
@ShaWizDowArd I shouldn't be penalized for using a mobile device, that is ridiculous. The website ALLOWS for undoing votes in a short period of time. Yet audits don't? This is BAD BAD BAD design. –  Austin Henley Jan 31 '13 at 9:01
7  
Think of it as a vacation, rather than a punishment. More free time for you. :D –  ronalchn Jan 31 '13 at 9:13
2  
@ShaWizDowArd While I'm ok with the review audit system, I don't really think the [robo/bad] reviewers are handled. There are plenty of recent examples, all from a user I came across yesterday. Obviously not his fault, though. –  Xavi López Jan 31 '13 at 9:22
    
@XaviLópez you got it wrong... the audit system is meant to block the bad reviewrs, those who approve bad edits... not those suggesting those edits. When enough bad edit suggestions are rejected, it cause automatic ban of 7 days from suggesting further edits. –  Shadow Wizard Jan 31 '13 at 9:30
2  
@ShaWizDowArd Indeed, those suggestions (which were bad edits) were all approved by bad reviewers. So, not the editor's fault if their bad edit gets approved. How will they know they're bad if they keep getting approved? –  Xavi López Jan 31 '13 at 9:31
    
@XaviLópez oh, I see your point now. Well, assuming those bad reviewers will bump into audits, they will eventually get blocked at some point and take the hint. Give it some time. :) –  Shadow Wizard Jan 31 '13 at 9:33
5  
@ShaWizDowArd It is very, very easy to circumvent audits since it is based on the first action rather than your action after clicking "I'm Done". Just takes a tiny bit more time but you can pass 100% of the time if you want to cheat. –  Austin Henley Jan 31 '13 at 9:36
5  
Yep, undo should be allowed, and I don't buy "don't waste my time in audit" at all. There is so little in the review queues (apart from the close vote queue) these days that it just is not urgent that you get on with other reviews. –  AndrewC Jan 31 '13 at 9:41
    
@AndrewC Updated the question with exact quotes to give more context. –  Austin Henley Jan 31 '13 at 9:48
2  
@XaviLópez - We're talking about different reviews here. There are currently audits in place that deal with problem reviewers on the Late Answers and First Posts queue (who were a severe problem for a while there), but until recently there was not a similar mechanism for suggested edit reviews. One is currently being tested and I'd imagine we'll start to see some action taken on the worst reviewers of suggested edits soon. Certain people have been manually banned from the suggested edit review queue after obvious incidents of abuse, though. –  Brad Larson Jan 31 '13 at 15:19
    
@BradLarson great! Didn't really notice this was about other review tasks, but this was exactly my point. Glad to see it's been implemented in suggested edits, I hope it will really mitigate the harm from robo/bad reviewers. –  Xavi López Jan 31 '13 at 15:42
2  
Austin, I think people are responding more negatively to your bug report here because of the original phrasing of your title. I reworded that to point to the legitimate problem, that's it's too easy to inadvertently trip an audit failure on a mobile device. Throwing the whole audit system under the bus isn't necessary here, because from the moderation side of things it has had a tremendous positive effect on the site. Banning the worst review offenders has made it so that upvoted spam and non-answers have all but disappeared once again. Your problem is a real one, though. –  Brad Larson Jan 31 '13 at 19:09
    
@BradLarson That is only one of my points (and originally, it wasn't even my main one). My main point was that now it only takes one (false) failure for me to be banned again. Which is why I said the system was making it impossible to review. I don't care if they don't like the title. Broke is broke. –  Austin Henley Jan 31 '13 at 19:41
1  
@Jack Very valid, audits are treated differently than the normal interface (being able to undo actions). –  Austin Henley Feb 1 '13 at 17:48

3 Answers 3

First time I got one, I really didn't like how the audit jumped ahead on first click, either. This could easily be solved by making the audit behave exactly like a normal review, and give you results only after you've clicked "I'm done" or "no action needed".

share|improve this answer
3  
I 100% agree. @GeoffDalgas disagreed with this however since audits should "not waste" time. Quoted from comments found in this question –  Austin Henley Jan 31 '13 at 9:05
    
The way it is currently done, I believe, avoids problems (further design) with somebody trying to flag or edit the post. Flagging could more easily be resolved with extra logic, but allowing somebody to edit a deleted post is just pointless. –  ronalchn Jan 31 '13 at 9:06
    
@ronalchn When you click flag, you instantly pass the review. They might as well do the same for clicking the edit button. So I don't see this as an issue. –  Austin Henley Jan 31 '13 at 9:07
    
Note, my question isn't only pointing out the inability to undo actions. It is also about the requirement/forgiveness of being blocked. –  Austin Henley Jan 31 '13 at 9:09
    
@AustinHenley: Meh, it's just 2 days :/ –  Manishearth Jan 31 '13 at 9:11
1  
@Manishearth No, it is not just 2 days. It was 2 days and now another 2 days after a single false failure. 2+2!=2 :( And this is only going to continue to happen until the broken system is fixed. –  Austin Henley Jan 31 '13 at 9:12
3  
@Austin so maybe reset the failure counter after getting back from the two days break, this one is something I totally agree to. (For all we know, it might be the design and you found a bug) –  Shadow Wizard Jan 31 '13 at 9:13
    
@AustinHenley: Well, yeah -- if someone has already failed a lot of audits, their "second chance" should be much less lax (in fact, I would think that the next reviewban would be 7 days). I can't see a way of removing that without making it less effective. What they may be able to do is remove your audit history, since it's a false positive (giving you a fresh start on the audits). –  Manishearth Jan 31 '13 at 9:15
3  
@ShaWizDowArd: Hmm, resetting the failure counter doesn't make as much sense. As a mod, I suspend people after repeated offences despite warnings. After the suspension is over, even if they repeat the offence once, I suspend again --they clearly haven't learned their lesson here. Same thing applies here. Austin is a false positive here, but the system shouldn't be made less effective to take care of the 0.001% of reviewers who get badly banned. –  Manishearth Jan 31 '13 at 9:17

This is very unforgiving. Even for a user who makes a real mistake, they shouldn't be blocked again for making a single "mistake".

You'll never be blocked for making a single mistake. But if you have a long history of failing audits, that can count against you when evaluating future failures - the expectation is that you'll do exactly what the failure message states: stop, and take a bit more time to evaluate future reviews. If you're more concerned with blazing through reviews than making good decisions, you'll eventually accumulate a history that'll make it very likely you'll be blocked again.

amelvin's speeding ticket analogy is a good one: Accumulate enough tickets, and you may find your license suspended. If you can't bring yourself to spend a bit more time on each review, you'll quickly end up spending a lot more time waiting out the block.

share|improve this answer
4  
Your answer doesn't account for Austin's problem being fat fingers -- he says «Oh, I accidently clicked Up Vote when trying to Down Vote while using my phone (happens to me EVERY day)?» (emphasis mine). –  Josh Caswell Feb 10 '13 at 20:16
5  
The solution to that would be a better mobile design for review, not more forgiving audits (which are already quite forgiving). And if you're making the same mistake, repeatedly, for the same reason... You should probably go out of your way to avoid it. I switched to my laptop - screwing up one flag because the UI sucks on mobile is the UI's fault; screwing up dozens because I'm too stubborn to stop banging my head against the wall is my fault. –  Shog9 Feb 10 '13 at 20:29
1  
You switched to your laptop just because the UI on your microdevice sucks? Mark my words: someday an SO user named DickTracy will complain that the site doesn’t look good on his wristwatch. –  tchrist Feb 10 '13 at 20:41
    
@Shog9 After being unblocked, it only takes one mistake before I am blocked again. This is what I meant by single mistake. –  Austin Henley Feb 10 '13 at 21:12
    
So I was correct in saying, "they shouldn't be blocked again for making a single "mistake"." –  Austin Henley Feb 10 '13 at 21:22
2  
Whether or not you get blocked again depends on your performance in the past, @Austin. So far as I can tell, you've only failed two audits this month - but you failed five last month. Some of those still count. Seriously - just slow down a bit, take time to read the post you're reviewing. There's absolutely no reason why anyone reading the suggested edit audits should fail one. –  Shog9 Feb 10 '13 at 21:56
    
@Shog9 I disagree with "There's absolutely no reason why anyone reading the suggested edit audits should fail one." There were a few that I was 100% certain on my decision yet failed. At worst, they could be considered border line but even in hindsight I would have made the same choice. –  Austin Henley Feb 10 '13 at 23:22
2  
You've only seen six suggested edit audits, @Austin - they've all been pretty clearly nonsense, including the ones you failed. Indeed, it would be a pretty astonishing occurrence if one actually made sense, given how they're generated. –  Shog9 Feb 10 '13 at 23:33
    
@Shog9 My mistake, it must have been something else that I was thinking of. That one looks to be a legit failure on my part then. It is a shame that one audit causes me to be banned now due to the false ones in the past. –  Austin Henley Feb 10 '13 at 23:37
    
Think about it this way, @Austin: you've done 434 suggested edit reviews. If your accuracy rate (as reflected by the audits) is 67%, that's 144 reviews you've gotten wrong. Of course, it's probably not that bad in practice. But still... –  Shog9 Feb 10 '13 at 23:42
    
@Shog9 I understand, that makes sense. –  Austin Henley Feb 10 '13 at 23:43

I got my first audit kick today - it feels like a good idea, because if you get caught speeding (like I clearly did to mistake the obvious vandalism for valid spelling/grammar fixes) you're not really auditing properly.

Perhaps a more explicit driving license style points system could be adopted - with endorsements to your license more obvious (not sure if everyone should be able to see your license).

@Manishearth 's comment suggests that mods have developed an informal system of how to deal with people who ignore the speed cameras (I'm going with this metaphor now, deal with it). Should there be a faq section on how this particular section of the highway code should work?

share|improve this answer
    
"faq section" - do you mean something like this post: Review Guidelines? "a stub for a proposed faq..." –  gnat Jan 31 '13 at 12:53
    
A note: the stuff about mods in my answer was regarding offences in general. We don't directly do anything about review speeding, we just pass it on to the higher-ups. –  Manishearth Jan 31 '13 at 12:54
    
@Manishearth higher up? You mean 'The Man'? –  amelvin Jan 31 '13 at 13:22
    
@amelvin: Nah, these guys. Jeff no longer works at SE, and Joel never really took part in the day-to-day proceedings of SE. –  Manishearth Jan 31 '13 at 13:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .