What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

A user with 5 review queue badges recently got a 7 day review ban after hitting too many audits

Shouldn't his stats/badges take a hit1 in such cases? Even better, keep a warning on the queue (or at least on the first failed audit message) that "badges/stats will be revoked if you hit too many audits"?

Also, we may need to do something about his reviews, re-review them (I have no clue how we can manage that, especially once we have more users who fail the audits).

(While this may seem targeted to a particular user, I'm looking for a general mechanism to deal with those who fail the honeypot audits)

1. Either revoke the badges and reset stats, or revoke some badges and reduce the stats, both work.

share|improve this question
2  
Definitely not. Once the badges are obtained I see no reason to give people a reason to go back and try to reobtain them (can't stop them but there's no point encouraging them). This would only work if someone was perma-banned. –  ben is uǝq backwards Dec 14 '12 at 7:25
14  
@Ben: Yes, please just perma-ban these people. We don't need them anymore; never did. –  BoltClock's a Unicorn Dec 14 '12 at 7:26
    
I advocated both badge revocation and perma-banning in a previous comment so I'm definitely in favour @BoltClock'saUnicorn. –  ben is uǝq backwards Dec 14 '12 at 7:28
    
@JeremyBanks - I linked to question instead of answer. What's problem? –  hims056 Dec 14 '12 at 7:30
    
@hims056: I've rolled it back and edited it again, to show you a more convenient way of linking to a question when you have the expanded answer link. –  BoltClock's a Unicorn Dec 14 '12 at 7:31
13  
7th review queue "suspected items from suspended reviewers" hm. I for one would be glad to take a look, even if (especially if) there were no badges for this kind reviews –  gnat Dec 14 '12 at 7:34
2  
@gnat: But in your heart you'd have a "Righteous Indignation" badge for each and every item you reviewed from that queue. –  McCannot Dec 14 '12 at 15:51
    
This issue just isn't going away. Why don't we just not have the badge? It's not worth it. Most of the review queues are always empty. The time for it is passed. –  AndrewC Jan 31 '13 at 9:50
    
@AndrewC: Well, this never was meant to be a one-time thing to clear the queues. The goal is to make the community as self-moderating as possible. (And we've had a goodly increase in community moderation on smaller sites once the queues came out -- these sites never really had that many posts in the queue in the first place). Removing the badge may just make less people review :/ –  Manishearth Jan 31 '13 at 12:52
    
@Manishearth Always empty is good, I agree. If they fill up again, I'd be wrong, I admit, but I don't think they would. They're not just empty, there's a bunch of people yearning for stuff to turn up there. There have been already a few questions just this month which boil down to "Eeeek, I was promised I could review 5 items but by the time I got there there were only 3. Ungf!" On StackOverflow there are over 20,000 users who can review and over 2500 already have a silver badge for it. (Sorry to be SO-centric, but I do feel there is an issue on SO with the quality of some reviewers.) –  AndrewC Jan 31 '13 at 16:49
    
@AndrewC: I know, and I believe that is being worked upon. While there are a lot of false reviews going on, there are also a number of good reviews which outnumber the false ones. It is doing more good than harm, from what I see. It is doing no harm on smaller sites. –  Manishearth Jan 31 '13 at 18:33

2 Answers 2

up vote 25 down vote accepted

I think this would only result in more pain. We'd basically be telling people:

Because you failed x out of n audits, every review you've ever made is bad

That could conceivably be the case in some instances, but weigh the hassle of reviewing every single action the user took against the hassle of some posts that were inaccurately reviewed. It seems like such an egregious waste of time. What reviews should they retain credit for in order to keep progressing toward the badge?

Remember, the point of suspending a user (from certain areas of the site, or the site entirely) is to give them the opportunity to improve and then welcome them back when they're able to contribute positively.

I tend to agree that someone who is permanently barred from reviewing most likely didn't really earn the badges it rewards. However, removing the badge is getting a little to close to punishing a user instead of simply preventing them from repeating problematic behavior. Even lengthy suspensions aren't designed to punish you, they're designed to keep you from punishing everyone else. We've managed to retain a model of rewarding the behavior that we want, and I'm not in favor of stepping back from that.

On the other hand ....

This is becoming an extremely large source of contention, and I realize that it isn't fair to the people that work really hard to earn the badge honestly. I'm a moderator, I typically take action on between 100 - 150 extremely smelly posts every day. There's nothing glorious about it. Properly reviewing posts can be extremely mentally taxing.

Instead of yanking badges, let's make them harder only for people gaming the system to earn. Add a criteria that you must complete the tasks while passing 95% of the audits presented to you. If you pass 1k reviews and failed more than 5% of the audits, you've just put the badge permanently out of reach. In other words, nobody did this to you, you did it to yourself.

Doing this, we do more to prevent the problem, which I believe is more in keeping with our philosophy.

share|improve this answer
    
What is the incidence of audits when compared to regular reviews? 95% could be too low... otherwise I like the idea. –  ben is uǝq backwards Dec 14 '12 at 8:45
    
@Ben I'm not quite sure to be honest, 95% is more or less a placeholder suggestion. I'm not sure if we have enough data from the audits yet to find a reasonable cut off. Si Shog Ng Siyam (Shog Of Nine) might be able to chime in more though. –  Tim Post Dec 14 '12 at 8:54
    
Shog naw/neun/nueve/neuf/nine (I'm out) might be able to indeed; if we're going down the non-English route... (that was really random, I like it!) –  ben is uǝq backwards Dec 14 '12 at 8:58
2  
"let's make them harder" - well besides suspension (7 days first time, more and more for next proven fraud), I think the next best thing in making this would be increasing delay for review action buttons. As a reviewer I wouldn't mind "just" staring at the post for 10-20 (heck even maybe 30) seconds pondering about what to do with it, especially knowing that these same seconds feel like an eternity for some robo-approver eagerly waiting to submit next brainless click –  gnat Dec 14 '12 at 9:56
3  
@gnat That's also a splendid idea, slow them down increasingly more with each failed audit until their accuracy increases appreciably. That could be the last set of guard rails before the badge becomes unattainable due to chronic audit failures. –  Tim Post Dec 14 '12 at 10:54
    
@TinyTimPost exactly so. By the way this makes idea of revenge badge revocation not too interesting to me. If we manage to make abuse hard enough for future robo-approvers (the sooner the better) I wouldn't worry too much about retroactively punishing those who leaked through before –  gnat Dec 14 '12 at 11:40
1  
This makes mucho sense. I'd like to make an addition: Whenever a user hits an audit and fails, I believe that they are shown a message. If so, prominently display something to the effect of "If you do this too many times, no badges for you!" (aside from mentioning it in the badge text somehow). But we should not display the failed audit percentage for a user, keep it hidden like flag weight is . –  Manishearth Dec 14 '12 at 13:24
    
@Manishearth I like the idea of more sooner than later informing someone that they aren't going to get a badge by going through the motions, but review is a rather subjective area. We could show it like a 'bridge closed' sign when they're near the edge, but if they got that far, they did so by not reading and paying attention to what they were doing. I'd me more in favor of 'Review may not be for you (yet)' as a message a bit earlier in the process, but not initially. I'm not sure it would be effective at all, to be honest. –  Tim Post Dec 14 '12 at 17:34
1  
@Manishearth Supplementary, some users not reading is one root of the problems that SO faces. I've said before that some users Don't read, don't care, or don't read and don't care. I firmly believe that this holds true in the cases of robot reviewers. –  Tim Post Dec 14 '12 at 18:12

First off, we shouldn't worry about revoking badges. Normal badges have never been automatically revoked before, and starting now will only lead to the landslide of "all badges should be revoked if these ones are" complaints. I think reducing their stats would make sense, but only if there was a logical way of determining which reviews were actually compromised, which I don't believe is possible since the review system can have a bit of an opinionated view sometimes.

As far as re-reviewing a user's actions that took place, we have to think of what we're accomplishing by doing that. Each post gets reviewed by multiple people (different amounts depending on where you're at), and only one of those reviewers was determined to have failed the audit. That leaves several other reviews that are potentially* valid for each post. By saying that their actions need to be reviewed again, you're asking another three or more people to come in and review that post.

Since (as stated before) the review panel can be opinionated, how exactly do you determine whether or not their actions were valid or invalid? I don't believe a review ban is a valid reason to invalidate all their actions; that would be unfair. There could be legitimate actions in there that other people may not agree with, and that's not really their call to make. So, you're then not really reviewing their actions so much as just reviewing the post again.

* Minus the ones that upvote everything but are still smart enough to pass the review audits.

So, in summary, no further action is really needed. The temporary ban does enough - it prevents further abuse (at least temporarily) and hopefully also increases the ban period if they continue to fail audits again and again after being banned previously. What's done is done, there's nothing we can really do about it.

TL;DR: You're making things way too complicated.

share|improve this answer
1  
Really nice and complicated image. :) –  hims056 Dec 14 '12 at 7:50
1  
9gag? You monster. –  BoltClock's a Unicorn Dec 14 '12 at 7:53
1  
The image is stolen from the Tavern. I looked and was like "oh, that's relevant." –  animuson Dec 14 '12 at 7:53
    
OK, then the guy who posted that is a monster. –  BoltClock's a Unicorn Dec 14 '12 at 7:57
    
"Normal badges have never been automatically revoked before" - that's not correct. Tag badges get revoked if you fall below the threshold. Could be the same for reviews - fail too many audits, lose X% of your "audit score"; if below the threshold, badge gets rescinded. (Or something like that.) –  Mat Dec 14 '12 at 8:34
    
@Mat: Tag badges are not "normal" badges. They have a completely separate tab. –  animuson Dec 14 '12 at 8:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .