Revoke review queue badges after an audit

Question

A user with 5 review queue badges recently got a 7 day review ban after hitting too many audits

Shouldn't his stats/badges take a hit¹ in such cases? Even better, keep a warning on the queue (or at least on the first failed audit message) that "badges/stats will be revoked if you hit too many audits"?

Also, we may need to do something about his reviews, re-review them (I have no clue how we can manage that, especially once we have more users who fail the audits).

(While this may seem targeted to a particular user, I'm looking for a general mechanism to deal with those who fail the honeypot audits)

^{1. Either revoke the badges and reset stats, or revoke some badges and reduce the stats, both work.}

Answer 1

I think this would only result in more pain. We'd basically be telling people:

Because you failed x out of n audits, every review you've ever made is bad

That could conceivably be the case in some instances, but weigh the hassle of reviewing every single action the user took against the hassle of some posts that were inaccurately reviewed. It seems like such an egregious waste of time. What reviews should they retain credit for in order to keep progressing toward the badge?

Remember, the point of suspending a user (from certain areas of the site, or the site entirely) is to give them the opportunity to improve and then welcome them back when they're able to contribute positively.

I tend to agree that someone who is permanently barred from reviewing most likely didn't really earn the badges it rewards. However, removing the badge is getting a little to close to punishing a user instead of simply preventing them from repeating problematic behavior. Even lengthy suspensions aren't designed to punish you, they're designed to keep you from punishing everyone else. We've managed to retain a model of rewarding the behavior that we want, and I'm not in favor of stepping back from that.

On the other hand ....

This is becoming an extremely large source of contention, and I realize that it isn't fair to the people that work really hard to earn the badge honestly. I'm a moderator, I typically take action on between 100 - 150 extremely smelly posts every day. There's nothing glorious about it. Properly reviewing posts can be extremely mentally taxing.

Instead of yanking badges, let's make them harder only for people gaming the system to earn. Add a criteria that you must complete the tasks while passing 95% of the audits presented to you. If you pass 1k reviews and failed more than 5% of the audits, you've just put the badge permanently out of reach. In other words, nobody did this to you, you did it to yourself.

Doing this, we do more to prevent the problem, which I believe is more in keeping with our philosophy.

Answer 2

First off, we shouldn't worry about revoking badges. Normal badges have never been automatically revoked before, and starting now will only lead to the landslide of "all badges should be revoked if these ones are" complaints. I think reducing their stats would make sense, but only if there was a logical way of determining which reviews were actually compromised, which I don't believe is possible since the review system can have a bit of an opinionated view sometimes.

As far as re-reviewing a user's actions that took place, we have to think of what we're accomplishing by doing that. Each post gets reviewed by multiple people (different amounts depending on where you're at), and only one of those reviewers was determined to have failed the audit. That leaves several other reviews that are potentially* valid for each post. By saying that their actions need to be reviewed again, you're asking another three or more people to come in and review that post.

Since (as stated before) the review panel can be opinionated, how exactly do you determine whether or not their actions were valid or invalid? I don't believe a review ban is a valid reason to invalidate all their actions; that would be unfair. There could be legitimate actions in there that other people may not agree with, and that's not really their call to make. So, you're then not really reviewing their actions so much as just reviewing the post again.

_{* Minus the ones that upvote everything but are still smart enough to pass the review audits.}

So, in summary, no further action is really needed. The temporary ban does enough - it prevents further abuse (at least temporarily) and hopefully also increases the ban period if they continue to fail audits again and again after being banned previously. What's done is done, there's nothing we can really do about it.

TL;DR: You're making things way too complicated.