3

Badges should encourage good behaviour. But what when a badge encourages more bad behaviour than good?

We hear it all the time - robo reviewers, badge-hunters. The review system fights with that users that only want badges and not want to help.

There are many suggestions how to find such robo-reviewers or how to make it difficult to robo-review. But why not cut them off at the source? Don't make them want to review for badges at all.

Remove the Reviewer and Steward badges! The Custodian Bronze badge should suffice to bring a user to the review system. And we really only want users doing reviews who want to help and not doing it for the badges.

  • 4
    With this logic the copy editor and electorate badges should also be removed - I don't think removing the badges is the right way to prevent the behavior... – Lix Jun 9 '13 at 11:06
  • 6
    Also, as mentioned in the last podcast, some people might just be trying to empty the review queue - not necessarily badge hunting... – Lix Jun 9 '13 at 11:10
  • @Lix: I doubt that many users want to help emptying the queue and do it with robo-reviews. – juergen d Jun 9 '13 at 11:12
  • @Lix: If you think there is a lot of abuse because of the Copy Editor badge I am totally supporting removing it. – juergen d Jun 9 '13 at 11:15
  • 22
    Maybe passing audits should be part of the higher badges. Fail 2 audits and your progress towards the silver/gold is reset, or something similar. – Bart Jun 9 '13 at 11:18
  • Down with "Informed"! – jonsca Jun 9 '13 at 11:30
  • @jonsca Why remove "Informed"? It does no harm. – ˈjuː.zɚ79365 Jun 9 '13 at 12:15
  • @user79365 Tongue firmly in cheek! – jonsca Jun 9 '13 at 12:16
  • 1
    Gut feeling: that leaderboard is a hell of a lot more motivating than the badges. You can get your picture on it! And no, I don't think we should get rid of the leaderboard. – Shog9 Jun 10 '13 at 3:13
  • @Shog9 I don't see a leaderboard mentioned here, are you referring to stackexchange.com/leagues? That is fine since reputation is gained in an acceptable manner (I'd certainly not propose reviewing votes), but the badges opposed to here are most certainly detrimental since they encourage grinding through the review queues as fast as possible no matter how incorrect, while those who take the time to make sure their judgement is correct more often than not end up with a "This (crappy) edit has already been approved, please visit it to edit" - which btw fails to count toward the badges. – Tobias Kienzler Jul 5 '13 at 7:48
  • Odd that this request has -2 votes while the top answer at meta.stackexchange.com/questions/149621/… has +180. I wish I could give +180 to this request. Removing the incentives means a higher ratio of people that like to do it : people that are just after the incentives. My comment on that answer was... – Jason C Aug 14 '13 at 2:09
  • 2
    This is exactly how I feel... Obviously giving incentives for reviews encourages people to review based on desire for the incentives rather than desire to help the community. That's kind of... "duh". I really enjoy reviewing suggested edits and low quality posts and would certainly go through even if there were zero incentives (and I mean zero, not even a privately visible count). Given that theres more reviewers than content, its not like SO is short on reviewers. I think there are enough people that like it to do it. – Jason C Aug 14 '13 at 2:11
8

I agree with Bart's comment here

Maybe passing audits should be part of the higher badges. Fail 2 audits and your progress towards the silver/gold is reset, or something similar. – Bart 1 hour ago

But I would go further and change the Reviewer/Steward Badges to something like:

Completed at least 250/1,000 constructive* review tasks. (*No Action Needed Not Included) This badge is awarded once per review type.

As mentioned in other posts on this problem the robo-reviewers tend to just hit the no action needed button to try and fly through as quickly as possible.

My understanding of badges, is that they are used to encourage behaviors that benefit the community.

When does "No Action Needed" really benefit the community?
Note: I'm not counting not creating additional problems as a benefit...

  • If a post is good up-vote it.
  • If a post is bad: down-vote, edit, leave a comment, and or flag.
  • If you're not sure: click Skip (Not no action needed)

To better illustrate my point a gratuitous freehand circle- enter image description here

I know there are cases when there doesn't seem to be any problem with a post, but you don't feel it warrants an up-vote, hence the No Action Needed button, but should we be handing out badges to people who just hit the button 250 or 1,000 times?

enter image description here

In response to those who worry about the robo-reviewers changing tactics and up-voting instead, why not set up an automated flag to trip whenever someone uses the same review response X number of times in a row.

As in:

  • Up-voted in review X times in a row: Flag
  • Down-voted in review X times in a row: Flag
  • No Action Needed in review X times in a row: Flag
  • And so on...

Of course we wouldn't want to auto-ban people who tripped a flag in this way, there is an outside chance that someone really did run into 5 or 10 posts of the same quality in a row, but as its unlikely, it would be an easy way to spot troublesome review patterns and bring them to a moderator's attention.

If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing?

  • 8
    You'd be promoting upvotes when the reviewer is not sure they are correct. Bad upvotes are bad. – John Dvorak Jun 9 '13 at 19:19
  • 1
    @JanDvorak I'm not trying to say that people should up-vote bad answers, I'm saying that they shouldn't get credit towards a badge for hitting the "No Action Needed" button a few hundred times. – apaul Jun 9 '13 at 20:35
4

Instead of removing these badges (which I am adamantly opposed to), let's do two things:

  • Move the goalposts a bit using the audit system, and
  • Actually punish those that do a poor job of reviewing posts.

I too like Bart's suggestion in that failing two reviews per review queue should be grounds for resetting progress, but resetting progress isn't exactly enough in my eyes.

If you really want to curb bad behavior and poor edits, use the number of times your progress has been reset towards badges to restrict or remove the user from the reviewer pool.

There's a catch: This doesn't really cover poorly done revisions to a post. That's difficult to encapsulate, honestly - what I might consider a poor revision another person might not, and that's happened plenty of times in the past.

At this point in time, I don't have a firm answer to that particular problem - ensuring that a post's content has been revised as quality is a tough thing to measure. Perhaps we could use the number of declined/rejected edits as some sort of measure.

(Which, again, does nothing for the 2K+ editors who do poor edits...)

3

I don't necessarily think this is a bad idea but, speaking particularly to the close vote queue, there's so little incentive to work on it as it is. We shouldn't remove incentive without adding some incentive. You know, besides gamification - the general warm feeling that work got done, the site is now a better place, etc., those things besides fake internet points that motivate people to do stuff.

So, I would not support your proposal without addressing the lack of good incentive to work on these queues.

  • I'd so much love to help with the close queue, but I can't. And Allow 1k users to review close votes, but count them only as 0.25 actual close votes is not exactly appreciated... – Tobias Kienzler Jul 4 '13 at 11:16
  • 1
    @TobiasKienzler that means it takes 20 votes from a 1k-3k reviewer to close. I sincerely hope our site offers better ways for 1k-3k users to use their time than letting them hold a convention to close a question. – djechlin Jul 4 '13 at 14:26
  • At least at the time I am in a reviewing mood, I find less than 10 posts that require a late answer / first post / suggested edit / low quality review, while the close queue has currently 57.8k entries on which I'd be willing to spend some time on. And maybe I'm not the only one annoyed by the thousands of duplicates that seem to never get closed. 20 was just a made up number, but since the rep for closing should really not be too low this is intended as a compromise. Something like 3 x 3kers + 8 x 1kers would also work by this logic. Not all conventions are bad after all... – Tobias Kienzler Jul 4 '13 at 14:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .