38

Possible Duplicate:
Can (or should) anything be done about someone who votes “Do not Close” a lot in /Review?

I came across some users that are heavily trained to hit the Do Not Close button.

They hit this button on every question on /review/close. I guess the maximum of close votes for a day is 112, these users stop the havoc at this count.

Here is an example (you can browse others pages).

Is it possible to do something against this? Like:

  • after 5+ close votes in a row: display a message to inform the user that (s)he should not only click on Do Not Close button
  • after 15+ close votes in a row: block access to the queue for XX minutes
  • after 30+ close votes in a row: block access to the queue for XX hours
  • after 50+ close votes in a row: warn mods?
Improve this question
18
  • I see the time gap between his casting is only the time to load next question and enable Do Not Close button.
    – Himanshu
    Oct 15, 2012 at 7:38
  • @hims056 yep, from 3s to 10s on average
    – j0k
    Oct 15, 2012 at 7:39
  • 7
    Pretty interesting if you ask me. In the review queue there's the same phenomenon, users just blindly hitting "Accept". I wonder what can be done about that, because I don't have a good idea. But what should go into considerations is that the big volume of SO is a part of the problem.
    – Time Traveling Bobby
    Oct 15, 2012 at 7:53
  • 26
    Let's just make review a 20k tool on SO and be done with this nonsense already. As for rep requirements, I'll just say that mods have a far easier time dealing with users gaming rep than users gaming review badges and blighting the site in the process.
    – BoltClock's a Unicorn
    Oct 15, 2012 at 7:54
  • 2
    @BoltClock'saUnicorn, I think a limit of 5k should be enough.
    – Toon Krijthe
    Oct 15, 2012 at 8:01
  • 3
    @GamecatisToonKrijthe: User in question (in this case) is 7k.
    – Time Traveling Bobby
    Oct 15, 2012 at 8:03
  • 2
    Sorry it is monday.
    – Toon Krijthe
    Oct 15, 2012 at 8:04
  • 4
    I doubt a rep threshold will work. We already have enough 20ks misusing the delete immediately privilege.
    – Mysticial
    Oct 15, 2012 at 8:07
  • 8
    Then the alternative is to remove review altogether. It simply isn't working.
    – BoltClock's a Unicorn
    Oct 15, 2012 at 8:10
  • 5
    @Mysticial: So basically, SO reached the critical mass that peer review gets (partly) crappy, but has not yet reached the critical mass so that it gets self-correcting (à la Wikipedia)?
    – Time Traveling Bobby
    Oct 15, 2012 at 8:11
  • 10
    @SulfurizedDemonbobby Or rather, the new review system probably needs to be better designed. Currently, the badge incentive is clearly too strong for some people to resist. Especially since it gives out gold badges (which are normally ridiculously hard to get aside from luck).
    – Mysticial
    Oct 15, 2012 at 8:13
  • 22
    Whatever it is, they need to do something about this ASAP. I'm sick and tired of this. I can't even be assed to hand out warnings and suspensions to poor reviewers anymore, much less contribute any reviews myself. And of course, to avoid adding fuel to the fire, this will be my last comment on the matter.
    – BoltClock's a Unicorn
    Oct 15, 2012 at 8:20
  • The user linked to has clicked on "Close" as well as "Do not close". Not as often, but the person has done so.
    – Andrew Grimm
    Oct 15, 2012 at 10:58
  • 4
    Can we do something against serial close voters, too, please?
    – slhck
    Oct 15, 2012 at 11:34
  • 2
    @gnat he's not the first one to expose a vulnerability of the review system: meta.stackexchange.com/q/149621/182741
    – j0k
    Oct 15, 2012 at 11:55

7 Answers 7

Reset to default
32

I suggest we simply remove the badges entirely.

The badges seem to create an undesired incentive to game the system and they are probably the only such incentive.

While this may very well reduce the number of valid reviews as well, I'd guesstimate that that loss will not be that big (the most active reviewers still review, although they already have those badges).

Yes, this is a partial duplicate of Mad Scientists answer, but I thought I'd be worthy to vote on removing the badges alone separately from his other suggestion.

Improve this answer
3
  • 4
    This definitely gets my vote. Remove the badge rewards for bulk reviewing (badges for one-off behaviour are useful nudges to good behaviour).
    – itsbruce
    Oct 15, 2012 at 10:59
  • I think this is the only solution. This will take the gamer's interest away from the review system and allow those actually interested in reviewing posts to do it properly.
    – vascowhite
    Oct 15, 2012 at 13:11
  • 7
    I second this. After seeing three obvious spam answers and piles of non-answers being upvoted in the last week, not to mention the terrible edits that are being approved, I've had it with the review badge gaming. I share BoltClock's opinion that the current review system is broken, and is actually reducing the quality of content on the site.
    – Brad Larson
    Oct 15, 2012 at 14:12
26

We probably need something like the serial voting script for review abuse. I don't think rate-limiting or higher reputation thresholds are enough.

The script could take the time between reviews into account as well as a suspicious deviation from the average percentage of close/not close decisions. But that could be not enough information, requiring at least two reviewers for each item would be a way to collect more information about the accuracy of a review. Then the script could also take into account how often a review decision is overturned by two other users.

The script could then roll back all review decisions of that user and block him for some time from reviewing.

Additionally, I think we should just scrap the badges for reviewing. They seem to cause more trouble than they're worth. This might even be enough alone, saving the effort of implementing other solutions.

Improve this answer
6
  • 1
    I'm not sure this is gonna work too well. I do find myself (legitimately) closing 10-20 consecutive questions from the queues. Though I do skip the majority of the ones that I'm presented. So maybe it can take into account for that.
    – Mysticial
    Oct 15, 2012 at 8:33
  • 2
    @Mysticial Pressing "not sure" is a pretty good indicator that you're taking the review seriously, so that should certainly be taken into account. I think having multiple reviews is likely necessary to get such a script to work, else there is not a lot of data to use.
    – Mad Scientist - on strike
    Oct 15, 2012 at 8:35
  • 2
    Now that I think about it again, yes such a script will probably work with enough fine-tuning. And to keep it harder to game, the reversals should be silent. +1
    – Mysticial
    Oct 15, 2012 at 8:43
  • 2
    @Mysticial I'm ok with the silent but we might have some users ask on meta "why my close votes where reverted?", no? Might be silent + in the faq I think.
    – j0k
    Oct 15, 2012 at 9:00
  • 1
    +1 for your last paragraph.
    – vascowhite
    Oct 15, 2012 at 13:13
  • I'm in agreement with your second paragraph. Win-by-two for suggested edit approval/rejection has seemed like a good idea to me for some time, and I think requiring some kind of concord between reviewers might be useful for the review queue, as well. It would be important not to show the other actions that have already been taken, though. This couldn't completely eliminate no-effort reviews, but it would make them harder.
    – jscs
    Oct 15, 2012 at 19:07
11

Although the review system is great, it also has a great attraction to badge hunters gaming the system.

In order to minimize this behaviour, we could add a threshold to the review tools. I think this should be a combined threshold:

  • A certain amount of reputation (for example 5000) to ensure familiarity with the site and the customs.
  • A certain participation on meta (for example 10 upvoted answers or 20 upvoted questions) This to show that the reviewer has some interest in maintaining the site.
  • Probably a basic list of maintenance badges (Strunk&white and Deputy).
Improve this answer
7

Limiting allowed number of Do Not Close votes per day (eg to the same number as daily limit of close votes) would tame this kind of abuse.

It would be also helpful to introduce daily limit for approvals and rejections of suggested edits.

Review badges seem to give incentive to sufficient number of users now so that individual daily limits won't have much negative impact on handling review items.

Improve this answer
4
  • 6
    Hum I will prefer to merge votes per day for "close" & "do not close". Otherwise, you will still be able to reach 50+50.
    – j0k
    Oct 15, 2012 at 8:58
  • @j0k yeah I would prefer that, too. Though even non-merged limit is much better than none
    – gnat
    Oct 15, 2012 at 9:08
  • 1
    There is indeed a limit on the number of suggested edit reviews that one can perform per day. In fact, this limit has been extremely frustrating in the past.
    – jscs
    Oct 15, 2012 at 19:10
  • @JoshCaswell yup in the past I would not even imagine considering limit of these reviews anyhow helpful not just too high. Things have changed indeed
    – gnat
    Oct 15, 2012 at 19:21
4

Badges are incentive, and seemingly strong ones to some people. Especially if they are willing to mindlessly do what you have shown here. Why not just change the incentive?

Make these gold and silver badges more constructive.

Silver Badge suggested change

Change "Completed at least 250 review tasks. This badge is awarded once per review type" to "Completed at least 250 review tasks where at least 25% of the completed tasks were edits. This badge is awarded once per review type."

Gold Badge suggested change

Change "Completed at least 1000 review tasks. This badge is awarded once per review type" to "Completed at least 1000 review tasks where at least 25% of the completed tasks were edits. This badge is awarded once per review type."

It will be much harder to go in and constantly edit as it takes more time. Moreover, it will stand out more to be flagged for abuse.

In fact, I just tried and noticed that it is possible to edit without actually changing any content. So perhaps this change is not best, but changing the badges to be different seems like a possible route to address this issue.

Improve this answer
3
  • About your last point, check this question.
    – j0k
    Oct 15, 2012 at 8:51
  • I was about to suggest something similar to this. Though, not edits, but a more diversity of votes. To take the example from the question: Require a maximum of XXX (non-consecutive) "Looks good" votes.
    – Rob W
    Oct 15, 2012 at 9:48
  • 1
    I don't think edits are a valid criteria for all the reviews especially Close Votes - from my experience most of the ones I see should be closed not edited (albeit I have filtered on off topic)
    – mmmmmm
    Oct 15, 2012 at 11:35
1

Badges are incentive and most likely should be left alone, to draw more users to process. However, to dissuade those who want to just automatically get them, there should be some kind of automated script that tracks such patterns. Most important part would be punishment: anybody that caught serially doing the bad thing should be somehow flagged to permanently (or at least for very long time) disable either entire review functionality or stats counting and badge.

Isn't it the best way to punish someone hunting for undeserved badge than to deny it them completely?

Improve this answer
7
  • 6
    Badges which reward mindless and repetitive behaviour are not a good thing. As it stands, the only reward for responsible reviewing is a personal sense of satisfaction for a job well done; the people who care to take the time to do it properly won't care about gaining a badge, but they may well be demoralised by the sight of unprincipled idiots gaming the system and degrading the site in the process.
    – itsbruce
    Oct 15, 2012 at 11:03
  • 1
    You can't take away that badge gets more people in process. People love gaming. You may as well argue for removing all badges and reputation because "personal sense of satisfaction" is a reward for pretty much every task on SO already. That's why leaving game alone, but punishing cheaters seems the best to me.
    – Oleg V. Volkov
    Oct 15, 2012 at 11:22
  • That's a straw-man argument. I'm not arguing for the removal of badges altogether, just the ones that reward mindlessly repetitive behaviour. Punishing cheaters can be very difficult: how do you detect the behaviour, how do you distinguish malign behaviour from ignorance etc. Removing rewards for repeat behaviour where it causes an obvious problem is much simpler. People come to the site because it is useful; gamers stay because they can collect badges. This makes the site less useful; in the end you have only the gamers, as others will go elsewhere for more data, less noise.
    – itsbruce
    Oct 15, 2012 at 11:28
  • Ignorance, as in "clicking accept just for fun" is just as bad. Exclude one bad user from review, job done.
    – Oleg V. Volkov
    Oct 15, 2012 at 12:16
  • ...and the good users that remain aren't doing mindless work. They're genuinely helping site and giving them a badge is both a good reward and good incentive.
    – Oleg V. Volkov
    Oct 15, 2012 at 17:41
  • Did I miss a meeting? Is it International Strawman Day?
    – itsbruce
    Oct 15, 2012 at 20:45
  • meta.stackexchange.com/questions/149621/…
    – itsbruce
    Oct 16, 2012 at 6:49
1

Before the review badges were unleashed, the Close Review Queue was somewhat quieter - rarely did anyone manage more than 55 reviews a day. Since the badges appeared we have had a little flurry of '119' reviews per day people. (I think they actually manage 120 per day, but there is a bug in the review count display.) This initial influx of 'gamers' seems to have calmed down a bit - they've all achieved their gold badges. So maybe things will get better with no further action.

Incentivising review is aimed at reducing the backlog of items. A 'false' DNC doesn't in itself lead to more work for everyone else, but increases the chance the post will exit the queue without being closed. I think the gamers rack up 70 DNCs and 50 closes per day, so their net effect is an excess of about 20 DNCs per day, which I'd view as 'friction' in the Close Review queue; the effect is not as bad as in, for example, the Suggested Edit Queue, where only two reviews are needed before an item leaves the queue. At the moment there are only a few non-review reviewers, and their 'reviews' are fortunately the minority.

Suggestions for improvements:

  • Don't have separate limits for DNCs and closes - have one limit which all reviews count to.
  • Make it harder to choose DNC. At the moment its too easy to DNC as compared to close. This would involve having a different Close dialog box when working in the review queues as compared to the one we have when close voting outside the review queue. DNC would appear as an option alongside the other close options.
  • Increase the badge requirements, or reduce the maximum daily limit. At 120 'reviews' per day, a shiny gold badge can be had in under 10 days of random clicky-clicky.
  • Make the DNC button inactive for a period after page load - 20 seconds perhaps? This could be applied to the Close button too. Nice side affect is that some mistakes made by 'real' reviewers in a hurry might be eliminated. If the daily limit was set at 50 reviews that'd be around 20 minutes of work in the review queue allowed per day.

I'd resist complicating the workflow or automated checking of reviews - too much work, and likely to cause a whole bunch of new problems.

In the end though, if the badge incentives can't be made to align user behaviour with what's needed, then they should go.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .