What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

There's some attempted gaming of the Steward badge visible in the Review / Close Votes history on SO.

If you look at the top reviewers for the day you can see the gaming clearly: there's a small group of users who have double the number of reviews of the 'true' leaders.

Looking at the review history, I'd expect to see Close/Do Not Close (DNC) appear somewhat randomly. If most initial flags are correct — which seems likely — then there should be more Closes than DNCs in the history, rather than an equal number.

The majority of reviewers do indeed choose Close much more often than DNC. But a few have a noticeably higher proportion of DNCs. Also, some people appear to use up most of their Closes, then switch to mostly DNCs. I spot checked a few of these 'late' DNCs and many seemed unlikely to be 'correct'.

Trying to achieve badges is not a bad thing, but in this case each incorrect DNC increases the chance of a post with problems leaving the queue before it has been fixed.

Can something be done to make reviewers treat each item in the queue fairly, regardless of their badge target?

share|improve this question
Having a single shared rate limit for Closes and DNCs seems like one solution. –  martin clayton Sep 24 '12 at 6:14
On the flip-side, most of the questions in the Close Votes queue are borderline leaning towards not close-worthy. That's why they are there. If they were obvious "should be closed" questions, they would've been closed long before they found themselves in the queue. –  Mysticial Sep 24 '12 at 21:58
@Mystical - perhaps, but there are many items there that are not closed through lack of views rather than any redeeming qualities. –  martin clayton Sep 24 '12 at 22:00
@Mystical - I don't know the exact numbers for my reviews but I think that most I see should be closed, I suspect as noone else looked at them. which is also my experience from browsing questions –  Mark Sep 24 '12 at 22:06
Should I convert this into a feature-request for a single limit for votes? –  martin clayton Sep 29 '12 at 23:52

5 Answers 5

This guy must read very very quickly...

enter image description here

share|improve this answer
He does that several times a day... –  Toon Krijthe Oct 9 '12 at 13:09

Can something be done to make reviewers treat each item in the queue fairly, regardless of their badge target?

Define "fair". There are - and always have been - folks who feel that closing is inappropriate in the vast majority of cases. That's why we have voting, re-opening, and - now - a Do Not Close function.

That being said, there's a difference between disagreement and carelessness. We'll be taking a closer look at folks whose review responses are uniform and/or rapid-fire. This is all tracked, and most of it is public - so fortunately, it's much easier to notice when folks aren't reviewing in good faith.

share|improve this answer
I figured someone might pick up on that. My meaning was as in your last paragraph - the reviewer should treat the post fairly, i.e. review it, not just execute an action that has no relation to the content of the post. I appreciate that there is a range of opinions about what should be closed or not though, and think that's a good thing. –  martin clayton Oct 16 '12 at 21:23
+1. One thing I am seeing a lot of is posts I flag as spam already having been upvoted twice due to the "First Posts" queue. It's almost the norm now, actually. –  Andrew Barber Oct 16 '12 at 21:31

An anomalous Close/DNC ratio could just be a sign of selective question picking.

If I were going for the Steward badge and got the most actions per day by casting 50 close votes and 70 DNCs, then that's the number I'd cast. It just makes game-theoretic sense. The way that I'd get that ratio is by skipping questions until I found a sufficient number that warranted DNC votes.

If that's the case, I wouldn't consider it gaming the system so much as taking into account all the arcane rules of the system. Given the number of questions in the queue, it shouldn't be hard to find plenty of questions warranting both close and DNC votes.

Your spot-checking suggests that there is some problematic gaming going on, but it's not a given that all the users with unusual numbers of DNC votes are always a problem.

share|improve this answer
If that were the case then the questions would still be open, but they are almost all not. It's hard to get evidence, but it doesn't appear that those who are racking up 70 or more reviews per day are 'cherry picking' items that don't deserve to be closed. It would be noble of them to do so, but it looks more like random clicking. In the history when groups of DNCs appear they are almost all now closed or migrated questions, which suggests that these users are at variance to the community. On the other hand, experienced users show up in the history as simple series of 50-ish closes. –  martin clayton Oct 2 '12 at 23:57
@martinclayton Sounds like there is something rotten there. It also sounds like the experienced users should change their review strategy if they want to get the Steward badge ASAP (or, more altruistically, to clear the very large queue ASAP). –  blahdiblah Oct 3 '12 at 0:11

I have jumped onto the review bandwagon just 2 or 3 days ago. It looked like a productive way to whittle away time, benefitting the site (the reason for the queues and badges!) while gaining some badges.

Now, if you look at my pattern, I have almost a 1:1 ratio. My recent question on Meta shows my conviction that there are some people who're simply too trigger-close-happy. Some of the items on the queue, actually quite a lot, are within the last 24 hours. These guys haven't even been to the site long and all they get is an angry torrent of closers shutting them down. Personally, I don't agree. And if you don't agree with me, that's fine too - we have a modicum of democracy here so exercise your right and I mine. In the past, these questions lived just a bit longer for the asker to get an answer before the question is closed and/or deleted.

There are also people happy to denounce R and matlab as programming languages. I had occasions myself where I had to ask borderline questions the typical answerers of which only visited once a week. I'm also far more liberal with what I believe SO can become, the questions it can host.

So what's your story?


(1) I rather think your logic that the higher the Do-Not-Close:Close ratio, the higher the indication of badge gaming is a bit flawed. I'm picking the closers here were "reviewing" the asker's first post, but the reasons chosen were just plain dumb.

Check wheather two column data present in another table?

It's a very real question (x3), and very much on topic (x1). Counts at time of writing. I would have DNC'ed the review just to provide balance.

(2) Another example of the close brigade. 4x exact duplicates at time of writing.

share|improve this answer
Richard, unfortunately there's not much data to go on, but I'm pretty sure there is some 'gaming' going on. To be clear: I don't have a problem with these users. People will go for badges, and good luck to them. But I think the badges should be designed so that 'playing the game well' aligns with 'improving the internet' as much as possible, and this bit is broken. I agree with you that there are other problems, for example with 'trigger-happy' closing. On the other hand plenty of rubbish goes unremarked upon in tags I frequent (e.g. 'ant'). –  martin clayton Oct 8 '12 at 21:51
To answer your question - my experience is that many items that have close votes do deserve them even if the reasons are not always right. I personally use 'not a real question' on the help vampires, which may well be wrong. I note your comments on the attitude to the [R] and [matlab] tags, and sympathise. It used to be the case that [tex] and [latex] were treated like that... now they just get migrated straight to their own ghetto. –  martin clayton Oct 8 '12 at 21:57
the reasons chosen were just plain dumb. An "I disagreed with the reasons chosen" would go a long way towards discussing this rather than insults. –  jonsca Oct 9 '12 at 11:02
@jon dumb is defined as lacking good judgement. When I make bad judgement, I call myself dumb. Those reasons didn't show any good judgement - will you try to justify them? At least martin took it in stride and didn't read too much offense into it. As recorded, if anyone wanted to do any sane analysis of close questions, the voted buckets are almost worthless. –  RichardTheKiwi Oct 9 '12 at 11:06
Well, I disagree that it was "lacking good judgement". I read the question carefully (see my comment on the post itself) and made a decision, and I stand by that. –  jonsca Oct 9 '12 at 11:08
If my justification is not convincing enough, there are thousands of questions that essentially say "Hey, write this for me because I can't" that get closed with that reason, so there is sufficient precedent. –  jonsca Oct 9 '12 at 11:09
@jon Precedence != justification // meta.stackexchange.com/questions/120474/… –  RichardTheKiwi Oct 9 '12 at 11:23
Wow, I'm not sure what that Meta post has to do with absolutely anything. Establishing precedence is crucial to the process of law. I think you're just baiting me here. –  jonsca Oct 9 '12 at 11:28
@jon I was only looking for a screenshot of the SE policy about precedence and justification. "Everybody does it" is just about the weakest argument one can make. Every teen will use it on their parent, doesn't make it a good argument. –  RichardTheKiwi Oct 9 '12 at 11:43
I offered you my justification in the other comments. You did not accept it. I offer this as an alternative explanation as to why it is a well-reasoned and valid choice. You can question anything you would like about "establishment" and the like, but the fact remains it is a justifiable reason to close such a question. If I've picked up a piece of litter off of the street, does it matter to argue whether it was a crime in that jurisdiction? –  jonsca Oct 9 '12 at 11:47
@jon - I didn't even realise you were one of the closers on the question. Explains the angst. lol. I'll keep quiet now because I know you won't. Nice to see the ends justifies the means argument turn up –  RichardTheKiwi Oct 9 '12 at 11:48
I regret that this turned into more of a philosophical discussion than perhaps either of us would have liked, but I felt that I had the right to rebut your insinuation that my process was not careful. I will let the matter rest as well. –  jonsca Oct 9 '12 at 11:55
Posts are closed because they are off-topic, or duplicates, or need improvement. How old they are has nothing to do with the price of milk. "An angry torrent of closers" is ad hominem nonsense. A good question will be able to stand on its own merits, whether or not it is old. Likewise, a question that needs to be closed belongs closed regardless of its age. If delete-worthy questions lived longer in the past, that's a bad thing. If someone asks for a recipe for a really tasty egg-salad sandwich, we should not provide an answer first and delete later. That turns the site into Yahoo Answers. –  ЯegDwight Oct 9 '12 at 13:16

When you run out of votes then in Review Close votes you don't get any more reviews and get a message

You have no more close votes today; come back in 1 hour.

On the review front page the review for close votes entry was moved to the bottom and greyed out (Sorry I did not keep a screen shot) so this is sort of happening.

share|improve this answer
Once you've used up all your closes, that queue is indeed no longer available. But if you have a even one close vote left, you can choose Do Not Close quite a few more times. I think there may be a separate limit for DNCs - a total count of 119 appears to be the ceiling. –  martin clayton Sep 24 '12 at 22:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .