What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

A long time ago1 in a galaxy far, far away (or certainly very far from the one I inhabit) three users were so desperate to get their silver Reviewer badge for the Late Answers review queue2 that they upvoted spam3.

This wasn't just any spam; this was spam that is so old the link doesn't even work.

I got really quite frustrated about this; luckily I was going out last night and had to stop writing the first draft of this post4.

The answer in question (10k only) had been placed into the Late Answers review queue, along with another spam post (10k only) by this same user as part of the drip feeding of old posts into the queue. The first review is the one that really annoyed me; but the second also got an upvote (before thankfully being downvoted again).

It's widely believed that the current review queues are broken. When you either believe a system is broken, or are unable to determine whether this is so, it's irrational to continue using it when you don't have to. As there's no need to place all the old late answers into the review queue let's stop.

Before yesterday this was a perfectly harmless spam answer, with a dead link, on an unconstructive question. No one would have paid it the slightest attention. It would now be a fairly highly upvoted answer, indicating community approval of the seems-to-be non-existent product, if I hadn't been there staring open-mouthed in horror as the upvotes piled in5..

So, can we please stop feeding the "trolls" and put no more old answers into the Late Answers review queue until a way of making the queues work well has been devised.

tl;dr

This feature request is not to propose a solution to any real or perceived problems with the queues, but to stop the problem getting needlessly worse until these problems have been fixed.

If no more old answers to old questions are placed in the queue then there is less chance of things like the above happening.

1. Yesterday.
2. Okay, okay, one already has it. Don't spoil the story...
3. A quick note on why I count this as spam. A user joined Stack Overflow. They posted one answer (10k only) in the same minute they joined and a second (10k only) 6 minutes later. Both answers linked to the same product (dead link). The user was never seen again.
4. The draft I woke up to was a rant about banning all low reputation users from upvoting in a review queue. Not a good idea.
5. I had run out of everything but moderator flags so want to thank Brad for closing the question and deleting the answers so quickly.

share|improve this question
3  
I think the point 4 is not a bad idea.. –  j0k Nov 10 '12 at 11:48
5  
It's not a good one though :-). It's the sort of thing that'll never get implemented and the problem isn't just caused by low rep users so I don't see how it'll solve the problem. –  ben is uǝq backwards Nov 10 '12 at 11:50
    
    
Another example of the broken review queue -- Review queue comment –  bluefeet Nov 10 '12 at 13:54
5  
I said it before, I'll say it again: Scrap the bloody badges! –  Daniel Fischer Nov 10 '12 at 13:54
    
yeah, away with all the review badges - but not before I got all them shiny golds :-) And yeah, it is disgusting to see obvious not-an-answer getting up to 4 upvotes until a flag stops the nonsense - but that applies to all late answers, not only the old late answers. –  kleopatra Nov 10 '12 at 14:29
    
I'd be very interested in one of the downvoters commenting or answering on why they think this feature request is a bad idea. –  ben is uǝq backwards Nov 10 '12 at 14:41
2  
I'm of the opinion that the badges for the review queues are ruining Stackoverflow's quality. Posts are being upvoted without even being read, useless comments added, etc, all in the name of "reviewing". The fact that some people go through the queues in a matter of seconds also removes the possibility for proper reviewing by people who actually care to do so. –  J. Steen Nov 10 '12 at 18:07
2  
Yesterday I saw a "Thank you" answer with 4 upvotes. It almost looks like some people only click on the upvote button to get a badge early... –  Toon Krijthe Nov 10 '12 at 20:34
    
This has more upvotes than the 21k jump in the Low Quality review queue! –  ben is uǝq backwards Nov 13 '12 at 18:21
add comment

4 Answers

There are two solutions:

  1. Given that people are so desperate to earn the badge, why not simply allow "skip" (renamed "Looks Good" perhaps) to count towards the review total?
  2. Remove the badges. I assume that this isn't going to happen.

The first solution means that the people who really, really wanted the badge could skip their way through to getting it (rate limit the number of reviews to 20 or 30 a day). Skips would still not remove the post from the queue for everyone else (unless it got 5 or 10 skips) so those who want to do proper reviews can still see the post and edit/comment/vote properly.

Yes, this will probably have the effect of devaluing the badge, but there are already badges (Enthusiast, Fanatic, the voting badges, etc) that can be got without any real effort so there is a precedent :)

share|improve this answer
14  
Why not just add a "Gimme the badge" button to speed things up? –  Bo Persson Nov 10 '12 at 13:08
    
@BoPersson - I'm assuming that a) the review queues won't be taken away and b) these people will do some decent reviews. –  ChrisF Nov 10 '12 at 13:13
3  
Maybe strike a balance between requiring action and not demanding unnecessary action? The previous reviewer badge only required 200 actions and 1000 "looks" –  Ben Brocka Nov 10 '12 at 14:41
1  
@BenBrocka - that might work. You'll still get some bad actions, but there won't be as many. –  ChrisF Nov 10 '12 at 14:46
    
Even if the queues were purely altruistic and incredibly difficult to use there would be "bad actions", just vastly fewer actions overall. there's definitely a balance to be struck somewhere. –  Ben Brocka Nov 10 '12 at 15:08
add comment

I disagree that this example shows an intrinsic problem with the late answers review queue or with the selection of posts to be reviewed. It was in fact a good idea to show this post to reviewers. This post should have been flagged as spam, which is one of the options in the late answers review queue.

The problem here is the “serial reviewers” — the people who frequent the queue and do anything to have their review counter increase regardless what should really be done about the post.

A minimum reputation requirement isn't going to work. I've seen plenty of low-rep users who do good reviews (I mostly observe from the suggested edit review, and I see them suggesting good edits). I've also seen high-reputation users doing bad reviews (again, from the suggested edit review, there are >10k serial approvers).

Removing the badges and the counters would help. They are making it a competition on who has performed the largest number of reviews. They encourage doing reviews for the sake of doing reviews, expending no effort.

I don't have a good proposal for reviewing reviewers, which is what is really needed. Until then, remove all the competitive aspects of reviewing. Scrap the badges and the visible counters. That way, we'll have review backlogs, but at least the reviews that will have been performed will mean something.

share|improve this answer
2  
Well this is an issue in lots of places on the site. We can't have a community that rewards attention to various things with badges and then complain when those things attract people who want to collect badges. There was once a website by some guy named Joel where he talked about rewarding the wrong things, both in context of job performance and bug tracking... so it's kinda surprising that the SO family of sites have what is essentially the same class of problem he described quite a long time ago now. –  RobM Nov 10 '12 at 13:43
1  
I'm not claiming that this post shows an intrinsic problem. I'm saying that as there is a problem can we please stop exacerbating it for no reason. I'm not even looking for solutions to the problem. –  ben is uǝq backwards Nov 10 '12 at 13:46
4  
I say just leave the bronze ones for now so we can at least teach people about the review system. But yes, the silver and gold ones seem to only be causing problems, at least on SO. I don't know if this is really a problem at all on smaller sites where the queues generally sit empty a lot more than on SO. –  animuson Nov 10 '12 at 14:35
    
You are not addressing the OP's question at all. He specifically was not looking for a solution to the review problem, and even agrees that with a problem-less review system that example would be fine in the queue. He said, "until it's fixed, why make the problem worse?". he didn't say "how do we fix it?" and he didn't say "this example is what's causing the problem". –  Ben Lee Nov 16 '12 at 0:00
1  
@BenLee I am addressing his question, by explaining why I disagree with his proposed solution (viz, stopping populating the queue). –  Gilles Nov 16 '12 at 8:24
add comment

There's an assumption here that this is primary about gaming. I disagree. I think that it is as much about the psychology of the available buttons.

Clicking 'skip' feels like shirking the job. Thus, it biases people in favor of doing something, and the upvote, as per @Chris Gerken, is sitting right there at hand.

I predict that the simple addition of an 'ok as it is' button would result in a positive change.

share|improve this answer
add comment

I think the solution lies in some sort of automated ban mechanism.

The current cheater's approach is to simply upvote, because that's the fastest way to enable the "I'm done" button. People whose reviews consists of nothing but[1] upvotes, should be automatically banned from review, and their serial upvoting reversed.

A different, more community based method would be to check their compatibilities with the other voters. If you said to upvote, but 3 others said delete, that disagreement would cost you some invisible "trust points". The trust points would be a measure of how much review you can do per day, where the floor is 0 (trust points, and reviews per day). The trust mechanism should have an aging period where an older conflict has less weight than a new one (this allow the ban to be gradually lifted over time).

When the offender begins to perform good reviews, (i.e. He chooses "close" where others have too), he'll gain trust points back, increasing the number of reviews he can do per day.

It's still a little sketchy, but I'm working on an official feature request.

1. around 80-90%

share|improve this answer
    
Just hope me saying "delete" and three others upvoting doesn't affect my trust. Guess this is hard! –  Bo Persson Nov 10 '12 at 13:11
1  
As long as you have to do something to move to the next review you're going to have lots of up votes. Don't implement this without also adding an "It's OK" button that adds nothing to anybody's rep. –  Chris Gerken Nov 10 '12 at 13:34
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .