Please stop putting old posts into the Late Answers queue; aka Leave my spam alone

Question

A long time ago¹ in a galaxy far, far away (or certainly very far from the one I inhabit) three users were so desperate to get their silver Reviewer badge for the Late Answers review queue² that they upvoted spam³.

This wasn't just any spam; this was spam that is so old the link doesn't even work.

I got really quite frustrated about this; luckily I was going out last night and had to stop writing the first draft of this post⁴.

The answer in question (10k only) had been placed into the Late Answers review queue, along with another spam post (10k only) by this same user as part of the drip feeding of old posts into the queue. The first review is the one that really annoyed me; but the second also got an upvote (before thankfully being downvoted again).

It's widely believed that the current review queues are broken. When you either believe a system is broken, or are unable to determine whether this is so, it's irrational to continue using it when you don't have to. As there's no need to place all the old late answers into the review queue let's stop.

Before yesterday this was a perfectly harmless spam answer, with a dead link, on an unconstructive question. No one would have paid it the slightest attention. It would now be a fairly highly upvoted answer, indicating community approval of the seems-to-be non-existent product, if I hadn't been there staring open-mouthed in horror as the upvotes piled in^5..

So, can we please stop feeding the "trolls" and put no more old answers into the Late Answers review queue until a way of making the queues work well has been devised.

tl;dr

This feature request is not to propose a solution to any real or perceived problems with the queues, but to stop the problem getting needlessly worse until these problems have been fixed.

If no more old answers to old questions are placed in the queue then there is less chance of things like the above happening.

<sub>1. Yesterday.
2. Okay, okay, one already has it. Don't spoil the story...
3. A quick note on why I count this as spam. A user joined Stack Overflow. They posted one answer (10k only) in the same minute they joined and a second (10k only) 6 minutes later. Both answers linked to the same product (dead link). The user was never seen again.
4. The draft I woke up to was a rant about banning all low reputation users from upvoting in a review queue. Not a good idea.
5. I had run out of everything but moderator flags so want to thank Brad for closing the question and deleting the answers so quickly.</sub>

Answer 1

There are two solutions:

1. Given that people are so desperate to earn the badge, why not simply allow "skip" (renamed "Looks Good" perhaps) to count towards the review total?
2. Remove the badges. I assume that this isn't going to happen.

The first solution means that the people who really, really wanted the badge could skip their way through to getting it (rate limit the number of reviews to 20 or 30 a day). Skips would still not remove the post from the queue for everyone else (unless it got 5 or 10 skips) so those who want to do proper reviews can still see the post and edit/comment/vote properly.

Yes, this will probably have the effect of devaluing the badge, but there are already badges (Enthusiast, Fanatic, the voting badges, etc) that can be got without any real effort so there is a precedent :)

Answer 2

I disagree that this example shows an intrinsic problem with the late answers review queue or with the selection of posts to be reviewed. It was in fact a good idea to show this post to reviewers. This post should have been flagged as spam, which is one of the options in the late answers review queue.

The problem here is the “serial reviewers” — the people who frequent the queue and do anything to have their review counter increase regardless what should really be done about the post.

A minimum reputation requirement isn't going to work. I've seen plenty of low-rep users who do good reviews (I mostly observe from the suggested edit review, and I see them suggesting good edits). I've also seen high-reputation users doing bad reviews (again, from the suggested edit review, there are >10k serial approvers).

Removing the badges and the counters would help. They are making it a competition on who has performed the largest number of reviews. They encourage doing reviews for the sake of doing reviews, expending no effort.

I don't have a good proposal for reviewing reviewers, which is what is really needed. Until then, remove all the competitive aspects of reviewing. Scrap the badges and the visible counters. That way, we'll have review backlogs, but at least the reviews that will have been performed will mean something.

Answer 3

There's an assumption here that this is primary about gaming. I disagree. I think that it is as much about the psychology of the available buttons.

Clicking 'skip' feels like shirking the job. Thus, it biases people in favor of doing something, and the upvote, as per @Chris Gerken, is sitting right there at hand.

I predict that the simple addition of an 'ok as it is' button would result in a positive change.

Answer 4

I think the solution lies in some sort of automated ban mechanism.

The current cheater's approach is to simply upvote, because that's the fastest way to enable the "I'm done" button. People whose reviews consists of nothing but^[1] upvotes, should be automatically banned from review, and their serial upvoting reversed.

A different, more community based method would be to check their compatibilities with the other voters. If you said to upvote, but 3 others said delete, that disagreement would cost you some invisible "trust points". The trust points would be a measure of how much review you can do per day, where the floor is 0 (trust points, and reviews per day). The trust mechanism should have an aging period where an older conflict has less weight than a new one (this allow the ban to be gradually lifted over time).

When the offender begins to perform good reviews, (i.e. He chooses "close" where others have too), he'll gain trust points back, increasing the number of reviews he can do per day.

It's still a little sketchy, but I'm working on an official feature request.

_{1. around 80-90%}