20

A lot has been said on Meta over the past couple of months about problems with the review queues; just have a look through the tag. According to one common theory, many of the worst individual reviews are submitted by users who are just trying to grind out the Reviewer and Steward badges.

These users completely ignore the intended purpose of the queues — or so the theory goes — and instead press whatever buttons allow them to increment their "reviews done" counters most quickly. That translates into a lot of upvotes/accept votes on terrible new user posts/suggested edits.

If the theory is correct, bad reviewers should stop reviewing as soon as they hit a thousand total reviews and get the shiny gold Steward badge, because additional work after that point would lead to no additional reward. Well, even considering the 20-review-per-day cap, those badge grinders have had more than enough time to get their virtual pieces of gold.

I'd like to put the theory to the test. Has the number of bad reviews per day decreased recently? How about the proportion of bad reviews to total reviews? Ideas for how to measure these values are welcome.

Improve this question
11
  • 3
    Now that's a plan! Anyone who gets their gold coloured bytes and then doesn't review for the next 5 times they go on the site gets their gold badge revoked and is forever banned from accessing /review!
    – ben is uǝq backwards
    Commented Dec 11, 2012 at 17:48
  • 7
    1. you assume no new badge grinders are being added, or that they're being added more slowly than they're getting the badge. That may or may not be the case. 2. Some people are just bad at reviewing, or feel bad about not upvoting/not accepting/etc. for posts. It's hard (possible, but hard) to differentiate the two types of bad reviewers. 3. Some people are doing lots of reviews to be on the all time review leaderboards, and are doing low quality reviews to get there.
    – Servy
    Commented Dec 11, 2012 at 17:48
  • @Ben Having to wait until they've performed 1000 bad reviews and waited several days is letting them do quite a lot of damage...even if you can stop them after that. Also note that there will be some people just reviewing for the badge who stop (or take a break) when they get it, but still took the time to actually review properly.
    – Servy
    Commented Dec 11, 2012 at 17:50
  • I haven't posted it as a feature request @Servy :-).
    – ben is uǝq backwards
    Commented Dec 11, 2012 at 17:51
  • @Ben And that means I can't reply to it or mention problems with it?
    – Servy
    Commented Dec 11, 2012 at 17:52
  • @Servy yes, I do assume that there was a large number of badge grinders who jumped on reviewing when the badges were introduced. I actually even said so explicitly in an early draft of this question, then decided it wasn't necessary, heh. I didn't know about the leaderboards, though. That's... ugh.
    – Pops
    Commented Dec 11, 2012 at 17:52
  • Then of course there might be people who think going into first/last posts and upvoting everything is actually helpful. Not sure if that's true or not, or if it's enough to be significant, but it's something to consider.
    – Servy
    Commented Dec 11, 2012 at 17:54
  • 1
    @PopularDemand Yeah, I imagine many of those grinding badges started right away, and the problem will decrease slightly as many of the get the badge, it's more a question of whether the new people joining the system just for the badge are still enough to effectively prevent the system from being usable or not. I honestly don't know the answer to that. Keep in mind there were also a lot of people approving everything in edits before the change, when there was no incentive to do so; they just though that was honestly helpful. The problem got worse with the new system, but it existed before.
    – Servy
    Commented Dec 11, 2012 at 17:57
  • Another thing to look at, at the moment there are only about 400 Gold Steward badges rewarded, being that a badge grinder can get between 4-6 of them, and assuming at least some are legit (I know I have one... And I don't think I'm a "badge grinder") it seems like either there aren't that many badge-grinders compared to users with > 2-3k rep (of course even a few can do quite a bit of damage), or not that many have gotten their gold badges yet.
    – Jack
    Commented Dec 11, 2012 at 18:04
  • For reference, here are those who recently received the gold review badge: stackoverflow.com/badges/2279/steward and the silver review badge: stackoverflow.com/badges/1478/reviewer
    – Brad Larson Mod
    Commented Dec 11, 2012 at 20:02
  • I think that the Steward badge can be awarded multiple times.
    – mathlander
    Commented Mar 13 at 17:09

1 Answer 1

Reset to default
7

For the suggested edits at the very least, I've actually been looking at some of the statistics

Most people with 0 rejected reviews, have less than 100 accepted reviews, and most people who approve 90% of reviews have less than the 250 badge goal.

Now, not all reviewers are super careful to evaluate all aspects of a review, but when a review is obviously a bad one(such as adding extra comments), than they normally actually do get rejected.

First posts is a little bit of a different story, and I've tried to look at statistics a little bit, but the only annotations I've seen next to reviewers are reviewed You have people go into those posts with good intent, often to find that there's nothing obviously wrong with the post, but that they're not enough of an expert on the subject to properly evaluate it to comment or edit, and it's not clear whether the post deserves either an upvote or a downvote.

Most people want to do something rather than just skip, and the easiest way to say that "this is not a problem post" is to up-vote the post.

Improve this answer
2
  • 6
    As regards that last line, it seems to me that there are more posts with upvotes these days that just do not deserve them, zero is about where they ought to be.
    – Remou
    Commented Dec 11, 2012 at 18:16
  • 3
    Indeed, I've been downvoting a lot more lately simply because there are now plenty of questions at +2 or +3 that are horrible and should never have been upvoted. It's insane.
    – Charles
    Commented Dec 11, 2012 at 23:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .