As most of us already heard (most of us have already complained about it), the new review queue is very comfortable. So comfortable it's very easy to game in order to get easy badges (5 gold ones).

The reason for that is that people simply go in, do the minimal job required to enable the "I'm good" button, and carry on to the next post. On most cases, that's the wrong thing to do.

My Suggestion

The problem we have is that people can review too many items per queue per day. There's no limitation. And how do you determine such limitation?

  • Reputation? That hardly seems the solution. Reputation is an indicator of how good of an answerer you are, or more specifically, how good is your knowledge and veterancy in the site. It's not a community trust measure.
  • Static? That doesn't solve anything really.

So my suggestion is some new point system called "Trust".

"Trust" points can be achieved mutually when your review votes conform to the rest of the community.

For example

A late answer pops up at the Late Answer review queue.

  • User A thinks it's a bad answer, and votes it down.
  • User B thinks it's a bad answer, and votes to delete it, leaving a comment for the answerer.
  • User C doesn't care, and votes it up, just to have another review point.

In the end of such day, Users A and B, would get N trust points, and User C would lose M trust points.

What do you do if the opposite happens?

  • User A thinks it's a bad answer, and votes it down.
  • User B doesn't care, and votes it up, just to have another review point.
  • User C doesn't care, and votes it up, just to have another review point.

I propose a less smart system for this case, you automatically ban users with "more than N% of upvoted reviews for X days", so for instance, if more than 80% of your reviews are consisted of upvotes, you'll be banned from reviewing for a week.

Now for the real deal

All users begin with a set number of trust points. Say 200. Those 200 Trust points would allow the user to review 20 items per day. If a user gains trust points, he'll be able to review more and more items per day per queue. If a user loses them, he'll be able to review less and less, and eventually none at all.

Some Points

  • Trust points can be either visible or invisible. If visible, a log, stating how each trust movements was earned, should be available as well (much like the reputation log).
  • Trust points should have an aging mechanism on them. A conflict which caused me to lose 20 trust points yesterday, would only cost me 15 next week (for example). This allows the "no review" ban to be lifted gradually, and if the user is willing to shape up, he'll get more trust points in no time.

Your thoughts? Ideas for improvements?

  • 12
    wouldn't the removal of badges solve this whole mess ?
    – teresko
    Nov 10 '12 at 13:05
  • 1
    It might improve the mess @teresko, but I doubt it would solve it. Nov 10 '12 at 13:06
  • Problem with this is if, in your scenario with a crappy post, if A and B blindly vote up, and C rightly votes down/deletes, C gets penalized...
    – Mat
    Nov 10 '12 at 13:06
  • 1
    @teresko: Indeed, but we still want to incentivise users who promote good reviews. Nov 10 '12 at 13:06
  • @Mat: Thoughts of that. In that case, you'll need a different, more strict system to start with (Users with 80% overall upvote reviews are automatically banned for N days). Just to get the party started. Once the number of blind voters is dulled a bit, this system should really kick in. Nov 10 '12 at 13:07
  • @MadaraUchiha , do you use the /review for badges? Would you trust people who use it just for the badges? Granted .. adding minor incentive (like only bronze badge for reviewing 20 posts) might help to initially attract people, but, without visible benefit, the "pokemons" will leave and only people who actually care about SO content will keep on using it.
    – teresko
    Nov 10 '12 at 13:08
  • @teresko: I don't use it just for the badges, I work towards badges as further incentive, because I like badges. It's like saying people answer questions here just for the reputation. It may be true, but if that helps generating quality content, why not? Nov 10 '12 at 13:11
  • 5
    In this case, the promise of gold badge is harming the content quality.
    – teresko
    Nov 10 '12 at 13:15
  • 1
    @MadaraUchiha Reputation largely seems to be driven by creating content of some quality (though there are some exceptions). Reviewing badges (unfortunately) are driven by the performance of whatever incomplete action might take you there the fastest.
    – Bart
    Nov 10 '12 at 13:30
  • 1
    What if upvoting the answer in your example was the right action and those who downvoted were wrong? Mat brought this up in comments already from a different angle, but my point is that also that downvoting isn't always the correct action to take.
    – Adam Lear StaffMod
    Nov 10 '12 at 16:29
  • @AnnaLear: Then in that case, the appropriate action was taken, so there aren't any problems. Nov 10 '12 at 16:39
  • 1
    @MadaraUchiha Unless I'm missing something, your system would still penalize someone who took the appropriate action.
    – Adam Lear StaffMod
    Nov 10 '12 at 16:40

This proposal is almost the same but not entirely different from the old flag weight system, which was introduced to promote people who consistently flag well with more flags higher up on the mods' lists and silence sources of noises with less flags that would only appear towards the bottom.

The problem here is that instead of having a gold badge that can be achieved once and for all, you're burdening the system with a new score, a new metric to rate users with, one that you aren't done farming when the badge comes. Worse off, it creates tension between users as one person's actions reduces another person's score - and if reputation didn't, flag weight taught us that losing score is a pretty big deal.

So no, let's not have a new score, a new scoreboard, a new number to grind on and new drama on why people are stupid and decreasing my score. Reputation is scoring enough.

If the problem really is careless reviewing in order to get the badge, you can do a few things:

  • Lower the bar so that somebody who is really determined to get that badge, no matter what we do, does less overall damage to the system. 500 bad decisions are half as bad as 1,000 and both show a lot of commitment.
  • Raise the bar by only counting the first 50 actions per day towards badges. Once you get to 50 you can keep reviewing, but the bars won't go up. As an additional bonus you get to see how many people only do it for the badge and consistently stop at the 50 reviews mark.
  • Remove the bar entirely by removing the badge and giving up.
  • 2
    I'm entirely in favour of point #3, but the second isn't a bad idea either Nov 10 '12 at 14:46
  • @Sathya I kind of dislike the options where the only solution is absolute despair :P
    – badp
    Nov 10 '12 at 15:33
  • You're already limited to 40 actions per queue
    – Ben Brocka
    Nov 10 '12 at 15:43
  • @BenBrocka There's more to "actions" than voting?
    – badp
    Nov 10 '12 at 15:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .