New Community Review badges

Question

The Community Review Dashboard is finally nearing something we can call "release quality". All of the major categories from the old system are in place, and the time has come to address those stinkin' badges...

The old reviewer badge had a fairly complicated set of criteria (1000 reviews, over 200 actioned, predicated on first earning Strunk & White), and that's not something I'm interested in explaining (or even understanding) any longer. So we're replacing these with three new, per-queue badges:

• Custodian is awarded for the very first review made in a queue.

  

• Reviewer is awarded for 250 reviews in a queue. This replaces the badge of the same name awarded for participation in the old system.

  

• Steward is awarded for 1,000 reviews in a queue.

  

Each of these badges can be earned once per queue - so currently you can earn up to five of each type. Badges earned under the old system will be retained, but can no longer be earned.

In recognition of the heroic effort put in thus far by the folks who've helped to test these systems and clear the backlog, we're awarding these retroactively. And because it was previously a separate system, we were able to preserve historical data on suggested edit reviews - so the folks who labored to keep suggested edits under control prior to the new system will be getting a nice pile of virtual precious metal.

We still have to implement a progress indicator showing your march from one badge to the next, but the criteria are simple enough that for the time being you can calculate it easily by viewing your current statistics in any queue.

Answer 1

I like this idea. It encourages reviewing in each category. The old system encouraged reviewing, but only in general. You could do 100 first questions, 100 late answers, and 800 first answers and call it done. This encourages people to continue and go for all types of posts that could potentially have issues.

However, there are users who have been reviewing consistently under the old system, both to improve the site and to earn the badge. Their efforts are (or can be interpreted as) being disregarded. I have been trying to earn the Reviewer badge here on Meta (not easy, everything gets reviewed real quick because this is where the active people are), but I've noticed lately that my number reviewed (in the old list) only increases dynamically (live update when clicking review, but resets when reloading the page).

We have put in effort. I don't think that should be discarded. I propose that users who have made significant progress towards the Reviewer badge under the old system should be able to earn it under the new.

Proposal: Users who have at least 500 reviews (at least 100 actioned) under the old system should be awarded the Reviewer badge for the old review system once they earn at least the Custodian badge for all review types and the Steward badge for at least one review type.

Answer 2

I'm not convinced this is a good thing. By looking only at the number of reviews, it encourages people to review carelessly.

From my experience in the suggested edits queue, there are a lot of unequivocally bad suggested edits. Most common are those that randomly make some words bold or mark them as code, and those that make a trivial formating change and also intraduce mispellings. I reject these, but looking at the history, a substantial minority ultimately get approved, and probably most have at least 1 approve vote.

Similarly, I feel there's a bandwagon effect going on in the close-vote queue, where people tend to overly quickly agree with the previous votes. Also, it seems that the new system is being gamed:

• Steward Badge gaming visible in the Review Close Queue
• The current review system encourages fake reviews; some people upvote everything rather than actually fixing problems.

Solution?

A possible solution for is to count toward the badge only when other reviewers agree with your vote (e.g. on suggested edits). But:

• Since I think most suggested edits are approved, this be less of an incentive to look carefully than it would seem.
• This doesn't work for close votes as currently implemented, since in the current implementation you can see others' votes (hence it would exacerbate the bandwagon problem).
• Granted, this would reintroduce some of the complexity from the old badge.

Answer 3

As reviewing of suggested edits is now covered by one type of Custodian/Reviewer/Steward badges, it seems the "Proofreader" badge ("approved or rejected 100 suggested edits") is now obsolete and can possibly no longer be earned. That would mean existence of a badge that can no longer be earned (like "Beta" and "Precognitive") for an action (reviewing suggested edits) that still can be performed (unlike "Beta" and "Precognitive"). That seems rather strange.

Therefore, I suggest to abandon the "Proofreader" badge. As "Reviewer" badges earned under the old system are retained, it seems that the proper action for already earned "Proofreader" badges is to convert them to "Custodian" badges. (EDIT: Like the retained "Reviewer" badges, these badges would be included in addition to the five standard "Custodian" badges; the description of the retained badges would read "Approved or rejected 100 suggested edits in the old review system".)