What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 129 Stack Exchange communities.

We're getting close, boys - we're gettin' close!

The Community Review Dashboard is finally nearing something we can call "release quality". All of the major categories from the old system are in place, and the time has come to address those stinkin' badges...

The old reviewer badge had a fairly complicated set of criteria (1000 reviews, over 200 actioned, predicated on first earning Strunk & White), and that's not something I'm interested in explaining (or even understanding) any longer. So we're replacing these with three new, per-queue badges:

  • Custodian is awarded for the very first review made in a queue.

    The custodian badge appearance and description

  • Reviewer is awarded for 250 reviews in a queue. This replaces the badge of the same name awarded for participation in the old system.

    The reviewer badge appearance and description

  • Steward is awarded for 1,000 reviews in a queue.

    The steward badge appearance and description

Each of these badges can be earned once per queue - so currently you can earn up to five of each type. Badges earned under the old system will be retained, but can no longer be earned.

In recognition of the heroic effort put in thus far by the folks who've helped to test these systems and clear the backlog, we're awarding these retroactively. And because it was previously a separate system, we were able to preserve historical data on suggested edit reviews - so the folks who labored to keep suggested edits under control prior to the new system will be getting a nice pile of virtual precious metal.

We still have to implement a progress indicator showing your march from one badge to the next, but the criteria are simple enough that for the time being you can calculate it easily by viewing your current statistics in any queue.

share|improve this question
2  
So, what happens if you already have the old reviewer? –  Yannis Sep 21 '12 at 23:13
    
So I've got to start again from scratch? I only ask as my counts appear to be zero while other people's are already in the 1000's –  ChrisF Sep 21 '12 at 23:13
    
@YannisRizos It looks like the Reviewer badge page will list the way each badge was earned. My existing Reviewer badge is listed as "1000 reviews, over 200 actioned in the old review system". –  Jeremy Banks Sep 21 '12 at 23:16
    
@ChrisF No, you keep your old badges. –  AndrewC Sep 21 '12 at 23:16
6  
Gaaah, and I was just closing in on 900 reviews after a year and a half, and those were for combined reviews of all types. Jebus knows how long it'll take to hit 1000 on each criteria now. Not that it's all about the badges of course. (Who am i kidding). –  JonW Sep 21 '12 at 23:17
1  
@AndrewC - I was thinking more of the counts –  ChrisF Sep 21 '12 at 23:17
9  
Nice, time to start filtering based on tags of knowledge area instead of klumping around just hitting those buttons like a blind man in a bowl full of peeled grapes –  random Sep 21 '12 at 23:17
    
And they'll award badges retrospectively. –  AndrewC Sep 21 '12 at 23:18
1  
@JeremyBanks Cool, just got the new Reviewer on ProgSE, and it sits side by side with the old one. –  Yannis Sep 21 '12 at 23:20
    
Is it me, or does the new system put edits from tags I frequent at the front of my queue? –  AndrewC Sep 21 '12 at 23:21
    
Ah, they appear to be back filling the data as I'm getting badges on some sites where I haven't reviewed recently –  ChrisF Sep 21 '12 at 23:23
    
Nice, i just wake up and start the lappy, and very happy to see one more gold badge,i got all three, thanks S.O. :) –  Lucifer Sep 21 '12 at 23:55
    
Does this mean you'll be transferring the stats? :D (Obviously the badge part of that is answered already.) Or will previous reviewers only get the silver badge still present? –  animuson Sep 22 '12 at 0:09
3  
The latter, @animuson. Since the other two badges didn't exist under the old system, and (apart from suggested edits) the new queues aren't exactly compatible, there'll be no other backfilling. –  Shog9 Sep 22 '12 at 0:19
1  
I just use The Gimp for that, @hims056. Select, distort selection, copy, paste, drop shadow. –  Shog9 Sep 22 '12 at 17:31

3 Answers 3

I like this idea. It encourages reviewing in each category. The old system encouraged reviewing, but only in general. You could do 100 first questions, 100 late answers, and 800 first answers and call it done. This encourages people to continue and go for all types of posts that could potentially have issues.

However, there are users who have been reviewing consistently under the old system, both to improve the site and to earn the badge. Their efforts are (or can be interpreted as) being disregarded. I have been trying to earn the Reviewer badge here on Meta (not easy, everything gets reviewed real quick because this is where the active people are), but I've noticed lately that my number reviewed (in the old list) only increases dynamically (live update when clicking review, but resets when reloading the page).

We have put in effort. I don't think that should be discarded. I propose that users who have made significant progress towards the Reviewer badge under the old system should be able to earn it under the new.

Proposal: Users who have at least 500 reviews (at least 100 actioned) under the old system should be awarded the Reviewer badge for the old review system once they earn at least the Custodian badge for all review types and the Steward badge for at least one review type.

share|improve this answer
3  
Unfortunately, the design of the old system made porting progress forward rather difficult; because it didn't need to track what you were doing in any one area, it didn't - in fact, the numbers have been wrong ever since the new system was introduced (something we didn't catch until two days ago). Keeping the old badges works because badges aren't normally revoked - only the descriptions had to change. But we can't easily award new "old" badges without investing significant effort into code that'll very quickly become dead weight. –  Shog9 Sep 22 '12 at 1:04
4  
It's not about recording progress in each area - simply check the number reviewed and number actioned to see if a threshold (I proposed 500/100) is passed - then check new review progress –  Jim Sep 22 '12 at 1:48
1  
@Shog9 Is there some way to just divide the number of old reviews evenly among the 4 remaining categories and give those credits back? I realize weighing the division by the category's popularity would be difficult, but is that bad if we're a little off? I mean we did work; so what if it's in the wrong category? If you really want, you could weigh it by per-site flow through statistics on each category, but that seems excessive to me. –  Double AA Sep 23 '12 at 1:55
4  
@Shog: I can understand people's frustration. I feel somewhat betrayed myself. I got to 780 reviews, almost 400 actioned, on a site where the review queue was all but empty 24/7, when the rug got pulled out from under me. And now I have to start all over again. Not good. –  ЯegDwight Sep 23 '12 at 1:57

I'm not convinced this is a good thing. By looking only at the number of reviews, it encourages people to review carelessly.

From my experience in the suggested edits queue, there are a lot of unequivocally bad suggested edits. Most common are those that randomly make some words bold or mark them as code, and those that make a trivial formating change and also intraduce mispellings. I reject these, but looking at the history, a substantial minority ultimately get approved, and probably most have at least 1 approve vote.

Similarly, I feel there's a bandwagon effect going on in the close-vote queue, where people tend to overly quickly agree with the previous votes. Also, it seems that the new system is being gamed:

Solution?

A possible solution for is to count toward the badge only when other reviewers agree with your vote (e.g. on suggested edits). But:

  • Since I think most suggested edits are approved, this be less of an incentive to look carefully than it would seem.
  • This doesn't work for close votes as currently implemented, since in the current implementation you can see others' votes (hence it would exacerbate the bandwagon problem).
  • Granted, this would reintroduce some of the complexity from the old badge.
share|improve this answer
    
If ever there was a prophetic answer... –  Adam Rackis Jul 10 '13 at 15:43

As reviewing of suggested edits is now covered by one type of Custodian/Reviewer/Steward badges, it seems the "Proofreader" badge ("approved or rejected 100 suggested edits") is now obsolete and can possibly no longer be earned. That would mean existence of a badge that can no longer be earned (like "Beta" and "Precognitive") for an action (reviewing suggested edits) that still can be performed (unlike "Beta" and "Precognitive"). That seems rather strange.

Therefore, I suggest to abandon the "Proofreader" badge. As "Reviewer" badges earned under the old system are retained, it seems that the proper action for already earned "Proofreader" badges is to convert them to "Custodian" badges. (EDIT: Like the retained "Reviewer" badges, these badges would be included in addition to the five standard "Custodian" badges; the description of the retained badges would read "Approved or rejected 100 suggested edits in the old review system".)

share|improve this answer
3  
Earning Proofreader is still possible; three were awarded this morning on SO. It's less important now that the other badges are in place, but provides a nice stop-gap between Custodian (1 review) and Reviewer (250 reviews) for suggested edits. Also note that on smaller sites, reviewing 250 suggested edits might take a lot longer than 100. –  Shog9 Sep 22 '12 at 17:29
    
@Shog9 If there's really a need for a "nice stop-gap" between a bronze badge and its silver variant, I'd rather suggest to tone down the requirement for the silver badge somewhat. (Right now, "Proofreader" is akin to a "Really nice, but not quite good question" badge for a question with, say, 20 votes.) –  lockstep Sep 22 '12 at 17:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .