Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

The problem

I found the review instructions confusing, mostly because I was expecting some kind of "item goes away from the list after I've done something to it" action. After quite a bit of poking around I finally get that that's not what happens, but I still have no real solid idea of when items leave the list.

For perspective on where I'm coming from, I have 4100 points on SO so it's not like I'm new to the site. My flag weight of 135 shows I am a fledgling in reviewing but I'm willing to help occasionally.

My feature request

How about creating some clear instructions for new reviewers so they can understand what the heck to do and don't click around getting all confused (like I did)? Answers by themselves without questions made it hard to review them, so I kept clicking to the actual question page, then when the item didn't disappear from the review list I figured I'd done something wrong and I had to make my upvote/comment/edit right in the review page. I lost a bit of time experimenting with that approach and still saw no result. I wondered what I'd done wrong and spent a fair amount of time thrashing around.

You could put the link to the instructions on the right side of the review pages so anyone who wants to read more about it all can do so and quickly get with the program. I'm imagining something very similar to the "How to Format" box to the right when writing a question. It has links at the bottom to pages with extended help, in case the brief explanation is not enough.

Building in the right incentives

I have to say it's a bit of a demotivator to participate in review to know that what I do won't in fact make items disappear. Because then I don't get that tiny intrinsic payout of "I made the number of items go down." Why would I spend my own valuable time reviewing if I can't see that I did anything? I need to SEE it, not just figure that I must be helping somehow in some nebulous way. Some kind of positive feedback loop that helps me see the result of my effort will be much more likely to make me come back and keep helping. I mean, I can see my reputation points go up almost immediately when I participate, but there's nothing like that when reviewing except delayed flag weight increases that are not visibly tied to a specific action I took.

For one example of seemingly spinning wheels, why does an answer to a closed question show up in the list? Does the answer even matter at that point? And what about comments? If I comment, does that do anything to the review items? Or if I edit does it make them go away? You get the idea.

Helping ensure the work is done

Last, given there are hundreds of items needing review, it seems like it would be important to have a list of truly unreviewed items so that some assurance can be had that each item needing review actually got it. If the review list is random, depending on how many people are volunteering to work the list, you may have items fall through the cracks that never get a review. By making some clear process that actually removes items from the "unreviewed" category, I think you will get better coverage of the task.

share|improve this question
Related questions/answers here and here – Benjol Jun 9 '11 at 8:40
@ben Thanks. I did search before posting. – ErikE Jun 9 '11 at 15:50
in other words, I couldn't agree more :) – Benjol Jun 9 '11 at 18:40
@Ben Further elucidation (agreeing with you even more): I actually read those questions before posting my question, and they still didn't really help. They only addressed working with SO questions, answers, and comments, but I get that, since that's what I do all the time. I need additional information about how reviewing itself works, not instructions about what are good answers and what are not. – ErikE Jun 9 '11 at 18:47
It would be nice to have a indicator that I have already reviewed a post. A checkbox would do. It would also be nice to be able to exclude posts that I have reviewed from my 30 results. It seems like you are asking for a counter on unreviewed items. The side bar has a count of posts in a category. Perhaps under the post could there would be a count of how many of those posts were reviewed by at least one person. – this.josh Jun 9 '11 at 20:22
up vote 6 down vote accepted

Sorry for the late response, Erik - and thanks for the well-considered, thoughtful feedback!

I have to say it's a bit of a demotivator to participate in review to know that what I do won't in fact make items disappear. Because then I don't get that tiny intrinsic payout of "I made the number of items go down."

It turns out, this was a huge problem with the old "list-style" review system. Seeing pages and pages of posts, in random order with little or no indication that you were making a real dent in the pile was incredibly demotivating. Initial efforts to correct this (badge progress, powerful filtering options, disappearing items) helped a little bit if you took the time to learn to use them, but the Augean Stables factor remained high for most potential reviewers.

The new system does its best to correct these problems:

  • You review only one item at a time. "Focus on what's in front of you" is an old productivity mantra, and for good reason: distractions impact both the speed and quality of your efforts. Each task includes specific instructions tailored for the queue and type of item being reviewed.

  • You don't see items you've already reviewed. Or closed items, deleted items, etc. There's still a significant amount of caching going on behind the scenes, but since only one item at a time needs to be displayed, it's a lot cheaper to make sure that item should be displayed... And to automatically move on to the next one if it shouldn't be.

  • Each item is reviewed only as many times as necessary. This is surprisingly hard to achieve in practice, particularly when the queues get close to being empty - but the end result is well worth it in terms of reduced frustration and duplication of effort.

  • Both personal progress and site-wide progress are prominently displayed. Even if the tiny dent you're making in the site-wide pile is hard to make out, you have a personal counter indicating what you've done that grows with each review - and a daily leaderboard showing your place among other reviewers to go with it.

share|improve this answer
Comparing the new review process now to what it was a year ago, you certainly have my applause and congratulations. You addressed every one of my points and I think really improved the whole thing. I don't spend a lot of time reviewing, but having a number go down when I take action really works in my mind. – ErikE Jan 19 '13 at 1:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .