146

As part of our Q2 roadmap, the Community Product team is working towards improving Review Queues. This project is still in the early stages of discovery. In addition to user research interviews, we are sharing our progress with you today to ask for your feedback before we begin implementation. This is the first post in a series of new communication we hope to have with the community that uses the new tag. After we collect community feedback, we will be open to including changes into the next design iteration and continue to share updates with you.

Objective & Goals

Community moderation is an important part of Stack Exchange and we want to provide the best user experience to those volunteering their time across our sites. In parallel to the Close Experience work, we hope these improvements will better support our curator and moderator communities.

We spoke one-on-one with many active reviewers across the Stack Exchange network, conducted data analysis on current Review Queue usage, and read through just about all of the existing issues raised by our community. We used this qualitative and quantitative data to inform our goals and initial design decisions.

In this redesign, we are exploring ways to improve the overall usability of Review Queues. This includes, but is not limited to:

Providing guidance and revisiting the banning experience
The current on-page instructions are not visible and could be misinterpreted, leading new reviewers to make poor decisions. Inappropriate actions on review tasks or audits can lead to a suspension of privileges with no obvious means of understanding what happened.

Adding more context to each queue
Users are opening browser tabs or using user scripts to view the original post they’re reviewing, other answers, and comments to help make educated decisions.

Improving filtering and discovery tools
The filter feature often goes unnoticed and unused. As it exists now, certain combinations of filters can leave queues empty. It is a poor method to discover or target new tasks.

Improving queue workflows and pathways
On Stack Overflow, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. H&I reviewers are then limited to either Skip or mark the task as Very Low Quality, having it return to the Triage queue and creating a frustrating endless loop.

Along with the Triage queue, First Posts and Late Answers are not fulfilling the goals that they were made to accomplish. These queues were intended to give good posts a head start toward success and filter out bad posts, but tasks could see better, faster intervention in other queues.

What’s next

We’re collecting feedback here on Meta and from 1:1 user interviews. If you have opted-in for user research, you may receive an email invite this week. We’ll be asking questions about how this design may improve (or worsen) review workflows and what you’d like to see in a curated review experience. You can opt-in to future user research opportunities by enabling Research emails in your profile settings.

This project is big. To focus your feedback, we have broken this announcement into three, digestible posts:

We’ll be using community feedback to continue to iterate on this work. We look forward to hearing your thoughts and concerns about this project. Please let us know if there are other Review Queue issues you’d like to see addressed as well.

  • 27
    "On Stack Overflow, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. H&I reviewers are then limited to either Skip or mark the task as Very Low Quality, having it return to the Triage queue and creating a frustrating endless loop." - Heyyy, I remember saying something along those lines! :D – Nick Apr 23 at 18:13
  • 23
    On Stack Overflow, situation with unsalvageable posts in H&I seems to be indeed worse than you describe. Reviewers who mark the task as Very Low Quality have system cast a VLQ flag on their behalf and it typically ends at moderators who decline it. Few declines of such flags trigger flag suspension. Essentially the only safe option for garbage in H&I review is Skip, can you imagine what kind of snaked toilet this creates. – gnat Apr 23 at 21:47
  • 38
    A) I'm so glad this project is happening. Review has been a sore spot for years. B) The iterative approach with gather community feedback along the way is a real winner! I'm gonna model it in my current job. Thank you! – Jon Ericson Apr 23 at 22:18
  • 19
    One fundamental thing to consider: if a queue is never at any point in time empty, then the system design is broken. Queues are supposed to handle peaks in input, so that they can be processed later. But if there is more items entering a queue then leaving, then no amount of queueing will save the system. This is the core reason why the review system occasionally fails completely. The goal must be to reduce manual review by having question wizards, smart scripts, maintenance bots etc. – Lundin Apr 24 at 7:52
  • 22
    Optionally: don't alienate all the people who did these reviews. Moderators and veteran users. SO has consciously and consistently been alienating such users for many years. So eventually you will end up with employees having to do the reviews (or become Quora 2), meaning that you should seek to automate the process so that you don't have to hire as many paid moderators. – Lundin Apr 24 at 7:55
  • 23
    Honestly, I'm really surprised anyone contributes to reviewing at all. It seems an utterly thankless task with occasional horrible pitfalls where you get silently banned, a constant suspicion that you're "cheating" somehow, and no upside whatsoever. So glad to hear it's being worked on. – Steve Bennett Apr 24 at 13:29
  • 5
    As someone who stopped reviewing (on Stack Overflow) very quickly, I am so happy to see attention paid to the queues! – Super Jade Apr 25 at 19:24
  • 8
    “We spoke one-on-one with many active reviewers across the Stack Exchange network, conducted data analysis on current Review Queue usage, and read through just about all of the existing issues raised by our community.” — This is great to hear, but have you used the queues yourselves to get a better feeling of where the pain points are? – user289905 Apr 26 at 10:17
  • 4
    @SteveBennett Reviewing is a thankless job indeed, but also a very, very important one if you want to maintain content-quality. I review a lot more on Code Review than on Stack Overflow, but that's why I do it. Site quality goes down if I and others stop. – Mast Apr 26 at 11:00
  • 19
    Banning Experience - Yes please. I'm not an idiot but I'm a new reviewer - and I've already been banned twice, and I was actually trying to be useful. I've basically rarely touched the queue since my second ban, because I don't want to get banned a third time. Very demotivating. Having some guidance or better support here would be great. – qaisjp Apr 26 at 20:27
  • 5
    It is probably worth talking to people who have done review queue work in the past but not for some time; people who are active reviewers are evidently not finding it painful enough to cause them to abandon it. I suspect if you look at active users who are eligible to run queues, only a small percentage actually do, and figuring out the reasons the rest of them don't, and fixing those, should be an important part of any overhaul on the review system. – Xiong Chiamiov May 4 at 16:39
  • 2
    How about getting a reward for actually spending time reviewing, closing and fixing things. It's a never ending job, it never stops. All the new people posting the same "I want this implemented, can someone help?" that we need to close is tiresome, and we don't really get a reward for cleaning up the site. Perhaps get 0.1 points for every successfully closed (voted) question etc? That would be more valuable than "tidying up the queues" which is probably needed, but in all honesty, feeling some kickback for actually cleaning up the site would be ever better. //SO user&"moderator" – Torxed May 7 at 8:27
  • 2
    Banning Experience - when you ban a user, how about also stopping the showing of the red "high volume" dot. Also, in place of the unhelpful 'There are no review queues available to you' how about a link to the review area link so that we can see what the issue was? – Andrew Mortimer May 7 at 11:02
  • 2
    It would be helpful if we could earn the privilege of not getting "tested" after some number of reviews. It is another thing that adds to the "thankless" feeling. – Katie May 7 at 16:14
  • 1
    @Katie I would be careful about this. I did tens thousands reviews in different queues on differrent sites and I fail review audits very infrequently, maybe once in few months but I still would prefer getting these because audits help me control whether I am sufficiently attentive – gnat May 23 at 22:29
146

I'll leave specific notes on your design overview posts, but wanna drop my overall impressions here: as someone who has spent a considerable amount of time and energy on these systems since their inception, I believe you've correctly identified the areas that most need focus (whether that focus is needed due to serious problems in their current behavior, or due to missed opportunities).

You've done your homework, and it shows. 👍

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .