Improving Review Queues - Design overview I: Onboarding and updating workflows

Question

This is one of three project announcements for Improving Review Queues. We’ve summarized the project objectives and goals here.

As a reminder, this project is still in the early stages of discovery. In this post, we are sharing proposed changes to the Onboarding and Updated Workflows/Paths. We are asking for your feedback before we begin implementation. After we collect community feedback, we will be open to including changes into the next design iteration.

Onboarding

Information and guidance about how to use the Review Queues is available, but it’s in several places. We are proposing different ways to include upfront instruction and guidance to better equip reviewers to use their newly earned privilege. We are continuing to make refinements to the language and copy in these messages as part of the discovery process.

Users will find an onboarding message on the homepage with general information about using the Review Queues. As they enter each new queue, they will be presented with a modal with unique instructions on how to best contribute to that queue.

Open questions:

• What did you wish was part of your Review Queue onboarding?
• What feature/aspect was difficult to discover as a new reviewer?
• What do you consider essential “day one” information for new reviewers?

Updated workflows and paths

We would like to update each queue with a new Actions menu that exposes all instructions and adds a bit of pause so that users can make more considered decisions. We will also be adding the new follow feature (available in the "…" button) so that reviewers can keep track of posts.

Reviewers will no longer have to open questions in a new browser tab to get more context. In addition to the post being reviewed, each review task will have the question, comments, and other answers on the review page to give context. We are also sharing a subtly different design in our user research sessions.

All changes will be supported by a Stacks update to the visual design along with responsive, mobile views.

Stack Overflow-specific changes:
We want to take a closer look at the success and failures of the First Post, Late Answers, Triage, and Help & Improvement queues.

In Triage, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. We want to remove the Requires editing action and replace it with two new actions. Selecting Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity) will be sent to the Close vote queue.

If a reviewer selects Very low quality in the Help & Improvement queue, the post will be sent to the Low Quality Post queue. By doing this, we hope to cease the endless cycle making these queues unsuccessful in helping moderation.

Finally, we are considering closing the First Posts and Late Answers queues and combining those tasks into Triage. These queues were intended to give good posts a head start toward success and filter out bad posts, but tasks could see better, faster intervention.

Open questions:

• Does it feel like any important actions or functions are missing from the review page?

• Does having the original post, comments, and additional answers all on the page make it easier to do a review?

  Stack Overflow only:

  • In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?
  • In Help & Improvement, do you anticipate any issues with sending Very low quality posts to the Low Quality Posts queue?
  • Do you anticipate any issues with removing the First Posts and Late Answers queues?

More updates

We share improvements to the review bans and new features in this final post: Improving Review Queues - Design overview II.

Answer 1

The onboarding guidance looks great - +1 for "Skip"; that cannot be repeated enough!

My recommendations for improvement would be simply to link to more extensive guidance for those who might want it. There are some great guides out there already; here are just a few:

• What are the guidelines for reviewing?
• What are the review queues, and how do they work?
• You're doing it wrong: A plea for sanity in the Low Quality Posts queue
• How does the Triage queue work?

In Triage, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. We want to remove the Requires editing action and replace it with two new actions. Selecting Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity) will be sent to the Close vote queue.

I would strongly recommend killing the "help and improvement" queue: it was never finished, and the crucial pieces of functionality which it lacks are now being planned as part of the close workflow redesign. It'd be one thing if it actually provided some value for the reasonable questions that make it in... But I couldn't find any evidence of this during the year after we first rolled it out, nor when I looked again a few years later - so far as I can tell, questions which are actually salvageable are overwhelmingly edited by people browsing their favorite tags outside of review - a queue is simply not an effective entrypoint for this sort of work.

Finally, we are considering closing the First Posts and Late Answers queues and combining those tasks into Triage.

Triage itself is a great UX for pre-vetting questions... But that purpose was never fully realized either! Instead, it evolved to become a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. - for questions. Eliminating other queues for pre-vetting questions in favor of Triage is an excellent plan, and I highly recommend making this change across all sites, leaving the Low Quality review to what it does best: allowing reviewers to quickly process problematic answers (this would also help to eliminate the broad class of bugs introduced by having significantly different workflows on SO vs. every other site).

Oh, and talk to the SOBotics folks. They've already been building far, far better replacements for the First Posts / Late Answers scenarios for years, and can help you to improve the heuristics for sending these posts into Triage & LQ Review (spoiler: the heuristics you're using now were good... 5-10 years ago. They're woefully archaic now!)

Finally, remember that these queues currently process an insane volume of posts... Any changes will have dramatic and difficult-to-predict effects far outside of review. Be prepared to monitor both review actions themselves, and their secondary / tertiary effects, and to make quick changes as required, for roughly 12-18 months after you begin rollout; this will need to be someone's primary responsibility for at least several months! Speaking from experience, when things go wrong they go very, very wrong very quickly - good luck!

Answer 2

It isn't necessarily related to this project, but I still feel like posting it here since it looks like the guidance for the options (currently visible when clicking the '(more)' link) will be more prominently visible as a sidebar widget.

I just noticed that for questions in the Low Quality Posts review queue the guidance says

Close if the question cannot be fixed and should be removed immediately

This is incomplete or even incorrect; we ultimately close questions to indicate that they need to be improved in order to be answered (and only if they're not, they will be removed). Reviewing a question in this queue doesn't have anything to do with deleting it, so please consider rewording it.

Answer 3

Reviewers will no longer have to open questions in a new browser tab to get more context. In addition to the post being reviewed, each review task will have the question, comments, and other answers on the review page to give context.

Taking this improvement a step further, is it possible to highlight the posts needing review to users with review privileges, in the course of normal browsing? Occasionally I find a post in the review queue after I've already seen it. If I'd known it needed review while reading, I could have already done it.

Answer 4

Close this post if it should be closed for any reason (consider leaving an explanatory comment)

I suppose that upon voting to close the question, the review queue will automatically load the next question, as it does now? In that case, I'd opt to add the word 'first'

(consider leaving an explanatory comment first)

since otherwise you have to navigate back to leave that comment.

I just noticed the current guidance does a better job in this regard, it says:

Close if this question should be closed for any reason; consider leaving an explanatory comment before voting

Answer 5

What feature/aspect was difficult to discover as a new reviewer?

The fact that if you use the back button after reviewing an item, you see how other users reviewed. As a new reviewer, this gives you some kind of peer review. (Of course, those reviewers might be worse than you are.) That could use more discoverability.

Or, taking it a step further: have an overview where your last X reviews disagreed with the final outcome. I know @rene has a script for that, but it would be nice as a built-in feature.

Answer 6

Questions and answers should be reviewed separately, not together

Reviewing questions and answers, in my opinion, requires different mindsets as well as different workflows. I've been tripped up quite a bit when I've been prompted to review a question right after reviewing an answer, but not realize that it's a question, and it's only when I try to review or flag it as a non-answer that I realize otherwise. The queues are nice about letting me know whether I'm looking at a question or an answer when I look in the appropriate spots, but looking at that as well as changing mindsets if changing between post types takes me time.

I disagree with your proposed change to remove First Posts and Late Answers and merge them with Triage. I'd be up for removing questions from First Posts and moving those to Triage, but please keep those separate queues for reviewing answers. Also, see my other suggestion of not moving questions reviewed as "Very low quality" in Help & Improvement to the otherwise answer-only Low Quality Posts.

Answer 7

When you guys incorporated the initial onboarding message for the Close/Reopen votes privilege, I was hopeful that you would apply the same logic to the review queues, but scaled up massively.

I have to say, I'm not disappointed. I like what I see.

What did you wish was part of your Review Queue onboarding?

I feel as though it's important to stress to new reviewers that reviewing correctly is paramount during their reviewing efforts. When you review incorrectly, and perhaps let a close-worthy question through the Triage queue, you increase the time that question remains open, which we know isn't preferable.

Additionally, I feel that it's important to stress how useful the Skip button is. When the reviewer is not concrete that an actionable button is the right choice (perhaps they've been reviewing a post for a long time?) it would be nice if it were made known that the Skip button is their best friend in their time of need.

What feature/aspect was difficult to discover as a new reviewer?

I feel as though the act of reviewing isn't too difficult to pick up on, provided the information given to the user is simultaneously succinct and complete. It's been mentioned a million times already, but the guidance for "Requires Editing" in the Triage queue is very difficult to understand for new and old reviewers.

On one hand, I would prefer not to dig through a lot of old meta posts to be able to review correctly. On the other, I really do want to review correctly, and I have spent a large amount of my time ensuring that my review choices were correct. My suggestion would be for you to word the onboarding explanations in a way that covers everything a reviewer might come across. Where that explanation fails should be instructions to Skip!

Lastly:

Does having the original post, comments, and additional answers all on the page make it easier to do a review?

Oh my goodness, YES. It's an excellent improvement, and reminds me of this under-review feature request. Having easy access to the comments and current answers (or the parent question of a review on an answer) gives a large amount of context to a reviewer, and should hopefully ensure they don't need to open the question into a new tab to browse the full contents of it.

Overall, the currently proposed changes are looking great, and I'm interested to see what other improvements are made along the way as other community members weigh in.

Answer 8

In the mockups, I note that the "Skip" link is small and subordinate to the "Submit" button, very easily missed. I'd argue that it should be a first-class button, given equal (or greater) emphasis in the UI.

Experienced users will learn an interface and promptly ignore it, so visual emphasis in a UI is primarily geared towards new users. For a new user the default action should absolutely be "Skip" (and, as other answers/comments have mentioned, we should explicitly call that out during onboarding), so the "Skip" button should be big and obvious.

Along those lines, I'd also suggest changing the order in which the other options are presented. In the mockup, the two least-used options ("Approve" and "Improve readability") are presented first. The vast majority of questions in triage need to be closed, so those two options should be near the top to help new users to understand that.

Overall, I think this is a really good start, and I'm really happy to see curators getting some love. I think with a few UI tweaks to subconsciously draw users towards the right actions it could be great.

Answer 9

As a general improvement to the workflows, add the ability to flag something as spam from any queue. You're likely to encounter spam in the "Close Votes", "Low Quality Posts", "Help and Improvement", "First Posts, "Late Answers", or "Triage" queues, but only the last three let you flag it as spam. From the "Close Votes" queue, you can only vote to close it, from "Low Quality", you can only suggest deleting it, and from "Help and Improvement", you can...well, you can't really do anything.

Answer 10

Potential unclear wording in new Triage options

I do like your proposed changes to the review actions in Triage. In fact, I was going to propose the same thing as you, to split it apart into four actions, with separate buttons for where the post simply needs details from its author and for when it needs to be removed as spam or rude or abusive. As far as I understand, these are the buttons you're proposing:

1. Looks OK - for questions that are good, on-topic, and don't need proofing
2. Improve readability - for questions that need some proofreading, but are otherwise good, on-topic (send to Help and Improvement)
3. Add clarity - for questions that need more details from their authors, or should be closed (vote or flag for closure)
4. Unsalvageable - for questions that are spam, rude or abusive, or need action by a moderator

Again, I like the categories, and those categories are exactly what I was thinking of before.

However, there is a small issue with the wording you're proposing: it's not entirely clear what action to take when reviewing a question that is fundamentally unsalvageable and needs closing (e.g. blatantly off-topic, a clear duplicate, etc.) but aren't spam or abusive. Your proposed wording for the "Add clarity" choice would imply that it's only for questions that could be improved by their authors, not for these types of questions that need closing but not red flag deletion.

I'd recommend changing the name of that choice to "Needs improvement", the same words used in the flag dialog, so it's clear that the closing option should be used for such questions.

Answer 11

Does it feel like any important actions or functions are missing from the review page?

For me yes. On Stack Overflow specifically, the H&I queue feels like it needs a "Close" button. Now, obviously you're trying to improve the workflow, and ideally posts that should be closed should never make it here in the first place, but unfortunately too many currently do. It'd be really nice if as a 3K user I could close it directly, or if there was some other action I could take to send it to the cv-queue.

Does having the original post, comments, and additional answers all on the page make it easier to do a review?

Yes, absolutely. I often find myself looking for this information anyway.

In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?

I have nothing particularly problematic to bring up, only I'm not sure this wording give quite as strong of a message as would be ideal. Perhaps there was some way to make clear that the OP needs to add clarity, and that can only done by the OP. On the other hand, improving readability can be done by anyone. It might be better in the long run if you can more strongly differentiate the two (i.e., what must be done by the OP vs other people).

But really, I think I'm just being nit-picky here.

In Help & Improvement, do you anticipate any issues with sending Very low quality posts to the Low Quality Posts queue?

There is a chance that people will mis-understand what is meant by "low-quality" here and might mistakenly push things that need to be closed into the low-quality queue. Not sure how major of an issue it will become, but something you might want to consider.

Do you anticipate any issues with removing the First Posts and Late Answers queues?

Not off the top of my head. If their intended functions can be better-served by other queues, then why not do that instead? (Obviously, you're already thinking of this; that was just rhetorical). No use having too much redundancy.

---

Oh, and BTW, I want to keep encouraging good things I see in you (the company) after the way 2019 ended. I want you to keep encouraging behavior I like, so I say this almost every time I give my feedback on meta. I say that because you might be tired of seeing me say stuff like this, but thanks again for asking. I see a lot of positive improvements for the queues here, and really appreciate that you have taken the time to ask us about it and listen to our feedback. So thanks again!

Answer 12

The problem with the current queues is that they are about how the post got there, not about what needs to be done with it or decided about it. It seems like you are (finally) moving away from this and I appreciate this. If you do so, please be thorough and get rid of the historical baggage.

For example, consider the following streamlined system of review queues, all focussing on what needs to be done or decided:

• A queue for providing guidance, editing, and checking for severe issues. This queue would be populated with first posts, late answers, posts with a bad quality heuristics, and possibly dedicated flags. The more reasons coincide, the more action a post needs to be de-queued. Possible actions would be like in the first-posts queue, maybe with a specific button for closure/needs to be deleted. Edits and comments from experienced users outside the queue can count towards de-queueing to some extent.

• A queue for deciding whether a question should be closed. This queue would be populated through close votes and flags and the above queue. It would work like the current close queue.

• A queue for deciding whether an answer should be deleted. This queue gets only populated by not-an-answer flags, a button in the first proposed queue, and possibly red flags. It only exists to decide about deletion.

• The reopen queue and suggested edits queue can mostly stay as they are.

This is a riff on this suggestion of mine.

Answer 13

I noticed a link to the timeline in the ... menu. Is that really useful? I never use the timeline while reviewing (probably because there was no direct link to it before) but I can't imagine it will be useful in more than 5% of the cases. I'd be inclined to leave the link where it normally is on the page, i.e. the clock icon under the downvote button (it's missing there right now).

Answer 14

Thank you so much for making these great improvements. I have a few suggestions.

I think the button "Improve readability" should open an edit dialog instead of sending this to the H&I queue. The Help and Improvement queue should be removed altogether. The only real action in that queue is to edit. All users who have access to Triage can either suggest an edit or have full-editing privileges. There's no need to push this question into another queue.

I would also recommend renaming the Low quality posts queue. Put questions in Triage and Answers in LQP. LQP can be renamed Low Quality Answers.

As for the automatic heuristics that Shog9 has talked about I strongly agree to add more. I have recently spent some time researching NAA heuristics and I can tell there are few give away phrases/signs that make a post definitely NAA. Also, code-only answers are ok most of the time, only if it's more than a few lines of code it should be reviewed.

Answer 15

"Very low quality" option should send straight to the correct queues

I like that choosing "Very low quality" in the Help & Improvement queue won't sent the post right back into Triage, causing the endless loop issue that we talked about earlier.

However, sending those into the Low Quality Posts queue isn't the right thing to do in my opinion, because of how the queue works. The only actions that can be taken from that queue are to either mark them as looking OK, or send them to the Close Votes queue. There's no way to handle a post that needs a spam or abuse deletion (category 4 here), and it simply adds an extra step for questions needing to be closed (category 3 here). I think the proper thing to do is to open the flag dialog when clicking the "Very low quality" button, and have the reviewer choose a closing or flagging option, so that they get sent straight to the right spots.

Additionally, this change would bring questions into the same queue, which for many years on SO only, has consisted solely of answers. This brings me to another improvement suggestion.

Answer 16

As you noted in Part II:

Most users learn how to be good reviewers by actually reviewing posts, but sometimes find themselves making wrong decisions.

Given this, I’d love to see this formalized into an explicit review training queue, similar to @PM-2Ring’s proposal, “Practice queues for review training”.

Review Training Queue

When a user first gains privileges for a queue, they will be presented with a set of curated audits which are hand-picked by moderators to be a representative sample of that queue’s review criteria.

When a user fails a training audit, there is no penalty. But they receive an explanation that educates them about the review criteria; e.g.,

Careful! While that answer may be incorrect, it is nevertheless an answer. Reviews should remove posts that are not answers; voting should be used to rank answers by usefulness.

Once the user has successfully completed each of the training audits, they will then receive a message:

Congratulations! You’ve finished the training period, and are now ready to review some actual posts. But be careful! After this point, unhelpful reviews may result in a review suspension. Still feeling unprepared? Be sure to read the FAQ.

Variations

• Require the reviewer to correctly pass a percentage of training audits before being let loose on real audits. This would require having a larger pool of curated audits.
• Require suspended reviewers to go back through the training audits before regaining their review privileges.
• Prompt reviewers to retake the training audits after a failed audit—or even if their reviews are suspicious (e.g., based on frequency of Looks OK posts or minority reports).

Moderator Responsibility

This ties in neatly with something @Cody has raised on Stack Overflow:

This is why I (and several of the other moderators) are staunch advocates of revamping the audit system to allow us to nominate posts as audit candidates (both false-positives and false-negatives). We would select posts that are obvious and unmistakable, yet still represent corner-cases that often seem to trip up reviewers. This would be far more effective for pedagogical purposes, and I believe it would still scale adequately, as we could serve similar audits to all users without defeating the purpose.

I’m not advocating for anything so expansive here. But this would offer a simpler version of that, while prioritizing the moderator effort toward onboarding and training new reviewers.

---

The goal is to provide an interactive simulation of reviews as an easier way of training new reviewers—and a friendlier way of addressing initial mistakes.

Answer 17

Here is something that always bothered me with the Triage queue: how to handle duplicates?

Here is the thing: I can review questions and decide whether they are OK or not even if I'm not an expert. I can see that there is a good problem description, what is expected, what isn't. I can also read code in a language I don't use and figure out it's representative. Or I can spot if any of these is missing. Not always, as it might be something very specific that is outside my knowledge but in many cases I can say whether a question is OK or is missing something.

The problem: I don't know if there is a canonical duplicate for a given issue. Let's put aside other duplicates for the moment - canonicals are the ones that tag experts would be seeing all the time as it's a frequent question. Since I'm not a tag expert, I can't really say whether something is a really common question or not.

Right now, duplicate closure falls under "Unsalvageable". So, I often feel uncomfortable choosing "Looks OK" for a question I'm not an expert in when it might actually be a duplicate. I often skip these. Yet at the same time, it doesn't feel like I should do that, as I feel the question, even if a duplicate, can still be classified as a good question.

If I'm not mistaken, under the new review system the same thing would apply - duplicates fall under "Close this post" in Triage. So, I have the same set of concerns.

Answer 18

Add better audits to Triage

I'd like to see better audits in the Triage queue. Right now, the only audits I see in the Triage queue are really good questions that should be reviewed "Looks OK", and blatant spam. There is no automated checking done to see if users are appropriately reviewing closeable questions as "Unsalvageable" (in the current model), which has led to a group of moderators handing out manual review bans in droves, to the extent that the majority of review bans resulting from Triage are manual.

Can we please add closeable questions to the review audit system for Triage, so that people can be prompted to pick the right choice ("Unsalvageable" in the current model, or "Add clarity" in your proposed new one)?

Answer 19

I see no mentioning of the Reopen queue.

The main thing that bothers me with that queue is that once a closed question has been edited, it enters the queue. This leads to many Leave closed clicks because a question that is closed as a duplicate shouldn't obviously get reopened after some grammar fixes...

One thing that could be useful is to show who actually did the edit (I couldn't find that at least in a convenient way). But another option is to take it one step further and either:

• Don't allow people other than OP to edit a closed question.
• Add a very visible message stating that this is a closed question and edits should be meaningful.

I think that allowing only OPs to edit closed question might sound extreme, but if you think about it, only they can really edit the question to make it on-topic. Whether the question is closed as a duplicate, needs details or focus, how can anyone else add details to make the question not a duplicate or more focused. If a question got closed, it probably means that only OP can save it!

Another, "softer", approach might be to allow edits from everyone, but only edits by the OP will send the question to the Reopen queue. Or, add a tick-box for anyone other than OP to mark if they believe the edit was significant enough to be sent to the Reopen queue.

Answer 20

The instructions say

• Consider using filters to find posts relevant to your interests and subject-matter expertise

Not all queues have them, and neither do all sites in the network. Are you planning to add them to more queues, or if not, will you remember to hide this bullet point if it's not applicable? There's little as frustrating as searching for promised functionality that doesn't exist.

Answer 21

Adding more context to each queue

More context is certainly welcome, but I think that no matter what, people are going to be opening the question in a new tab. When triaging, and it is obvious that a question was well written this isn't as important, but certainly before closing or deleting a post I always like to get the full context of what was going on before making such a decision.

Perhaps you could just embed the original question in an iframe or make a plan around using HTML Portals when they come out.

Answer 22

What did you wish was part of your Review Queue onboarding?

More emphasis on QoL features

Filtering is pretty much the best way that you can prevent new reviewers from getting discouraged with the review queues. This would make them feel more effective and confident of their decision. Make it part of the onboarding route to set up a filter for the queue you acquired privileges, then introduce the skipping and afterwards the rest.

Answer 23

In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?

I think this is exactly what is needed, but please can it be worded to clarify whether the editing needs to be done by the OP or others in the community? Also, the wording "Add clarity if the question is clear" seems very odd.

Answer 24

What did you wish was part of your Review Queue onboarding?

More guidance. I have had access to reviews for so long I do not recall when I gained the privileges. However, I have rarely used them. Despite going through the onboarding guidance and trial reviews a couple times, I still do not know what it is looking for and have zero confidence I would review items correctly. I think some examples of how things should be reviewed would help. I'm not talking about easy cut and dried items either. The trial ones are great in that it's difficult to decide whether to mark for improvement or close. But as I recall there wasn't much guidance on why they should have been marked for closing.

Answer 25

Regarding the Suggested Edits review queue, it is very disappointing to see no attention placed on the canned message choices available to the reviewer, or consideration of whether the review UI could offer some way of allowing for a reviewer to be more helpful in training new editors.

It feels ironic that the canned messages are so harsh in the face of recent campaigns for a more welcoming and friendly environment. Why do I say harsh? Well, at some point, I thought I could increase my contribution to the SE sites by beginning to edit to improve. I recall getting slammed down with a terse canned message on occasion and recall that the message associated with the rejection offered zero help in understanding why a well-intentioned edit (a significant one) was rejected with no apparent avenue for being reconsidered. It seemed that the reviewer missed really understanding what was behind the edit. The overly harsh, terse, rigid, and unhelpful canned messages with no place for the reviewer to "explain", or for me, as the editor, to "explain", successfully caused me to stop trying to improve posts.

Before today, I'd only done 9 reviews, and I'd forgotten what it was like to review edits in light of my experience, so I went through a days allowance of reviews today to (re-)experience for myself how a reviewer might feel when trying to be sensitive to people learning how to edit. The experience was illuminating and very disappointing.

It seems that at some point reviewers must get hardened to how their rejects are going to be received by the people whose edits are being rejected because the choice is either to not review or continue to use the square-peg-in-a-round-hole canned messages.

After today's experience, I better understand now why I felt slammed. The system actually promotes it. The messages are canned, and terse, and are (mostly) not written to allow for cases where a reviewer might be overly picky or where the editor was really not so far off the rails as to warrant being slapped down with a radio-button-selected canned message that was the best of all the poorly fitting one-shot messages possible.

Clearly some edits seemed to warrant, to some extent, the harsh, succinct canned messages, but a lot of the time, chose to skip because there was no way to tell the new (low rep) editor what it was about their otherwise okay edit made accept not viable. In some cases I edited and left the modified edit for someone else to review, but it wasn't uncommon that I wasn't really in a position to edit either - not necessarily being a subject matter expert, or not really having enough context supplied in the review UI.

The exercise revealed that what felt like to me a personal response was not in fact anything personal at all, but rather a canned message that offered no way for me as a reviewer to hint to the editor specifically why I could not in good faith accept the edit even though the edit contained noteworthy and helpful changes.

I did skip a LOT to avoid cases where I didn't know the material well enough to make an edit better (to avoid reject), or to avoid gnarly cases where an edit had a big enough problem to not warrant accept but did not warrant a response written as severely as the canned message options offered.

I saw over and over that while one fixed canned message was more appropriate than the rest, it was deficient in one way or another and that it was truly unfortunate not to be able to provide a low-rep editor some more pointed feedback right then and there with a text entry box as appears by Causes Harm.

In some cases, Causes Harm with its edit box was usable even if it required a bit of a loose interpretation of "harm". (What is harm by the way?) More often than not, though, I knew that having to radio-button-select a more appropriate canned message was going to be frustrating to new editors in the queue.

I think there is much more here to do than training reviewers. The queue tools themselves need to be much better geared toward helping new reviewers tell editors (nicely) what they are doing wrong so they aren't put off by harshly worded canned messages that primarily fit worst-case edits.

Regarding nicely, see Also: What’s the course of action when you think someone is being unjust?

It seems sad that the answer to the question here is that the new editor needs to cherry pick what to edit to avoid getting slammed by a reviewer until one reaches a (huge) threshold of 2K edits, and that the harshness of the canned messages isn't addressed by any answer. I suspect it may not be uncommon for people who feel unjustly slammed to stop contributing rather than do what this editor did (posting a question to meta).

On another note, there were some edits where it felt like it would be better to move the edited message to a poor-quality question/answer review queue instead of spending time reviewing formatting/spelling on a post that really needed content work by the original poster. Some edits were entirely legitimate for formatting/spelling improvement, but it was clear that the effort was wasted unless the OP improved the question. It seemed pointless to reject the edit, but also pointless to accept it. It wasn't immediately clear how a new reviewer would know how to handle this situation from the review UI. It seems like that could have been integrated rather than require the reviewer to find another way to do it (i.e. flag).

I don't really know if this is the place to register observations or not. This series of posts doesn't actually make it very clear where community contributions are to be placed.

Answer 26

What did you wish was part of your Review Queue onboarding?

What would have been great is a short (10-15 minute) video explaining how the review process works. For the triage queue specifically, it could include two or three examples of questions for each category along with a discussion of why the question gets the category it does.

Following the intro video, the new reviewer could be given a quiz with 20-25 carefully curated questions - similar to the audit questions, but instead of suspending them for getting one wrong, they could be given the "correct" answer along with an explanation. Only when they get enough quiz answers correct would they be given access to the real Review Queues.

What feature/aspect was difficult to discover as a new reviewer?

I had no idea that review suspensions were even a thing until I discovered them the hard way.

Answer 27

feature-request

Edit as an option in Close Vote Queue should be removed.

People that have just started using queues will use it in a wrong way, and experienced users don't need such option nor guidance.

If the question can be sufficiently edited by a reviewer, they can always use Leave Open option and subsequently edit question.

I am not in position to comment how that option works on other sites, but on Stack Overflow, if the question is closeable, usually there is very little reviewers can do to put the question in shape. Situations where they can make question on topic are quite rare.