New onboarding for review queues

Question

The Public Platform team is continuing to work towards improving the review queues and we’re here today to announce a new onboarding experience for the review queues.

In the product discovery phase (see this earlier Meta announcement), we learned that each queues’ instructions could be easily misinterpreted and additional information was difficult to find. Without proper guidance or context, new reviewers would contribute inconsistent reviews and risk having their review privileges suspended.

With this release, our goals were to 1) better highlight the review privilege and 2) provide upfront instruction and guidance for successful reviewing.

New privilege announcements

Community moderation is a significant aspect of Stack Overflow and the Stack Exchange sites. We want users to be aware and familiar with this newly earned privilege, but also celebrate the opportunity to contribute to the community in a unique way.

Many users were discovering the review queues by chance. Once the privilege is earned, a new review queue icon with a red alert would appear in the top-right part of the navigation. We are adding a popover to bring more attention to this easy-to-miss icon. We will also be adding notices for when users earn access to other, higher reputation queues.

Additionally, new reviewers will also receive an email notification about the newly earned privilege. They will only get the email if they earn the privilege and don't perform a review within 24 hours (anyone who doesn't want to receive email notifications like this can visit their email settings preferences and turn off "Tips and Reminders").

We are also showing a popover to users who earned access to review queues in the past, but have never done a review or haven't reviewed in the last 30 days. This popover is only appearing for the next 90 days. The goal is to engage users who may not have interacted with the queues since we made visual design changes and improved communication about suspensions.

Welcome banner

If a reviewer clicks “Learn more” on the on-page popover or goes to the review queue home page (/review), they’ll be greeted with an onboarding message with general information about the review queues.

Informational modals

Queue-specific modals will appear upon page load as you visit each queue. They summarize and highlight important aspects and tips about reviewing in a particular queue. This information can be revisited at any time by clicking “Learn more” under the page header.

Feedback

As always, please leave your feedback and any bugs you may discover related to this release below this post. We will be monitoring this post for the next few weeks (May 4, 2021). After May 4th, please report any further issues as new questions.

Answer 1

bug status-completed

The popup shows even when I have used the review queues a lot on other SE sites.

I am aware of how the queues work.

I don't want to be diverted to looking at how they have changed since last time I looked a few minutes ago. (Hey maybe they have, why would they alert me if they haven't?)

Answer 2

bugstatus-completed

The posting states,

We are also showing a popover to users who earned access to review queues in the past, but have never done a review or haven't reviewed in the last 30 days.

I am seeing the popover for queues where I have reviewed within the past two days on the very same site (specifically Stack Overflow). Here's a screen shot of a "Suggested edit" review I did two days ago. When I click through to the review in my review history, the popover is displayed:

As you can see I worked in this queue two days ago, but I'm still getting the popover.

I am also getting the popover if I start a new review in the "Suggested edits" queue:

Answer 3

bugstatus-completed

There's a grammatical error in

• Most posts require x votes to close it

posts is plural, so the last word must be them.

Answer 4

feature-requeststatus-completed

I really like the idea of notifying new users who have just gained the review privilege. That's cool! And I can understand that you might also want to show it to old hands as well if they haven't reviewed for a while. However, many of us are active on many sites, so now I need to dismiss a popup telling me that I can learn about reviewing on every site!

I regularly visit Unix & Linux, Ask Ubuntu, Meta SE, Stack Overflow, Biology, Bioinformatics, Science Fiction and Fantasy, English Language and more rarely various others. Please don't make me have to dismiss the same popup everywhere I go.

Next, if I want to do a review, I need to click through 6 other popups (one per queue) with onboarding info. And, once more, I need to do this for every site on which I have review privileges. In my case, that is 10 sites, which means I would have to click through 10 popups like the above, and then another 60 introductions to the review queues¹.

Now, I admit I am not the most active reviewer these days, I do most of my reviewing on the posts directly and on the site I moderate. But others are more active reviewers than I and also have the rep on many sites. Also, this makes me much less likely to touch the review queues now that I know I'll need to click through all these banners to get to the actual reviews.

So, could we please only show these things once per user? If I have dismissed the popup on one site, dismiss it from all. And if I have clicked through the introduction to Queue X on Site Y, then also dismiss the introduction to Queue X on all others sites I have review access on.

---

¹_{OK, that's a slight exaggeration. I don't have access to all 6 queues on all 10 sites. So it would actually be 57 (or thereabouts), not 60.}

Answer 5

discussionstatus-planned

It's probably too late for this, but if the goals included:

a new onboarding experience for the review queues

and to

provide upfront instruction and guidance for successful reviewing.

(my emphasis)

... then I would suggest that veteran reviewers be excluded from the target audience. Consider excluding people who have one or more Steward badges for that queue, for example. I feel like this change is targeted at new users, which is great, but please consider the use-case of established users.

Answer 6

bugstatus-declined

Daily review limit

You can review 20 tasks a day per queue.

I see that on sites where I'm a ♦ moderator and I consequently don't have a daily review limit.

Answer 7

bugstatus-completed

The Suggested Edits modal looks like:

I don't think we are able to delete posts from the suggested edits queue (unless that's a new feature you haven't told us about)...

Looks like it made its way there by mistake.

Answer 8

status-completed

The reopen vote queue popup talks about 'recommend closure flags' that will be declined. As far as I know, recommend closure flags are marked helpful when a post is closed, and you can't flag a post that's already closed to recommend more closure.

Shouldn't this line be in the close vote popup instead, saying such flags will be declined if enough people vote to "Leave Open"?

Answer 9

bugstatus-completed

The Low Quality Posts queue has questions as well as answers (except on Stack Overflow); the guidance popup seems to be solely directed at answers. Questions cannot be deleted from that queue (unless you are a ♦ moderator); they can be voted or flagged for closure.

Answer 10

status-completed

Inconsistent capitalization:

vs.

Unsure why "Review" is capitalized.

Answer 11

feature-request status-review

The modal popups are super annoying. Please put a button on them that says "Don't show me any of these popups on any site again."

And please modify your design rules to make that an essential part of any future modals you add to SE. I'm honestly extremely surprised it isn't already part of your rules.

Answer 12

discussion proposal status-completed

Screenshot for reference; suggestions/issues listed below:

1. "Welcome to Review queues"

   I would suggest that "the" be added here... articles do wonders for readability: "Welcome to the Review queues". The feature toast already does it this way:

2. Help improve question-and-answer quality!

   Shouldn't this just be "question and answer quality", sans hyphens? Also, why a newline/separate div after this? Both lines can easily fit on a single line. Text will wrap automatically anyway for extremely small viewports if necessary:

   

3. The sideways carets make me think they are menu options that I can click to expand each section to read more. If they are just bullet points, I would recommend just making the bullet icon a traditional bullet point or dash.

4. is moderated by you and the community's efforts.

   For consistency this should probably be "moderated by you and the community" or "moderated by your and the community's efforts". Otherwise it sounds like my actions should somehow be different from the community's actions.

5. Review queues (also known as review tasks)

   Review queues are not known as Review tasks... individual review items are called tasks... right? This seems borne out by the explanation under the "Daily review limit" bulleted item. This line is confusing to me... maybe the parenthetical statement should be after the word "posts" rather than after the word "queues"?

6. Your screenshot above is lacking an Oxford/serial comma between "Edit" and "and Leave open". My screenshot is also lacking an Oxford/serial comma on the last bulleted item between the listed badges "Reviewer" and "Steward". AFAIK the network uses Oxford/serial commas everywhere in their official copy.

Answer 13

status-completed

I have a problem with the Late Answers modal:

First of all, it should probably state that you need to choose "I'm Done" when you're done, but we can let that slide as it's pretty obvious.

I'm more worried that it doesn't say anything about skipping and makes it look like those are the two options: action on the post, or looks ok.

That worries me because it only takes one such action to remove the post from the queue. That means that unsure reviewers might wrongly remove the post from the queue before it gets a proper review.

Is that intentional and relies on the general skipping notice of the welcome banner? Shouldn't we emphasize that skipping is fine?

Answer 14

feature-request

Can we please get the same onboarding for the 10k-tools, seeing as they are under the same dropdown menu in the UI.

Answer 15

bugwordingstatus-declined

In the modal for the close vote queue it says:

"Most posts require 5 votes to close it."

On SO, the 5 is replaced with a 3, which is how it should be. However why emphasize that there are exceptions, binding moderator votes or gold badge duplicate closures, here in this modal. These are exceptions to the general rule that a post requires 5 close votes, or any other number of votes set for that site, in order to be closed.

Emphasizing that there are exceptions here, will lead to questions. Questions like: What will cause a post to require less votes?

This all deviates from the intent of this onboarding, making users known with how to use the system. As there also is no way in which the review system lets you see if your vote did close a question or not, it is also not relevant to the system of reviewing that there are exceptions to the number of close votes needed.

So, can we please reduce possible confusion and simply state:

"It takes 5 votes to close a question"

Answer 16

feature-requeststatus-completed

The dialog for the reopen queue has the following text:

posts whose bodies have been edited within 70 days of closure (the "on hold" period)

However, the "on hold" period is no longer a thing since the rollout of the new post notices.

This text seems to have been taken from an old revision of the general review queue FAQ here on Meta.SE. The text in question is a reference to the former behavior in which questions that were closed would show up as "on hold" instead of "closed" for the first five days of closure, to emphasize that they could be edited and reopened. However, as the first post I linked says, that behavior was removed, and "on hold" is no longer shown anywhere in the system.

Can the parenthetical note about the "on hold" period please be removed, as it only serves to make confusion since that period hasn't existed for a long while now?

Answer 17

feature-request status-completed

The welcome message for the low-quality queue says:

This queue contains posts that the system or other users have flagged as potentially problematic.

You can choose Looks OK, Edit, or Delete.

How to work through this queue:

• Consider leaving a comment when appropriate.
• Delete new answers to old questions that are redundant and add no additional value to the post.

Read more about Low quality posts in our help center.

The only specific advice is:

Delete new answers to old questions that are redundant and add no additional value to the post.

I find this very surprising. While deleting such answers is one of the queue’s (disputed) purposes, it is certainly far from the most frequent or important one. I suggest to replace the guidance with something like (assuming that you manage to finally remove questions from this queue):

How to work through this queue:

• Leave a comment or edit when appropriate.
• Delete posts that do not attempt to answer the question, are incomprehensible, or contain nothing valuable but a link.
• Choose Looks OK for posts that attempt to answer the question, even if you consider them wrong.

Answer 18

feature-request status-completed

Since the modals link to the "How do I use the [X] queue" page, can we change the text:

To "Read more about the [X] queue in our help center", since we want to know about how the queue works and not the suggested edits themselves?

Answer 19

feature-request status-norepro

The sentence on the daily review limit blanketly states:

You can review 20 tasks a day per queue.

However, this isn't always correct, even on non-Stack Overflow sites. The review queue size limit can increase to 40 for a particular queue if its size is larger than a threshold (150 on Stack Overflow, 1,000 on other sites). Additionally, one can review suggested edits to their own posts even if they've already reached the limit in the suggested edits queue, allowing them to exceed the limit in that case.

Can the sentence please be changed on all sites to:

You can generally review 20 tasks a day per queue. However, if there are enough items in a specific queue pending review, you'll be allowed to review 40 tasks in that queue.

Update: Glorfindel commented that the parenthetical note "(40 in larger queues)" shows up on Stack Overflow only. However, the queue size being 40 is not a Stack Overflow-only thing, it can also happen on other sites: it's just seen more often on Stack Overflow due to their reduced threshold of 150 pending tasks instead of 1,000. Also, a "larger" queue can become small enough one day to only allow 20 review queues, so I think it's better to explain when and why the size becomes 40, as my text says.

Answer 20

bug

Minor point, but you will also get the welcome banner to appear even if you don't have review privileges. Just visit the review page of a site where you're below the threshold. You can even dismiss it and, as I tested, when you go back, it stays gone. Which means that on the day that I finally do get review privileges for that site, I presumably won't see the welcome banner.

On a related note, you also will see the welcome banner on review pages on sites where you're not even registered. I consider these very low-level low-priority bugs, but felt they should be noted so atleast they are somewhere in the queue.

Answer 21

discussion wording status-completed

The current text in the VLQ queue doesn't seem accurate at all with prevalent guidelines, and the fact that the post might be "an answer to an old question" is more appropriate to "Late answers" queue than VLQ.

Answer 22

bug status-planned

The review guidance for Close Votes says:

• If enough people vote to reopen a question, recommend closure flags will be declined.

This is a bit strange, since we can't vote to reopen a question which isn't closed yet. Even

• If enough people vote to leave a question open, recommend closure flags will be declined.

won't work; if there's only one user who votes to close with the same close reason, that's enough for the flag to be marked helpful (even if the question isn't closed in the end). But at least it's closer to the actual behaviour than the current version.

Answer 23

supportsuggested-edits

_{until confirmed, using the [support] tag - not sure if this is by-design}

Something definitely shifted in the sorting of the review tasks in the Suggested Edits review queue. Previously, they were interspersed between suggested edits on questions and answers, but today they are exceptionally closely grouped.

This is how the review history looks like for today (orange rectangles highlight the wiki and excerpt suggestions, the green ones - normal tasks). What's more, I also had a suggested edit of mine on a tag wiki approved that was pending for more than a month:

Compare that to a similar review history from April, 10th and see the difference:

I do not mind the change being heavily invested in tags on SO if that is intentional, but there is nothing in the announcement or the change plan post that could indicate the change is intentional.

Answer 24

triagebugwordingstatus-completed

In the triage review modal there is the following text:

The primary goal of this queue is to quickly sort potentially problematic questions for further review.

You can choose Looks OK, Needs editing, or Flag questions.

How to work through this queue:

- 3 users must say a post Looks OK for it to publish.

• Choose Needs community edit if you could personally improve the post.
• Choose Needs author edit for questions that can’t be answered as-is.
• Flag posts that may need to be closed.

_{emphasis mine}

This should either be rewritten as:

3 users must say a post Looks OK for it to publish to the front page.

Or as:

3 users must say a post Looks OK for it to be cleared from the triage queue.

Seeing that a post does get published when it goes up for triage, just not on the frontpage. When one browses a specific tag page the post is visible, according to the info in this MSO topic

Answer 25

feature-request status-declined

Applicable dialog(s): pretty much all of them: Close Votes, Reopen Votes, Low Quality Posts, First Posts and Late Answers.

Shouldn't the point of the dialogs be to explain when reviewers should take which action? As in try to get people to review correctly and effectively.

Currently this part of the dialogs are somewhat vague or nonexistent:

Close Votes: If a question is off-topic for this site or requires additional work, it may need to be closed. [No explanation of any close reason, nor when you should edit]

Reopen Votes: None. [No explanation of when to reopen]

Low Quality Posts: Consider leaving a comment when appropriate. Delete new answers to old questions that are redundant and add no additional value to the post. [No mention of edit or close or other reasons to delete]

First Posts: Interact with the post by voting, editing, commenting, flagging, closing, and deleting. Perform as many actions as you see fit. Choose No action needed if the post is fine as-is. [Decent, but might not be that useful if you don't know when to vote, edit, comment, etc.]

Late Answers: Interact with the post by voting, editing, commenting, flagging, and deleting. Perform as many actions as you see fit. Choose No action needed if the post is fine as-is. [Same as First Posts]

This is explained in more detail when you click "Read more", but it seems important enough to be featured prominently instead of being put behind a generic link.

Explaining things like filters is okay, but things that happen behind the scenes (like how many votes are required to close something) seems like supplementary information at best and should probably only be behind the link.

---

Suggested Edits is closer to what I would consider good:

• Edits should maintain the post author’s original intent.
• Reject edits that are spam, attempt to reply to the post author, or clearly worsen the post.
• Even small changes can be good edits! Choose Improve edit if a post could use more changes.

Answer 26

feature-requestwording status-completed

For the close vote review the modal states:

"If a question is off-topic for this site or requires additional work, it may need to be closed."

Can we please highlight here that only posts that require additional work from OP should be closed? Any other post that requires work in the form of lay-out or grammatical editing should be fixed by the community / the reviewer rather then be closed.

Answer 27

bug

The "learn more" link (button, to be precise) opening the modal only works on the "review tasks" tab, but still present and clickable on the "stats" and "history" tabs. It should either be made clearly disabled (or removed) on these tabs, or, better, result in the popup being shown.

Answer 28

I really like the hand waving icon, it's funky and keeps with the funny themes SE has opted for over the years.

I do want to ask what the other icon means? Is it a party hat? An announcement speaker? Both combined?