Review queue workflows - Final release

Question

All good things must come to an end and so must the review queues project. This project would not have been successful without the community. Thank you all for the hours of reviewing, testing, and sharing your feedback along the way. With that said, this will be our final major release. In this post, I’ll summarize the changes we’ve made to the overall queue workflows and the specific adjustments to the Reopen votes queue.

Changes to the First posts queue

First posts → First questions AND First answers queue

• To avoid context switching, and recognizing that asking and answering require slightly different skill sets, we are creating two separate queues to handle the individual post types.
• The purpose of these queues remains the same – helping new users learn how to use the site. With that in mind, reviewers will be able to choose from three actions: Looks OK, Edit, and Share feedback.
• It can take time to develop the skills to write a good question or answer. A user’s first few posts may require extra attention and to help new users get the hang of things, First questions and First answers will accept up to three posts per user if their first couple aren’t successful.
• Reviewers are encouraged to leave comments for the post-author, but now they can also choose from a few options of canned feedback which will appear as a comment from the Community account.

Changes to the Reopen votes queues

1. “Significant edit” option

   

   • This feature appears when editing a closed question and allows editors to indicate if they’re making a minor edit (i.e. grammar, spelling) or a significant edit. By submitting an edit for review, the editor agrees that their edit attempts to resolve the question’s close reason(s) and should be considered for reopening in the Reopen votes queue.

   • Once a significant edit has been submitted, editors are still able to return to the edit screen and make further adjustments. Significant edit status will also be indicated in both revision history and post timeline pages.

   

2. Significant edits in the Suggested edits queue

   • Users who do not have enough reputation to submit an edit outright will have their significant edits go into the Suggested edits queue accompanied by a post notice.

   

   • Actions in the Suggested edits queue will change based on whether the edit has been flagged as a Significant edit. In the case of a Significant edit, reviewers will be able to Approve and reopen or Approve and leave closed.
   • It takes two approve votes (any combination of Approve and reopen or Approve and leave closed) to remove the task out of the Suggested Edits queue. If at least one of the votes is Approve and reopen, a new task will be created in the Reopen votes queue.

3. Leave closed reasons

   • If a Reopen votes reviewer elects to leave a question closed, the reviewer will be presented with a “Leave closed reason” modal (similar to the close reason modal) that will return feedback to the editors. For example, if a question was originally closed because it lacked details, it may still require further details even after an edit, or new issues with the question might present themselves like asking too many questions at once.

   

   • In addition to inbox notifications, the closed question’s post notice will also be updated with more information.

   

4. Multiple opportunities to send question for review

   • We’ll be launching this feature without any limits – users will be able to submit multiple edits on individual posts. We’ll take a look at the data in a few weeks to see if any abuse of the feature has occurred and take mitigation steps as needed.

   • [Stack Overflow only] Reviewers will not see posts that

     1. they’ve voted to reopen since last closed, and
     2. voted to close since the post was last edited.

     Moderators are exempt from this restriction.

5. Bug fix: “Don't cause the reopen vote of the user, whose edit pushed the closed question into the reopen queue, to invalidate the review”

   This bug fix is also included as part of this release. You can read Kyle’s detailed response here.

Other changes to the Review queues

1. Turn off the Help & improvement queue

   • H&I has been our least productive queue. Over the last 90 days, only 9 tasks went into the queue each day on average, compared with 2,215 for First Posts (our busiest queue) and 188 for Reopen Votes (our second-to-last busiest). Consequently, this queue will be turned off. It will remain in the listing until its current items are exhausted, and will then be removed. History for and badges awarded from this queue will remain as they are.
   • With the addition of the First questions queue, we expect that questions with potential can get the same assistance they need more quickly.
   • Questions that would have ended up in this queue from Triage (with a Needs community edit action) will now enter the First questions queue.

2. New actions for Late answers queue

   The Late answers queue has been successful in identifying and removing low quality posts, repeat answers, and spam. Reviewers were able to take any number of actions on a task, so the objective of this queue wasn’t all that obvious. To clear up any ambiguity and focus on identifying problematic answers, we’ve defined three actions: Looks OK, Edit, and Delete.

3. Keep Triage queue (Stack Overflow only)

   • Triage will continue to handle questions only.
   • Triage will be available on all sites, but only turned on by default on Stack Overflow. Have a meta discussion and ask a moderator to status-review if your site wishes to add the Triage queue.

4. Name change: Low quality posts → Low quality answers (Stack Overflow only)

   This queue will continue to handle answers only.

5. Misc. We won't issue new Reviewer or Steward badges for the queues that were turned off, but this won’t have any effect on badges already earned in these respective queues.

   Along with all these changes, here is an updated visual of the Review queues map –

   

What about issue X, Y, Z?

As mentioned earlier, this is our final major release. We will not be pursuing the proposed suggested tasks feature at this time. We didn't feel that it was going to significantly increase curation activities on the site and therefore didn't warrant the high development cost.

There are still a few outstanding issues related to suggested edits that have proved to be technically challenging. We’ll respond in those original Meta request posts as soon as we decide on a course of action.

Feedback

Please leave your feedback and any bugs you may discover related to this release below this post. We will be monitoring this post until Friday, September 10th. Report any further issues after September 10th as new questions on Meta.

Answer 1

support

Turn off the Help & improvement queue

Can the company give badge stats of the H&I queue now that it's closing? (If the review statistics for the queue stop being visible it would also be good to have them posted.)

SEDE doesn't reveal for which queue badges were awarded. I think reviewers who were active on the queue would enjoy knowing the final breakdown.

Answer 2

bug status-completed

Missing space between OP and OP's reputation

At least in the Close queue, the UI now shows the OP and their reputation just below the title of the question. There is currently no space between the two, which looks pretty crowded.

Answer 3

Sorry, but FWIW I hate the new thingy. Too many steps, but mainly too locked-in to a hard-coded sequence of steps, with no turning back. Bad UI design, IMHO.

One simple example (and yes, similar to what some others have said): You must decide up front whether to OK a post or edit it. Once you choose Edit you're done - no going back. An Edit apparently implies OK after editing. No, it doesn't. You might want to skip or close or edit again or...

Why would you need to decide the ultimate fate before being able to try to improve the post? We used to be able to edit, comment, save, wash, rinse, repeat,... whatever -- before deciding on a review fate.

That's just one tiny example. This seems to have been designed by someone who's never used an editor. Reviewing involves editorial actions. The hard-coded flowchart here is horribly misguided, not to mention a time sink and a general bother.

It wasn't broken. Shoudn'ta tried to fix it. Or shoulda gotten a high-school intern to improve it.

But maybe the real aim was to get cut-and-dried metadata about reviews, rather than let humans use their brains. Multiple-choice on steroids, for automatic pigeon-holing? The result will be poorer quality reviews, and likely an abandonment of some reviewing.

Just one opinion.

Answer 4

bug

Similar to this bug which has already been fixed: if you go to an open First Answers or First Questions review item in a private browser window, you don't see the "Please log in to review First Answers" banner you'd see in e.g. Close Votes, but an empty banner. Also, the post is grayed out.

Answer 5

feature-request status-completed Related to this other feature request.

Since the First Posts queue is retired, can the "How do I use the First posts queue?" link be removed from the Help center's general preview? It would still be accessible from the "View more" link, but now there'd be no more "obsolete" info on a page that's frequently linked to, or looked at by, new users; which sounds slightly better in terms of UX.

It could be replaced in the general preview by a link to the Close votes queue help page; linking to First Questions without having its twin "First Answers" on the same page would seem unbalanced.

Answer 6

Name change: Low quality posts → Low quality answers (Stack Overflow only)

This queue will continue to handle answers only.

Why only on Stack Overflow? In several discussions on the purpose of having low-quality reviews for questions (as opposed to close reviews), the only (reasonable) answer was that it makes sense on Stack Overflow due to its scale. Everywhere else, it only slows the process down since another step is needed to bring the question to the close queue. After all, closing (or improving) is all that can be done with low-quality questions anyway. Also see: Why can questions be flagged but not closed for very low quality?

With low-quality reviews for questions being gone on Stack Overflow, what reason is there to keep it at all? So, why not send all questions flagged or auto-flagged as low-quality to the close queue directly?

Ceterum censeo: Remove the very low quality flag for questions (or altogether), since other flags already serve the same purpose.

Answer 7

bug

First Question queue audit

On this audit. All I want to do is comment to ask for more details - more focused than the canned answers.

So I add a comment and then when I hit save, a box pops up, saying this is an audit. So there isn't any use for me to exit and a good way to find audits which is not good.

This is a more general issue. The share feedback button needs to give more options or allow us to enter a comment.

Answer 8

bug

Missing share link

The share link under the post is missing:

There is only follow and flag. It was there previously in the first posts review and is same present in the late answers review:

Answer 9

bugstatus-completed

The "Staff" label is not aligned with the user detail in the review queue.

Answer 10

bug

In the First questions queue, if there is a pending suggested edit on the post and you press the inline edit button after reviewing the suggested edit you are taken out of the First questions queue and find yourself in the Suggested edits queue.

Happened to me in this post.

Answer 11

The "Flag" option is missing in the Low Quality Posts queue:

This option is important even in this queue since it may be necessary to flag for a moderator's attention (e.g., to remove PII, convert to a comment, notify the moderators that user(s) are being rude, etc.). Without this option I have to exit the review to raise the flag, but users should never need to exit the review in order to properly review it.

Frankly, I don't understand why any of these options are removed from the post in the review queue in the first place -- it's not like they take up a lot of space.

Answer 12

The Reopen Votes queue lacks an option to add a comment to the post, edit it without reopening it, or flag it:

Several times I have wanted to add a free form comment to tell the querent why the post still isn't acceptable (perhaps the post had multiple issues and the querent only addressed one of them). Without an option to add a comment to the post, however, I have to exit the review to help the querent with a comment.

Similarly, there is no flag option. What if the querent added some rude text in frustration over the closure of the question, and I want to raise a flag to notify the moderators that they need to keep an eye on (and perhaps mod message) the user? To do that I again have to exit the review. And what if I want to edit out the rude text myself without voting to reopen -- again, I have to exit the review to do that.

Once again, I don't understand why the review queues are missing options normally visible on a post -- they don't take up much space but are sometimes needed.

Answer 13

feature-requeststatus-completed

Adding "Edit and share feedback".

Say that you are reviewing a post that has too much code. You can edit out the unnecessary code, but you also want to share feedback, so the user knows why you edited and hopefully won't make the same mistake again with future posts. To do so, you would have to open the post in another tab, click "Share feedback", then edit it in the other tab.

Adding an this option would make situations like this a lot less cumbersome and would also encourage such behavior.

Answer 14

feature-request

I would like to see the all queues link somewhere (maybe at the top, next to the title, or like a button/tab) on each review page as a quick way to choose next queue where I can participate.

Currently I have to use top bar, tap Review queue icon, then select "all queues" link.

Answer 15

bug

In the First Questions & First Answers queues, the Flag and Edit post menu items are missing in from posts shown on the completed/ non-active review page:

These options were added back to posts that you're actively reviewing, but they're still missing in this case.

Answer 16

bug

Thank you for fixing the bug that prevented the modals in the two new queues to be movable.

Unfortunately, the new "leave closed" modal seems to have the same problem:

Answer 17

bug - duplicate of Navigating to review /stats and /history causes the URL to change to a specific review item

The URL for the Stats or History for a review queue changes from /review/{queue}/stats or /review/{queue}/history to /review/{queue}/taskID once the page is loaded.

Clicking the refresh button then loads the review task rather than refreshing the stats/history page.

This seems to happen across all queues where I am able to perform more review tasks (i.e. where I've not hit the daily limit or where there are tasks available to perform).

Reproduced on { Stack Overflow and Meta Stack Overflow }, tested on { Chrome 93.0 on Android 11, Chrome 93.0 on Windows 10, and Firefox 92.0 on Windows 10 }.

This also breaks navigation, e.g.:

• Navigate to History (the page ends up with the URL /review/{queue}/taskID)
• Click on a review task in my history
• Press back
• Instead of going back to my history, /review/{queue}/taskID is loaded instead.

Answer 18

As a heads-up, since Increase close and reopen vote weight for tag badge holders - Meta Stack Overflow has not been implemented yet, I'm still on the strike.
Moreover, there's no mention of if it was even considered.

Reducing the amount of needed close/reopen votes from 5 to 3 was helpful, but this would reduce it even further -- since as was shown, the power of reviewers at SO is too inadequate for the workload for even that to fully compensate.

Answer 19

I came across an answer in the First Answers queue to an off-topic question. The question was asking about recommendations for software on Stack Overflow, and someone answered it. What now? The suggestions in Share Feedback hardly seem relevant; the answer is clear and doesn't need more information. Is Looks OK the proper response to this?

Answer 20

bug design

There is weird word wrapping on some queues for non-English sites, e.g. here:

Answer 21

status-completed

Review first answers

1. opened review queue
2. skipped few answers
3. found one non-answer
4. flagged it as being non-answer
5. now what ?

Answer 22

The "Other Action" option is great! I'm glad you added it. However, I disagree with voting causing this to be selected for two reasons:

1. When a post is upvoted, the reviewer should choose "Looks OK", not "Other Action"

If I choose looks OK and then upvote, the other action option is automatically selected, so I have to take another step and click "Looks OK" for the second time. This seems to either discourage upvoting good posts or choosing "Looks OK" on good posts.

2. Voting is separate from reviewing

A post can fundamentally be OK, but you can disagree with it (as is the case with most MSE posts, for example). Voting is a personal opinion, not an alternative to reviewing. By having voting select "Other action", voting becomes a review action.

Answer 23

status-declined

Thanks for the update! Regarding, the "Leave closed" option in the Reopen Review Queue, I'll repeat the feedback that I provided in a comment to another post:

If the question was put in the Reopen Review Queue because of an edit, the listed options should include a "The edit did not resolve the original close reason" option and it should be checked by default.

You've indeed added that option, which is great! Can we please have it checked by default though? I imagine that will be the option that is chosen in most cases and saving a click goes a long way :)

I know that there are good arguments against this but just to put it into perspective, in the old design of the Reopen Review Queue, one only needed to click one button to say that the question should remain closed:

Currently, it requires 4 clicks just to communicate the fact that nothing substantial has changed about the question:

Answer 24

In the new version Late Answers, the Recommend Deletion radio button leads to a dialog that sort of repeats the flag link.

For example, for a link only answer, you can both use the flag link to flag NAA as one used to do (and then the answer would be shunted off to the Low Quality Posts queue),

and, in the new version of the Late Answers review, also click the Recommend Deletion radio button and then select "Link only answer (not spam)".

Why the duplication? This confusing UI leads to a poor UX.

I'm sure that it wasn't like this previously.

Answer 25

feature-request status-review

Due to comments and downvotes, I want to make it clear: the following suggestion/request is based on the current implementation of things and assuming they stay that way. If there are going to be changes in the design of the queues that request might become obsolete.

---

In the Late Answers queue:

There is now both an action for edit (red square), and also the edit link under the answer (blue square):

Please remove the one under the answer (blue square in the image) to avoid confusion (and be consistent with the new First Q/A queues). I accidentally pressed that in a review and when I finished editing I was thrown back to the original page and out of the queue. Of course I could go back and mark as Looks Ok but it is quite annoying and I'm pretty sure it is bound to happen to others.

Answer 26

bug

This is now its own queston. Please direct any comments etc.... there.

If you reach your vote limit, you also reach your review limit! Don't know how active this thread is, and this is probably minor anyway.

Edit: This is a bit more serious than I thought. I'm now locked out of the Late answers queue! If there is no response here in the next day or so. I'll make a separate post about this.

Note: I can still review the First Questions queue.

Answer 27

Audit context expectations

I'm not sure where this fits, so I'll just give my feedback here for historical purposes and we'll see where it goes. I just reviewed a first answer and clicked 'Edit' to fix up its issues:

Now what I expect is that when that post is shown here, that all of the glaring grammar issues are sorted out. After all I did 'pass' my audit because I wanted to edit the thing, right? I don't see any previous editing effort here in the Congrats banner. So just for certain, I go to the answer itself. It's exactly the same. I fix it up. I've passed three audits now following this scenario.

However this seems strange to me. I agree that an edit should count as a pass. You only make that action if you are paying attention to what you are doing, and that's a proactive curative thing to do. PASS. But, as choosing 'Looks OK' correlates with a relatively large number of upvotes, it feels like 'Edit' should correlate with a good solid previous edit that's cleaned the post up. I have no idea how to tie together this context though. In fact, it's impossible I think.

1. You can't predict what a future edit will really do.

2. A grammatical analysis of a post is process heavy, and on motley sites like these with very specialized syntax and formatting for each, such an analysis is almost certain to flag up perfectly appropriate grammar as incorrect.

We may just have to all agree that this is the best we can do. Sure makes for an odd workflow though.

Answer 28

status-review

I'm still worried about the wording here. Edits should attempt to fix all issues with the post. While yes, the close reason is one, remember that posts can be closed by multiple reasons, just one is shown. Wording here should be less about the specific close reason and more about following the site guidelines like the message for the reviewers say.

Answer 29

bug

There is a typo in first question (answer) queue actions block:

It should be "Other actions" or "Another action".

P.S. English isn't my native language.