Review queue workflows - Final release

Question

All good things must come to an end and so must the review queues project. This project would not have been successful without the community. Thank you all for the hours of reviewing, testing, and sharing your feedback along the way. With that said, this will be our final major release. In this post, I’ll summarize the changes we’ve made to the overall queue workflows and the specific adjustments to the Reopen votes queue.

Changes to the First posts queue

First posts → First questions AND First answers queue

• To avoid context switching, and recognizing that asking and answering require slightly different skill sets, we are creating two separate queues to handle the individual post types.
• The purpose of these queues remains the same – helping new users learn how to use the site. With that in mind, reviewers will be able to choose from three actions: Looks OK, Edit, and Share feedback.
• It can take time to develop the skills to write a good question or answer. A user’s first few posts may require extra attention and to help new users get the hang of things, First questions and First answers will accept up to three posts per user if their first couple aren’t successful.
• Reviewers are encouraged to leave comments for the post-author, but now they can also choose from a few options of canned feedback which will appear as a comment from the Community account.

Changes to the Reopen votes queues

1. “Significant edit” option

   

   • This feature appears when editing a closed question and allows editors to indicate if they’re making a minor edit (i.e. grammar, spelling) or a significant edit. By submitting an edit for review, the editor agrees that their edit attempts to resolve the question’s close reason(s) and should be considered for reopening in the Reopen votes queue.

   • Once a significant edit has been submitted, editors are still able to return to the edit screen and make further adjustments. Significant edit status will also be indicated in both revision history and post timeline pages.

   

2. Significant edits in the Suggested edits queue

   • Users who do not have enough reputation to submit an edit outright will have their significant edits go into the Suggested edits queue accompanied by a post notice.

   

   • Actions in the Suggested edits queue will change based on whether the edit has been flagged as a Significant edit. In the case of a Significant edit, reviewers will be able to Approve and reopen or Approve and leave closed.
   • It takes two approve votes (any combination of Approve and reopen or Approve and leave closed) to remove the task out of the Suggested Edits queue. If at least one of the votes is Approve and reopen, a new task will be created in the Reopen votes queue.

3. Leave closed reasons

   • If a Reopen votes reviewer elects to leave a question closed, the reviewer will be presented with a “Leave closed reason” modal (similar to the close reason modal) that will return feedback to the editors. For example, if a question was originally closed because it lacked details, it may still require further details even after an edit, or new issues with the question might present themselves like asking too many questions at once.

   

   • In addition to inbox notifications, the closed question’s post notice will also be updated with more information.

   

4. Multiple opportunities to send question for review

   • We’ll be launching this feature without any limits – users will be able to submit multiple edits on individual posts. We’ll take a look at the data in a few weeks to see if any abuse of the feature has occurred and take mitigation steps as needed.

   • [Stack Overflow only] Reviewers will not see posts that

     1. they’ve voted to reopen since last closed, and
     2. voted to close since the post was last edited.

     Moderators are exempt from this restriction.

5. Bug fix: “Don't cause the reopen vote of the user, whose edit pushed the closed question into the reopen queue, to invalidate the review”

   This bug fix is also included as part of this release. You can read Kyle’s detailed response here.

Other changes to the Review queues

1. Turn off the Help & improvement queue

   • H&I has been our least productive queue. Over the last 90 days, only 9 tasks went into the queue each day on average, compared with 2,215 for First Posts (our busiest queue) and 188 for Reopen Votes (our second-to-last busiest). Consequently, this queue will be turned off. It will remain in the listing until its current items are exhausted, and will then be removed. History for and badges awarded from this queue will remain as they are.
   • With the addition of the First questions queue, we expect that questions with potential can get the same assistance they need more quickly.
   • Questions that would have ended up in this queue from Triage (with a Needs community edit action) will now enter the First questions queue.

2. New actions for Late answers queue

   The Late answers queue has been successful in identifying and removing low quality posts, repeat answers, and spam. Reviewers were able to take any number of actions on a task, so the objective of this queue wasn’t all that obvious. To clear up any ambiguity and focus on identifying problematic answers, we’ve defined three actions: Looks OK, Edit, and Delete.

3. Keep Triage queue (Stack Overflow only)

   • Triage will continue to handle questions only.
   • Triage will be available on all sites, but only turned on by default on Stack Overflow. Have a meta discussion and ask a moderator to status-review if your site wishes to add the Triage queue.

4. Name change: Low quality posts → Low quality answers (Stack Overflow only)

   This queue will continue to handle answers only.

5. Misc. We won't issue new Reviewer or Steward badges for the queues that were turned off, but this won’t have any effect on badges already earned in these respective queues.

   Along with all these changes, here is an updated visual of the Review queues map –

   

What about issue X, Y, Z?

As mentioned earlier, this is our final major release. We will not be pursuing the proposed suggested tasks feature at this time. We didn't feel that it was going to significantly increase curation activities on the site and therefore didn't warrant the high development cost.

There are still a few outstanding issues related to suggested edits that have proved to be technically challenging. We’ll respond in those original Meta request posts as soon as we decide on a course of action.

Feedback

Please leave your feedback and any bugs you may discover related to this release below this post. We will be monitoring this post until Friday, September 10th. Report any further issues after September 10th as new questions on Meta.

Answer 1

bug status-completed

Verified in first answers, not sure if it holds elsewhere but I imagine it does.

I stumbled over a link-only answer, and my first instinct is to downvote and flag. Voting is fine. The flag menu, however, acts as if the post is deleted (which, in this case, it was), only giving "in need of moderator intervention". Only deleted posts give one option, which is a dead giveaway that the post is an audit.

Aside being a dead giveaway that the post is an audit, combined with the lack of being able to flag and then go on to "I'm done" means there's not a sane way to end the review, even if the flag dialog was present. I've flagged a couple real answers and I have no idea what to press afterwards. There's no applicable feedback, "looks OK" is wrong, and so is edit. Allegedly, this part is supposed to be fixed, but I don't see any change in the available options, and flagging isn't enough to complete the review (which is a massive regression, in my opinion. You used to be able to vote and flag, and pick "I'm done" to move on).

Ended up clicking "share feedback" just to end it, which also doesn't make sense as a pass case for this particular type of posts given the current feedback options. There's not really any good way to handle audits at the moment, though. Here's the audit in question (10k+? on Stack Overflow), for anyone interested.

Answer 2

feature-requeststatus-review

Undo accidental wrong feedback

I stumbled across question that needs to provide some codes, so I clicked the share feedback button:

I wanted to click "Question needs some code", but I accidentally clicked the "question is too broad" option and shared the wrong feedback to the user.

Is there a way I could undo the feedback? Because I can't seem to do anything with the comment.

Answer 3

status-completed

https://stackoverflow.com/review/first-answers/29670070

After flagging:

I got the 'Thanks. We'll look at it' banner. And that's it.

I also left a comment, yet still have the 'Share feeback' option available. I had no choice but to Skip this as there was no other valid option.

Answer 4

bug status-completed

In the new First Questions review queue, if you cast a close vote (i.e., by flagging a question) and it passes the threshold to actually close the question, then it will refresh the page to include the closed message, and add:

You are not able to review this item

It makes perfect sense that a question that is now closed would be removed from the review queue. And it makes sense to provide immediate feedback to the flagger that the question is now closed. But I would also expect that voting to close a question would qualify as already having reviewed that question.

I assume this only applies to people who have the cast close and reopen votes privilege (3,000 reputation).

Answer 5

status-review

Don't know if you are aware of that but currently, Triage blocks questions from getting to close queue for about a day average. I think this is a bit too much:

...we get the situation when system correctly and quickly detects most troublesome questions and then, essentially shields them from getting to close queue - for a full day average, go figure.

This is really really weird. The way how things work now, system essentially incentivises askers of poor questions to pay money for fraudulent release of asking limits. Seeing multiple poor questions hanging open and seemingly unattended for days can only convince them that their ban is mere misfortune and that it makes sense to try their luck again...

I would appreciate if you could do something about that.

Probably least effort consuming option to address above issue would be to allow questions getting to close queue while they are on triage (possibly with some reasonably small delay, if you prefer).

Answer 6

feature-requeststatus-review

Can the 'Edit' option be disabled if you don't have the privilege to edit all posts and suggesting an edit is not possible (because the suggested edits queue is full, or you have already too many pending suggested edits)?

Answer 7

The removal of the "I'm Done" option from the Late Answers queue raises some issues:

We can no longer:

• Vote up/down
• Flag as Rude/Spam or for moderator attention
• Leave a comment

As a review action.

If this is not intentional - please add an "I'm Done" button or anything similar.

If this is intentional (maybe these are not intended to be actions in this queue and I missed that) - then this queue falls back to be identical to the Low Quality Answers review queue. Is this intended? It certainly feels like a weird and confusing duplication. You said, regarding the Late Answers queue:

The Late answers queue has been successful in identifying and removing low quality posts, repeat answers, and spam. Reviewers were able to take any number of actions on a task, so the objective of this queue wasn’t all that obvious. To clear up any ambiguity and focus on identifying problematic answers, we’ve defined three actions: Looks OK, Edit, and Delete.

To me, the objective of the queue is even less obvious now. If its purpose is to remove low quality posts - we already have a queue for that. If its purpose is to remove repeated answers - we didn't really get the tools for that (not NAA and none of the recommend deletion options applies). If its purpose is to remove spam - we could already flag spam before and it doesn't really require a deletion option (enough flags will delete the post anyway).

So, what is the current objective of the queue?

Answer 8

Anonymous comments: They're bad

The thing I like about reviewing on SE, and why I think it works so well, is that it's human. You can ask what's wrong with your post, and quite often get helpful feedback pretty quickly. There's transparency, so you can see exactly what's going on with your post (and who's doing it).

Giving reviewers the option to post anonymous feedback goes very much against this. It sounds like a good idea on the surface, but all it does is introduce issues. When I comment under a post with things that need to be fixed, I have never once had second thoughts because of worries about my name being associated with that. I put a lot of effort in to ensure my comments are helpful, and if I'm asked for clarification I want to be there to help.

Anonymous comments mean the feedback given will always be a bit too vague to really be helpful. One size fits all doesn't fit anyone. Even worse, if the post author cares enough to try to improve it, they can't reach me to get help doing that. To add on to the problems, if these comments are misused, there's currently no way to flag them (as far as I can tell). That's not rare either, I've seen it multiple times already (not maliciously, just incorrect usage).

If automated, useless comments become the norm for reviewing, it'll only reinforce the idea that SO is unwelcoming to new users. Robots are...robotic. Nobody who's new to the site wants to be greeted by being told there's something generally wrong, without being told or being able to ask for specifics.

If these comments were only ever used when appropriate, they'd be fine. But the problem is, the people who use these canned comments often will be the same people who won't take the time to actually weigh whether or not leaving specific feedback would be better. It just gives reviewers the option to be lazy, in a part of the site where laziness will 100% lead to a worse site all around.

This feature feels like a poorly thought out tool to help review grinders grind faster, and it doesn't feel appropriate for any site other than SO (where effiency and quantity of reviews is necessarily much more of a priority than other sites). Please, please get rid of it.

Answer 9

bug status-completed

Message for reaching daily quota is blank for the "First questions" queue

Today, I reviewed twenty First questions and as much Suggested edits at the Mathematics Stack Exchange site. Now, when I look at Review queues page, I see that there is a light blue rectangle below Suggested edits (see the picture below), in which it is written “Thank you for reviewing 20 suggested edits today; come back in 8 hours to continue reviewing.” So far, nothing is new. But below First questions there is also a blue rectangle… with nothing written in it.

Answer 10

feature-request

Display site-specific close reasons in the Reopen queue

When reviewing a question that was closed for a site-specific reason, the display in the reopen queue simply shows (e.g. on Stack Overflow):

without stating if it was closed for "seeking recommendations", or for "needing debugging details", etc.

This means if I decide to leave the post closed, I need to navigate to the post in order to see which reason was used, in order to decide whether "the original close reason was unresolved" or if I need to pick a different reason.

These additional actions are time consuming. Also, I've been tempted to just choose "the original close reason was unresolved" without verifying what that was, which is not ideal.

Please display this information on the page itself, preferably just below the above message.

Answer 11

status-planned

One quick thing, everything looks good on a glance through! Can the First Posts queue be moved out of the way to a section at the bottom like what is done with "Retired Badges"? Similarly "First Posts" now probably doesn't warrant showing in the review queue drop down from the top bar.

Answer 12

bugstatus-completed

On Stack Overflow, if I go to the First Questions or Answers queue, I see this:

Because I don't have the required reputation to review in that queue. But the empty blue banner should look like this, as it does in the other queues there (which I also don't have access to):

As it does in the other queues, e.g:

This also happens if a sub-X reputation user looks back at previous review tasks such as this, which are useful if you need to flag as NAA-, spam, etc. Again, this only happens with the First Answers or Questions review queues, not the Late X ones.

Answer 13

feature-request status-completed

Phrasing nitpick(s):

The Help center page for the Low quality queue says:

Note: On Stack Overflow, low quality answers are exclusively handled in the Low quality answers queue. You will not see questions in this queue.

This sounds like it's saying that the Low quality answers queue is the only place that handles low quality answers (when really, answers are also handled in other queues), when really it's that the queue only handles answers. It should probably be something like:

Note: On Stack Overflow, the Low quality answers queue exclusively handles low quality answers. You will not see questions in this queue.

---

Also, ideally "Low-quality posts" and "Low quality answers" would be hyphenated the same way. I think the technically correct answer is hyphenated, but it's really the inconsistency that bothers me here.

---

^{Yes, these are both super nitpicky.}

Answer 14

support/feature-request status-completed

Let individual sites decide whether they are a "code-related site" for the purposes of the code-specific canned Community comments

kristinalustig remarked that the code-specific canned Community comments are supposed to be available only on "code-related sites". However, it is unclear to me how it is determined whether a site is code-related. I'd like this to be an option that the sites can decide for themselves, as it is not always obvious whether a site prefers their posts to contain code. For example, posting code should generally be avoided on Computer Science. While a site probably has to decide only once whether they want to be 'code-related', it would be nice if this is a setting that e.g. the CM's can touch.

Answer 15

bugstatus-review

On the Code Review site one of the options for First Questions is Share Feedback. It has 2 options that mostly only represent Stack Overflow. The old workflow allowed any comment from the reviewer to be used as shared feedback but that is no longer an option. There are more reasons on Code Review on how a question can be improved.

The Community comments don't really represent the CR community and I can't flag it for moderator attention, and the moderator can't delete the comment.

I see this as a bug.

Answer 16

bugstatus-completed

Answers I have already reviewed in the First Answers queue reappear for me in the Late Answers queue. I believe this is not the intended behaviour, since in the past an answer I reviewed in the First Posts queue would no longer show up in the Late Answers queue for me. In any case, it does not seem to make sense for a single user to review the same post twice.

Answer 17

feature-requeststatus-completed

In the popup and the Help Center page for First Posts, mention that it's retired (and add links to its successors).

Answer 18

feature-request status-deferred

When I follow a post, and Share Feedback in one of the new queues, Community posts a comment on my behalf. Because it's a comment not posted by me, I immediately get a notification in my inbox. Could those be suppressed please?

Answer 19

bug status-completed

I can't perform the most important action in "First answers queue". I can't flag the answer for deletion or vote to delete it myself. How can I push the post into "Low Quality Answer" review?

Answer 20

We have a problem at SO.es (I supposed Ja, Po, and Ru has the same issue)

On the first question queue, we have only 2 motives to close a question...

But in all international sites, we have another problem.. question that are not in the official site language...

So we need a third motive that let close because the question is in another language. This is important for us.

Answer 21

Honestly not sure if this has been covered already, but it looks like the link to get back to the main Review Queues dashboard has disappeared from the individual queue pages—can we please bring that back, and maybe make it more prominent? Going through the menu is a bit out-of-the-way for me.

Answer 22

One feature I'm REALLY missing here is the possibility to use a custom comment when leaving feedback.

Sure, I can write a custom comment and then press Looks ok. But that would be wrong. I could write a custom comment, press Edit, make some small changes that are not really changes, and then save. But that would also be wrong.

Well, what I can do, as a really clunky workaround, is that I can choose one of the premade comments. Then I hit back in my webbrowser and click the question or answer I just reviewed. And then I delete the comment I chose earlier and write my own. It's possible, and everything turns out right. But it's a VERY clunky thing to do something that is what I want to do for 99% of the questions I don't seem fit for "Looks good".

Answer 23

bug status-completed

Selecting "Other Action" in First Questions or First Answers does not clear the post from the queue

I saw that the "Other Action" option was enabled yesterday for both the First Questions and First Answers queues. However, when I select that option, the review is not marked as completed in the post timeline, and it seems that the post stays in the queue till someone chooses one of the other options. Alternatively, there might be a minimum number of "Other Action" reviews expected for it to be cleared, but that has not been documented yet from what I see.

Example of post staying in queue after two "Other Action" reviews:

Example of post cleared after one "Other Action" and one "Looks OK" review:

Answer 24

bug status-completed

The How do I use the First questions queue? page says "answer"

Share feedback if the post has the potential of being a good answer, but cannot be improved by anyone but the post-author. ...

Answer 25

feature-requeststatus-deferred

Now that "Significant Edits" are firmly a thing, is there any possibility of Allow non-bumping minor edits, but review them on /review becoming a thing? Non-bumping edits would be a very useful thing to have, and it seems as though the "Significant Edit" feature would be the first step towards implementing this.

Answer 26

feature-requeststatus-deferred

Per Yaakov's recommendation, I'm posting this as a specific feature-request:

One of the review queue changes I was most excited to see during our research interview was the option to add an "ignored tags" filter... where we could include tags for questions we absolutely did not want to see review items for. However, it seems like it was completely abandoned almost as soon as it was announced. Please re-consider implementing this killer feature to the close & reopen queues.

Answer 27

A handful of suggestions for "Share feedback":

Question / answer contains image of code

Please post your code as text, not an image, so others can better understand or reproduce the problem.

Answer is only a link

A link to a solution is welcome, but please ensure your answer is useful without it: add context around the link and quote the most relevant part of the page you're linking to in case the target page is unavailable.

Answer doesn't attempt to answer the question

This does not provide an answer to the question. You can search for similar questions, or refer to the related and linked questions on the right-hand side of the page to find an answer. If you have a related but different question, ask a new question, and include a link to this one to help provide context.

Question / answer contains meta-commentary

This post contains content unnecessary to the post. Please make your posts only about the question you're asking / answer you're proposing.

Answer 28

feature-request status-review

First Answers queue - Link to other answers

I have a review where the answer just repeats another answer.

I would like to say this in a comment to the new answer but I can't as there are no share buttons on the Other Answers Page

Answer 29

feature-request status-review

Change the text of the "Submit" button to "Continue" if the user chooses a (cancellable) option that has a second step

This is especially useful for the "Share feedback" option (because it opens a modal dialog) but maybe the community should decide if it's a good idea to use it for other options as well (e.g., the "Edit" option).

I think it's more intuitive that way. It will also be in line with my other request about failing audits only at the last step.

Answer 30

bug status-review

I was reviewing the Triage queue.

When select Spam flag using the button below audit questions, the action is treated as if clicked Looks OK instead of Flag.

What happened:

It would be great if you can rectify the result in these audits,

https://stackoverflow.com/review/triage/29702035

https://stackoverflow.com/review/triage/29693259

Original Post for reference.