Review queue workflows - Final release

Question

All good things must come to an end and so must the review queues project. This project would not have been successful without the community. Thank you all for the hours of reviewing, testing, and sharing your feedback along the way. With that said, this will be our final major release. In this post, I’ll summarize the changes we’ve made to the overall queue workflows and the specific adjustments to the Reopen votes queue.

Changes to the First posts queue

First posts → First questions AND First answers queue

• To avoid context switching, and recognizing that asking and answering require slightly different skill sets, we are creating two separate queues to handle the individual post types.
• The purpose of these queues remains the same – helping new users learn how to use the site. With that in mind, reviewers will be able to choose from three actions: Looks OK, Edit, and Share feedback.
• It can take time to develop the skills to write a good question or answer. A user’s first few posts may require extra attention and to help new users get the hang of things, First questions and First answers will accept up to three posts per user if their first couple aren’t successful.
• Reviewers are encouraged to leave comments for the post-author, but now they can also choose from a few options of canned feedback which will appear as a comment from the Community account.

Changes to the Reopen votes queues

1. “Significant edit” option

   

   • This feature appears when editing a closed question and allows editors to indicate if they’re making a minor edit (i.e. grammar, spelling) or a significant edit. By submitting an edit for review, the editor agrees that their edit attempts to resolve the question’s close reason(s) and should be considered for reopening in the Reopen votes queue.

   • Once a significant edit has been submitted, editors are still able to return to the edit screen and make further adjustments. Significant edit status will also be indicated in both revision history and post timeline pages.

   

2. Significant edits in the Suggested edits queue

   • Users who do not have enough reputation to submit an edit outright will have their significant edits go into the Suggested edits queue accompanied by a post notice.

   

   • Actions in the Suggested edits queue will change based on whether the edit has been flagged as a Significant edit. In the case of a Significant edit, reviewers will be able to Approve and reopen or Approve and leave closed.
   • It takes two approve votes (any combination of Approve and reopen or Approve and leave closed) to remove the task out of the Suggested Edits queue. If at least one of the votes is Approve and reopen, a new task will be created in the Reopen votes queue.

3. Leave closed reasons

   • If a Reopen votes reviewer elects to leave a question closed, the reviewer will be presented with a “Leave closed reason” modal (similar to the close reason modal) that will return feedback to the editors. For example, if a question was originally closed because it lacked details, it may still require further details even after an edit, or new issues with the question might present themselves like asking too many questions at once.

   

   • In addition to inbox notifications, the closed question’s post notice will also be updated with more information.

   

4. Multiple opportunities to send question for review

   • We’ll be launching this feature without any limits – users will be able to submit multiple edits on individual posts. We’ll take a look at the data in a few weeks to see if any abuse of the feature has occurred and take mitigation steps as needed.

   • [Stack Overflow only] Reviewers will not see posts that

     1. they’ve voted to reopen since last closed, and
     2. voted to close since the post was last edited.

     Moderators are exempt from this restriction.

5. Bug fix: “Don't cause the reopen vote of the user, whose edit pushed the closed question into the reopen queue, to invalidate the review”

   This bug fix is also included as part of this release. You can read Kyle’s detailed response here.

Other changes to the Review queues

1. Turn off the Help & improvement queue

   • H&I has been our least productive queue. Over the last 90 days, only 9 tasks went into the queue each day on average, compared with 2,215 for First Posts (our busiest queue) and 188 for Reopen Votes (our second-to-last busiest). Consequently, this queue will be turned off. It will remain in the listing until its current items are exhausted, and will then be removed. History for and badges awarded from this queue will remain as they are.
   • With the addition of the First questions queue, we expect that questions with potential can get the same assistance they need more quickly.
   • Questions that would have ended up in this queue from Triage (with a Needs community edit action) will now enter the First questions queue.

2. New actions for Late answers queue

   The Late answers queue has been successful in identifying and removing low quality posts, repeat answers, and spam. Reviewers were able to take any number of actions on a task, so the objective of this queue wasn’t all that obvious. To clear up any ambiguity and focus on identifying problematic answers, we’ve defined three actions: Looks OK, Edit, and Delete.

3. Keep Triage queue (Stack Overflow only)

   • Triage will continue to handle questions only.
   • Triage will be available on all sites, but only turned on by default on Stack Overflow. Have a meta discussion and ask a moderator to status-review if your site wishes to add the Triage queue.

4. Name change: Low quality posts → Low quality answers (Stack Overflow only)

   This queue will continue to handle answers only.

5. Misc. We won't issue new Reviewer or Steward badges for the queues that were turned off, but this won’t have any effect on badges already earned in these respective queues.

   Along with all these changes, here is an updated visual of the Review queues map –

   

What about issue X, Y, Z?

As mentioned earlier, this is our final major release. We will not be pursuing the proposed suggested tasks feature at this time. We didn't feel that it was going to significantly increase curation activities on the site and therefore didn't warrant the high development cost.

There are still a few outstanding issues related to suggested edits that have proved to be technically challenging. We’ll respond in those original Meta request posts as soon as we decide on a course of action.

Feedback

Please leave your feedback and any bugs you may discover related to this release below this post. We will be monitoring this post until Friday, September 10th. Report any further issues after September 10th as new questions on Meta.

Answer 1

After editing, an answer went through the first answers queue multiple times.

This doesn't make sense. There is only one first answer here, and it could lead to multiple comments from Community, which would be confusing for anyone who doesn't have knowledge of the review process.

Is this intended behavior?

Answer 2

bug

User not returned to review queue after edit

---

Summary

A change in behaviour (of the Edit link beneath the post).

Note: This report is not the same bug/feature request as Tomerikoo's which requests the removal of the edit link.

---

Details

Previously, in a review queue (any review queue, IIRC, but I am currently seeing this in Late Answers), if you edited a post (by clicking on the edit link beneath the post), then once the edit was submitted, you would be automatically returned to the review queue to continue/complete the review of that particular item.

However, now, once the edit has been submitted, you are put on to the question's page instead. You have to hit the browser's back button twice, in order to get back to the post that you were reviewing - in order to complete the review (No action needed, I'm done, recommend deletion, etc.).

Also, if you cancel the edit, then you are not returned to the review page either, but again you are placed on to the question page.

Why has this changed? Please can we be returned automatically to the review queue, rather than having to click an extra few times on the browser's back button to get back to the review.

Answer 3

bugstatus-completed

When logged out, review links for First Questions, First Answers, and Late Answers appear completely greyed out. This does not happen for the other queues (for me).

Example from First Questions on Academia SE:

On the other hand, an example from the Suggested Edits queue, that appears normal:

Answer 4

support

When I start to edit this old question of mine, which until yesterday had only 29 views, I see the new checkbox

This edit resolves the original close reason and the question should be considered for reopening.

It seems that box is always shown (I don't think that question qualifies for the popularity criterion). How does that work with the edit-to-reopen criteria mentioned here? Are those obsolete now, and is the only criteria whether this box was ticked or not?

Bonus question: what happens if I my scripts edit a question via the API? Do those ever count for reopening? If so, my scripts might need to be more careful with editing questions.

Answer to that bonus question: those edits don't seem to count for reopening (which is good IMHO). Here is an example: no Reopen Votes review queue item to be found.

Answer 5

feature-request

This answer describes several issues regarding having the canned feedback appear as a comment from the Community account. Also, this answer asks to allow having those comments be flagged, this one requests being able to undo accidental wrong feedback, and this answer asks for an option to be able to (easily) delete anonymous comments.

In addition, Zanna's comment states:

Please let me (as a mod) delete these comments!

and then further explains in this comment:

Nope! I just get a notification saying the comment will be deleted automatically when the post gets fixed, ...

I don't recall where I read this, but it seems there's doubt about when & how it's determined that the "post gets fixed" so these comments are then deleted. Nonetheless, I believe that at least diamond moderators should be trusted to delete these comments.

The Community bot already does a lot of other things which, in my opinion at least, are not particularly closely related to this new use. I believe creating a new special bot, called something like "Review", to use instead would solve, or at least help mitigate, many of these issues, as well as allow more flexibility in any future added features. These benefits include:

• The new name would make it clearer to anybody reading the comments where they actually come from. I, even as a relatively knowledgeable user, found it confusing to see comments from the "community" when they are actually from specific users while in the review queue instead. This alternate name should also make it less likely that users think it's from a "moderator", as mentioned in this answer.
• This should help reduce the chance that people think it's some automatically generated comment, e.g., from a computer algorithm (as explained here), instead of being from a site member during review.
• The display and behavior can be tailored for the "Review" comments so, for example, they can be modified and deleted by diamond moderators, flagged (as requested here), more easily deleted by regular members (as asked for here), etc.
• With any other review related features added later that involve using a bot, then having a specific "Review" one available may more easily allow additional customizations and flexibility compared to using the "Community" bot instead.

Answer 6

status-review

There should be a custom comment option in the New answers and New questions queues because most of the time the default (i.e. the new "community bot" comments) comments do not pertain to the new answers or questions. For example, we also get new questions that should be on the Meta of the site but there is no option to let the OP know.

In answers, we also get answers that should be comments but there isn't any option.

Answer 7

The text in the help center article for the low-quality posts queue doesn't match the capitalization according to the style guide.

Headings

Don’t	Do
Performance Overview	Performance overview
Edit Your Profile	Edit your profile

Feature or product names

Don’t	Do
Job Alerts	job alerts
Blog	blog
Profile	profile
Stack Overflow talent	Stack Overflow Talent
stack exchange	Stack Exchange

The heading of the article is mixed case, not sentence cased:

How do I use the Low-quality posts queue?

This is also inconsistent with the header for the access review queues page, which is all lower case even though it appears to me to be a header.

References to the feature (low-quality posts queue) are sentence cased, not lower case:

Posts entering the Low-quality posts queue have been flagged as problematic by users or recognized by the system’s quality checks to be potentially low quality. (Apparently there must be a blank line separating text from tables even if you put a double blank at the end.)

Frankly, it doesn’t appear as if there is a consistent understanding of what is a header and what is a feature name. If you’re going to capitalize something in the middle of a sentence, it shouldn’t be sentence cased unless it is quoted or otherwise distinguished from the rest of the sentence. To do otherwise just makes it unnecessarily difficult to parse the sentence. There are plenty of questions on ELL where a non-native speaker had trouble understanding a sentence because it wasn’t clear which words were modifying other words.

If this wacky "let's treat something like a proper noun, but sentence case it instead of properly capitalize it" is intentional, which it seems to be based on the review queue help center articles, here's a list of a few pages that don't follow that convention either by (correctly IMO) leaving the feature names as all lower case or capitalizing feature names as if they were proper nouns:

1. What are review queues?
2. Why was I suspended from the Review Queues?
3. approve tag wiki edits "other posts in the Suggested Edits review task" (links referring to the new queues are sentenced cased as if they are considered headers, so this is inconsistent.)
4. What does it mean if a question is "closed"? "...is automatically added to a reopen queue for community review." (An aside... is it "the Reopen votes queue" or "a reopen queue"?)
5. Privileges

2,000	access review queues	Access first posts and late answers review queues

There are more, but I think that's enough to make my point that the way the style guide is being applied in the updated review queue articles is not intuitive or consistent with other pages. On top of that, it serves no purpose in terms of communicating information. Randomly capitalizing Words in the middle of a Sentence when they aren't proper nouns is annoying.

And I do understand that it's annoying to have your orthography nitpicked. When you look at that sentence and think "What does it matter? I know exactly what that text means." consider that the reader probably doesn't know what your style guide says; they just know what they typically see. Using unconventional styling for documentation makes it harder to understand.

Answer 8

feature-request

There's a need for the canned responses to deal with one of the most common problems: link-only answers. I noticed on this deleted SO NAA that the canned comment was

Please provide additional details in your answer. As it's currently written, it's hard to understand your solution.

That's fine and good, but it's too vague for the situation (and I could easily see users responding with "Well, all the details are in the link!"). I personally use this canned comment (offered for an example of what I'd like to see)

A link to a solution is welcome, but please ensure your answer is useful without it: add context around the link so your fellow users will have some idea what it is and why it’s there, then quote the most relevant part of the page you're linking to in case the target page is unavailable. Answers that are little more than a link may be deleted.

Answer 9

bugstatus-completed

The "Learn more" button from the Stats and History tabs of any review queue does not work

Oddly enough, it does seem to work when under any "Review tasks" tab, but under the other two, it does not.

Answer 10

bugstatus-review

First Answers Canned Responses Give Help Link to "How to Ask"

The help links in the First Answers queues incorrectly lead to the how to ask help page.

The links should lead to how to answer: https://stackoverflow.com/help/how-to-answer

Below is an example of a posted canned response with the incorrect link after selecting Answer needs supporting information.

Answer 11

feature-request status-completed

The text in the closure banner explaining that there's a reopen review available for a question is unclear:

An edit was submitted for review 59 mins ago.

An edit wasn't submitted for review, an edit was made and the post was submitted for review. Can this be change to reflect the correct meaning?

Answer 12

feature-request status-completed

Add OP shading to the revisions summary in the Reopen queue

In the Reopen queue, when summarizing the changes to the question, please highlight the owner of the question, using the standard highlighting already used for OP throughout each thread.

This is good for consistency in terms of how information is presented elsewhere on the site, but here it is also a useful handle to be able to parse quickly what the edit history looks like. Was the question extensively edited by OP or by somebody else? This is an important piece of information and the design language already has the tools to communicate the answer effortlessly.

Answer 13

bug status-norepro

Inconsistent font size in the reopen review box.

Maybe it’s my phone’s fault (iOS Safari), but I reviewed a post for reopening and this is what I saw:

These should use the same font size.

Answer 14

status-declined

Thank you for all the improvements.

Can you please remove the option to "leave feedback" by community user from "First questions review"? While this option makes sense in "Low quality answers" and in "New Answers review", having this option in "First questions review" is unnecessary. If I say that the question is unclear I expect that I cast a close vote on it and push it into the close review. I don't want a noisy comment from community user that states the same thing as the blue banner does. I can't even self-delete this comment. I would have to flag it and ask moderators to delete it.

Please replace that option with "close" option instead.

Answer 15

bug status-completed

The first questions and first answers review queues don't show up in red, even when there are tasks to review:

When I looked at it from the inspector, I noticed that the first questions queue has the HTML:

<li class="-item danger-active">
    <a href="/review/first-questions" class="js-gps-track" data-gps-track="review_popup_item.click({ item_type: 12, in_danger: false })">
        <span class="-title">First questions</span>
            <span class="v-visible-sr">(Active)</span>
    </a>
</li>

As opposed to late answers:

<li class="-item danger-urgent">
    <a href="/review/late-answers" class="js-gps-track" data-gps-track="review_popup_item.click({ item_type: 5, in_danger: true })">
            <span class="-title">Late answers</span>
            <span class="v-visible-sr">(Urgent)</span>
    </a>
</li>

The

<li class="-item danger-active">

and

<li class="-item danger-urgent">

seems to be the culprit here. danger-active doesn't have the proper styling, whereas danger-urgent does. When I run:

$(".danger-active").removeClass("danger-active").addClass("danger-urgent");

The issue is fixed.

So it seems Stack Exchange is implementing a new class name, but forgot to add it to the others and add proper CSS styling.

Answer 16

feature-request status-review

As I said in the product discovery, the wording for this button is problematic. Remember that this appears as part of an edit, so most users will simply think they are submitting their edit for review, as they are used to.

It should indicate a lot clearer that you are editing the question, and expecting it to be reviewed for reopening. As I previously suggested some reasonable alternatives are:

• Submit edit for question's reopen review
• Substantial Edit

Answer 17

support status-completed

The "First Posts" review queue is now retired, but what are the plans to make it disappear from:

• the Review dialog in the top bar
• the main review page

On most sites, it has been cleared and will not get any new entries.

Answer 18

feature-request

Please add a close link to the question in the first answer queue.

Every now and then you come across a low quality answer in the First Answer queue, well just because the question was low quality to begin with. It would be great to have a direct option to cast a close vote on the question then and there, instead of having to navigate away to the question itself to be able to do that.

I already looked at the question to see if the answer is doing a job at answering the question at hand. If I then identify the question to be problematic it should be easy to handle that as well.

Answer 19

Pointless request for more details (from Community) on "Me Too" NAA

FWIW, I came across this comment when it was in the Late Answers review queue on Ask Ubuntu.

As the screen shot shows, a totally inappropriate comment is left by community on a "Me Too" answer. There clearly aren't any more details available for a solution as the poster has the same issue as the OP and doesn't know how to solve it.

In addition, when I try to flag it as no longer required, in an effort to remove it, you get a message saying that it can not be removed and will be removed only when the issues have been addressed:

Note: I wanted to flag it for removal because the intention of this "Need more info" comment is to prompt the new user to expand upon their pointless answer, thereby wasting their time working on an answer that will clearly be deleted in the near future.

---

This comment is obviously auto-generated somehow, but not by the Recommended deletion option in the Late Answers review - possibly/probably by some mechanism from another review queue (I don't know if it was generated in one of those two new First Questions/Answers review queues).

Answer 20

bug

When voting to close a question in the First Questions queue, actually having to flag because the close option isn't available(!), the review item hard refreshes. My vote to close, via flagging, was a binding vote as a mod so not sure if this happens when the close vote doesn't cause the question to be closed. This seems to be happening because of the review item hard refreshing not soft.

The resulting effect here is that the reviewer actually loses the review that they've just performed and can't complete it as the item that comes back is the invalidated review, due to the closure.

---

And side not, perhaps more importantly. Voting to close should really be an available option in First Questions. To be honest the idea behind these queues is good and this is a good start but we've lost way too much functionality from just plain old First Posts that things have gone back too many steps.

Answer 21

bug

With the recent update, the other actions item will stay selected if you post a comment on a post, and then delete it, allowing you to complete the item without technically doing anything:

Probably very unlikely most people would do this, but it could be abused.

Answer 22

Similar reactions from the Low Quality Posts/Answers and Late Answers queues are not merged together (i.e. upvote existing comment instead of posting a new one) which leads to an unnecessary redundancy:

The first comment originated from the Low Quality Answers queue and the second, with the same type of deletion recommendation, from the Late Answers.

Answer 23

(Possible) bug

The "Late answers" and new "Low quality answers" queues have very similar (in fact, almost identical) UIs. In particular, the canned comments available when selecting "Delete" or "Recommend deletion" are the same.

If a second reviewer in one of those queues selects a comment that is already present, then a new comment isn't added; rather, the 'original' comment is automatically upvoted.

However, when the same comment is added from the 'other' queue, it is added a second time, as here (10k on SO required to see it, as it's already deleted). Should this also just upvote the existing comment?

---

Related: Do "Recommend deletion" votes from those two queues act cumulatively? That is, if (say) four such votes are required to kill the post, will one from "Late answers" and three from LQA finish the job?

---

Screenshot for those lacking 10k privileges:

Answer 24

status-completed

Getting extra review tasks:

I didn't see this mentioned in another answer or the original post, but please ping me if it is there and I'll ditch this. I've poked around the stats and I've noticed I'm not the only one. I figured it might be a mod thing or high-rep thing, but at least a couple of the other reviewers have lower rep than me, and none of them are mods (at least not on SE).

Update: Along the same lines, a banner telling me I have reached my limit, while the progress bar tells me I still have more to do if I wish.

Answer 25

bugstatus-completed

Similar to this and this (both already marked completed), another blank/missing state: when there are no more posts in the queue (appears to affect all queues).

This also occurs when you have completed your reviews for the day, suggesting that the fix for this has since broken.

Answer 26

bugstatus-completed

The How do I use the First answers queue? help page refers to "this question":

One review from a user is enough to dismiss the task so review carefully to ensure that this question meets site expectations.

Answer 27

bug / feature-request

The canned comments do not include the "From Review" link, similar to the ones from the Low Quality Posts queue.

Example from Late Answers: https://stackoverflow.com/review/late-answers/29721791 I chose the option for "I'm having this problem, too" and this is the comment that was automatically posted, without the "From Review" link.

A matching review from the Low Quality Posts - https://stackoverflow.com/review/low-quality-posts/29725913 which includes the "From Review" link.

I'm not sure about the new First Q/A queues with the community feedback. Providing such link might be an anti-pattern for the anonymity purpose, but it should be there for the planned regular feedback.

Answer 28

bug

The "Asked {x}" text in a review displays the question's age in a way that's different to how it's displayed when the question is viewed.

---

For instance:

Review	Reported in review	Reported in post
Link
Link
Link
Link
Link

The precision for questions posted in the last 24 hours (just reported as "today" in review) is especially annoying because when doing some review tasks, the relative timing of a question and its answers can be important.

Answer 29

status-completed

Thanks for this modification.

Can you add multiple new options in "Share feedback" options such as "This is not an answer" (for people who are asking for help in an answer).

Also, when we add specific comment (and not premade one), we can't validate the review such as before.

Edit: Include in this post and will be fixed (planned)

Answer 30

bug

A review task in the first answers queue had multiple users review it: