83

This is one of three project announcements for Improving Review Queues. We’ve summarized the project objectives and goals here.

As a reminder, this project is still in the early stages of discovery. In this post, we are sharing proposed changes to the Onboarding and Updated Workflows/Paths. We are asking for your feedback before we begin implementation. After we collect community feedback, we will be open to including changes into the next design iteration.

Onboarding

Close votes with new guidance

Information and guidance about how to use the Review Queues is available, but it’s in several places. We are proposing different ways to include upfront instruction and guidance to better equip reviewers to use their newly earned privilege. We are continuing to make refinements to the language and copy in these messages as part of the discovery process.

Users will find an onboarding message on the homepage with general information about using the Review Queues. As they enter each new queue, they will be presented with a modal with unique instructions on how to best contribute to that queue.

Informational modal for Close Votes

Open questions:

  • What did you wish was part of your Review Queue onboarding?
  • What feature/aspect was difficult to discover as a new reviewer?
  • What do you consider essential “day one” information for new reviewers?

Updated workflows and paths

Close votes queue

We would like to update each queue with a new Actions menu that exposes all instructions and adds a bit of pause so that users can make more considered decisions. We will also be adding the new follow feature (available in the "…" button) so that reviewers can keep track of posts.

Reviewers will no longer have to open questions in a new browser tab to get more context. In addition to the post being reviewed, each review task will have the question, comments, and other answers on the review page to give context. We are also sharing a subtly different design in our user research sessions.

All changes will be supported by a Stacks update to the visual design along with responsive, mobile views.

Stack Overflow-specific changes:
We want to take a closer look at the success and failures of the First Post, Late Answers, Triage, and Help & Improvement queues.

In Triage, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. We want to remove the Requires editing action and replace it with two new actions. Selecting Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity) will be sent to the Close vote queue.

Triage queue

If a reviewer selects Very low quality in the Help & Improvement queue, the post will be sent to the Low Quality Post queue. By doing this, we hope to cease the endless cycle making these queues unsuccessful in helping moderation.

Finally, we are considering closing the First Posts and Late Answers queues and combining those tasks into Triage. These queues were intended to give good posts a head start toward success and filter out bad posts, but tasks could see better, faster intervention.

Open questions:

  • Does it feel like any important actions or functions are missing from the review page?
  • Does having the original post, comments, and additional answers all on the page make it easier to do a review?

    Stack Overflow only:

    • In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?
    • In Help & Improvement, do you anticipate any issues with sending Very low quality posts to the Low Quality Posts queue?
    • Do you anticipate any issues with removing the First Posts and Late Answers queues?

More updates

We share improvements to the review bans and new features in this final post: Improving Review Queues - Design overview II.

  • 1
    The instructions for the Close Votes queue now contain the sentence "Review edits or add your own to reopen them on the site." - I hope that's some kind of placeholder? Also, the "Don't show this me again" button needs a grammar fix (me this or this to me). – Glorfindel Apr 23 at 19:17
  • 17
    Despite me posting four answers already, I really like the direction in which this is going. Mobile reviewing - I can't wait. Keep up the good work! – Glorfindel Apr 23 at 19:17
  • 7
    Please consider moving the part about Triage / Stack Overflow to Meta Stack Overflow - we usually close/migrate questions about those topics when they are posted here. – Glorfindel Apr 23 at 20:17
  • 2
    @Glorfindel Thanks for pointing out the typo. I've updated the image. The copy in these screens are all open to discussion and change. – Lisa Park Apr 23 at 20:46
  • 1
    I really like your proposed improvements. It's pretty clear that you nicely considered what I said back during our previous chat together, and I like that! – Sonic the K-Day Hedgehog Apr 23 at 22:14
  • 17
    "Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity)" You need to indicate who must/can edit in the button. "Add clarity" is just as unclear as "requires editing". – philipxy Apr 24 at 6:06
  • 1
    when reworking Late Answers, please make sure that you keep the criteria justified by Jon Ericson here. I made over 2 thousands LA reviews at SO and other sites and as far as I can tell this queue plays fairly important role in maintaing quality in most visible (and hence, most important) questions – gnat Apr 24 at 7:12
  • 1
    I'm with @SonictheStay-HomeHedgehog on this one - thank you for a really, really thoughtful and great write-up! – Sébastien Renauld Apr 27 at 8:40
  • Turns out I had no idea this post existed yesterday when we talked... I need to scan MSE more :-( Sorry if I repeated stuff you already covered here! Also as a small note: the 'welcome' modal for the CV Queue mentions 50 votes a day, but users may only spend up to 40 of their 50 total votes in the CV Queue. – TylerH Apr 28 at 18:01
  • @TylerH No problem at all! The whole point of our user interviews is to go through these mock-ups in full and get your perspective. Really appreciate your time and thanks for the note. – Lisa Park Apr 28 at 19:15
  • 2
    Will this fix the concerns mentioned on MSO, or is that (much) later on the agenda? Triage needs to be fixed urgently, and users need to be notified upon receiving a review ban! – Mast Apr 30 at 17:37
  • A triage update is much welcoming! No more only 10 "Unsalvagable" per day - would be awesome, as you already decide on what to do with the post based on multiple reviewers agreeing on a taken action! Because at one point, there is nothing to do, and I managed to get a queue where I was out of Reports in the first 10 questions :| – Tomasz Juszczak Sep 12 at 16:07
  • I recently wrote this discussion/feature request to encourage people to do more reviews meta.stackexchange.com/questions/354255/… – Luuklag Sep 16 at 11:26

27 Answers 27

58

The onboarding guidance looks great - +1 for "Skip"; that cannot be repeated enough!

My recommendations for improvement would be simply to link to more extensive guidance for those who might want it. There are some great guides out there already; here are just a few:

In Triage, the Requires Editing action is sending a surplus of poor, unsalvageable posts to bloat the Help and Improvement queue. We want to remove the Requires editing action and replace it with two new actions. Selecting Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity) will be sent to the Close vote queue.

I would strongly recommend killing the "help and improvement" queue: it was never finished, and the crucial pieces of functionality which it lacks are now being planned as part of the close workflow redesign. It'd be one thing if it actually provided some value for the reasonable questions that make it in... But I couldn't find any evidence of this during the year after we first rolled it out, nor when I looked again a few years later - so far as I can tell, questions which are actually salvageable are overwhelmingly edited by people browsing their favorite tags outside of review - a queue is simply not an effective entrypoint for this sort of work.

Finally, we are considering closing the First Posts and Late Answers queues and combining those tasks into Triage.

Triage itself is a great UX for pre-vetting questions... But that purpose was never fully realized either! Instead, it evolved to become a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. - for questions. Eliminating other queues for pre-vetting questions in favor of Triage is an excellent plan, and I highly recommend making this change across all sites, leaving the Low Quality review to what it does best: allowing reviewers to quickly process problematic answers (this would also help to eliminate the broad class of bugs introduced by having significantly different workflows on SO vs. every other site).

Oh, and talk to the SOBotics folks. They've already been building far, far better replacements for the First Posts / Late Answers scenarios for years, and can help you to improve the heuristics for sending these posts into Triage & LQ Review (spoiler: the heuristics you're using now were good... 5-10 years ago. They're woefully archaic now!)

Finally, remember that these queues currently process an insane volume of posts... Any changes will have dramatic and difficult-to-predict effects far outside of review. Be prepared to monitor both review actions themselves, and their secondary / tertiary effects, and to make quick changes as required, for roughly 12-18 months after you begin rollout; this will need to be someone's primary responsibility for at least several months! Speaking from experience, when things go wrong they go very, very wrong very quickly - good luck!

| improve this answer | |
  • In the early weeks of Triage, "Requires Editing" used to be called "Should be Improved". Why was that? – Sonic the K-Day Hedgehog Apr 23 at 21:39
  • 10
    Naming is hard. We were aiming for a sweet spot encompassing questions that were answerable, but unlikely to fair well. – Shog9 Apr 23 at 21:44
31

It isn't necessarily related to this project, but I still feel like posting it here since it looks like the guidance for the options (currently visible when clicking the '(more)' link) will be more prominently visible as a sidebar widget.

I just noticed that for questions in the Low Quality Posts review queue the guidance says

Close if the question cannot be fixed and should be removed immediately

This is incomplete or even incorrect; we ultimately close questions to indicate that they need to be improved in order to be answered (and only if they're not, they will be removed). Reviewing a question in this queue doesn't have anything to do with deleting it, so please consider rewording it.

| improve this answer | |
26

Reviewers will no longer have to open questions in a new browser tab to get more context. In addition to the post being reviewed, each review task will have the question, comments, and other answers on the review page to give context.

Taking this improvement a step further, is it possible to highlight the posts needing review to users with review privileges, in the course of normal browsing? Occasionally I find a post in the review queue after I've already seen it. If I'd known it needed review while reading, I could have already done it.

| improve this answer | |
  • 1
    That should definitely be a profile option then and disabled by default. It's a good idea, I would enable it myself, but most people wouldn't. – Fabian Röling Apr 25 at 17:09
  • @FabianRöling: Alternatively, one might imagine enabling it for “active reviewers” (however you want to define that). There’s obviously a big difference between people who are eligible reviewers, have previously reviewed, and currently participate in reviews on a regular basis. I imagine the latter would be interested in this feature, while the others wouldn’t be. – Jeremy Caney Jun 10 at 21:45
22

Close this post if it should be closed for any reason (consider leaving an explanatory comment)

I suppose that upon voting to close the question, the review queue will automatically load the next question, as it does now? In that case, I'd opt to add the word 'first'

(consider leaving an explanatory comment first)

since otherwise you have to navigate back to leave that comment.

I just noticed the current guidance does a better job in this regard, it says:

Close if this question should be closed for any reason; consider leaving an explanatory comment before voting

| improve this answer | |
  • 5
    Good catch. I can definitely see some people who plan to comment after voting but are unable to do so. – V2Blast Apr 23 at 23:44
  • 1
    I do that most times (outside of reviews, just on new questions in followed tags), so it would be nice if closing did not automatically forward me. In fact, thanks to the stupid new "Does this answer your question?" auto-comment, I have to edit my comment after voting to close. – Fabian Röling Apr 25 at 17:07
21

What feature/aspect was difficult to discover as a new reviewer?

The fact that if you use the back button after reviewing an item, you see how other users reviewed. As a new reviewer, this gives you some kind of peer review. (Of course, those reviewers might be worse than you are.) That could use more discoverability.

Or, taking it a step further: have an overview where your last X reviews disagreed with the final outcome. I know @rene has a script for that, but it would be nice as a built-in feature.

| improve this answer | |
  • 1
    I've suggested something to this effect years back with Shog and Tim: kind of a dashboard of aggregated data. You can see all your reviews on the all-actions tab of your user activity tab (yeah, not very discoverable!) – Braiam May 3 at 3:16
  • I mentioned this on the other design thread (in the comments), but as someone who makes a habit of auditing my reviews, I love the idea of making this easier and more obvious. That’s been one of the most useful exercises for calibrating my standards—or, at least, pushing me to validate my assumptions. Sometimes I come to the conclusion that other reviewers are misunderstanding or violating the criteria (e.g., mistaking “Have you tried…?” as new questions), but it’s still useful for identifying potential misunderstandings in my own rubric. – Jeremy Caney Jun 10 at 22:00
20

Questions and answers should be reviewed separately, not together

Reviewing questions and answers, in my opinion, requires different mindsets as well as different workflows. I've been tripped up quite a bit when I've been prompted to review a question right after reviewing an answer, but not realize that it's a question, and it's only when I try to review or flag it as a non-answer that I realize otherwise. The queues are nice about letting me know whether I'm looking at a question or an answer when I look in the appropriate spots, but looking at that as well as changing mindsets if changing between post types takes me time.

I disagree with your proposed change to remove First Posts and Late Answers and merge them with Triage. I'd be up for removing questions from First Posts and moving those to Triage, but please keep those separate queues for reviewing answers. Also, see my other suggestion of not moving questions reviewed as "Very low quality" in Help & Improvement to the otherwise answer-only Low Quality Posts.

| improve this answer | |
  • What would be detrimental about merging First Posts and Late Answers with Triage? They are all "front line" intake activities, and Triage is certainly the most concise name. – Z4-tier Apr 24 at 19:08
  • 1
    @Z4-tier It's the fact that you're reviewing both questions and answers at the same time. Reviewing one requires a different mindset and workflow from the other, and having to switch between them repeatedly is a pain. Also, you can get confused and review the wrong way if you don't check whether you're looking at a question or an answer. As I said, I'd like questions from First Posts to be moved to Triage, while First Posts and Late Answers are kept solely for answers, so that they're reviewed separately. – Sonic the K-Day Hedgehog Apr 24 at 19:14
  • I wholeheartedly agree. This is precisely why I always filter by either Answers or Questions when reviewing First Posts. I otherwise find it really difficult to context shift between them. The conditional rubric just becomes too complicated for my brain to juggle. – Jeremy Caney Jun 10 at 8:58
  • 1
    Though, in my case, I’m also just far more comfortable reviewing answers at this point. After a few review suspensions in Triage back in March, I remain really nervous about reviewing questions—even though I’ve since reviewed thousands of answers between Low Quality Posts, First Posts, and Late Answers. The criteria for questions continues to feel a bit capricious to me, with too much room for suspensions over what feels like reasonable disagreement over judgment calls. By contrast, the criteria for answers feels far more deterministic and well-defined. – Jeremy Caney Jun 10 at 9:11
17

When you guys incorporated the initial onboarding message for the Close/Reopen votes privilege, I was hopeful that you would apply the same logic to the review queues, but scaled up massively.

I have to say, I'm not disappointed. I like what I see.

What did you wish was part of your Review Queue onboarding?

I feel as though it's important to stress to new reviewers that reviewing correctly is paramount during their reviewing efforts. When you review incorrectly, and perhaps let a close-worthy question through the Triage queue, you increase the time that question remains open, which we know isn't preferable.

Additionally, I feel that it's important to stress how useful the Skip button is. When the reviewer is not concrete that an actionable button is the right choice (perhaps they've been reviewing a post for a long time?) it would be nice if it were made known that the Skip button is their best friend in their time of need.

What feature/aspect was difficult to discover as a new reviewer?

I feel as though the act of reviewing isn't too difficult to pick up on, provided the information given to the user is simultaneously succinct and complete. It's been mentioned a million times already, but the guidance for "Requires Editing" in the Triage queue is very difficult to understand for new and old reviewers.

On one hand, I would prefer not to dig through a lot of old meta posts to be able to review correctly. On the other, I really do want to review correctly, and I have spent a large amount of my time ensuring that my review choices were correct. My suggestion would be for you to word the onboarding explanations in a way that covers everything a reviewer might come across. Where that explanation fails should be instructions to Skip!

Lastly:

Does having the original post, comments, and additional answers all on the page make it easier to do a review?

Oh my goodness, YES. It's an excellent improvement, and reminds me of this under-review feature request. Having easy access to the comments and current answers (or the parent question of a review on an answer) gives a large amount of context to a reviewer, and should hopefully ensure they don't need to open the question into a new tab to browse the full contents of it.

Overall, the currently proposed changes are looking great, and I'm interested to see what other improvements are made along the way as other community members weigh in.

| improve this answer | |
  • 5
    +1 agree that stressing SKIP can't be done enough. In the welcome-popup I'd even suggest linking to one of the canonical "here is why skipping is so good" posts--exposing new CVers to the reasoning behind the system is huge, IMO. – nitsua60 Apr 23 at 19:21
17

In the mockups, I note that the "Skip" link is small and subordinate to the "Submit" button, very easily missed. I'd argue that it should be a first-class button, given equal (or greater) emphasis in the UI.

Experienced users will learn an interface and promptly ignore it, so visual emphasis in a UI is primarily geared towards new users. For a new user the default action should absolutely be "Skip" (and, as other answers/comments have mentioned, we should explicitly call that out during onboarding), so the "Skip" button should be big and obvious.

Along those lines, I'd also suggest changing the order in which the other options are presented. In the mockup, the two least-used options ("Approve" and "Improve readability") are presented first. The vast majority of questions in triage need to be closed, so those two options should be near the top to help new users to understand that.

Overall, I think this is a really good start, and I'm really happy to see curators getting some love. I think with a few UI tweaks to subconsciously draw users towards the right actions it could be great.

| improve this answer | |
  • 2
  • @gnat: How would you see that interacting with the proposal to award Steward badges for every 1,000 reviews (and not just the first 1,000)? Would the Skip threshold apply to all reviews, or just those performed since the previous Steward badge was rewarded? (This isn’t an issue either way; I’m just thinking through the implementation details.) – Jeremy Caney Jun 10 at 22:22
  • 1
    @JeremyCaney I don't have firm preference here because my personal experience sends rather mixed signals. On one hand after first few hundreds reviews I knew importance of skip by heart meaning there's no need to teach me yet again. OTOH I keep skipping a lot and would appreciate if system kept regularly reminding me about its importance... – gnat Jun 10 at 22:48
15

As a general improvement to the workflows, add the ability to flag something as spam from any queue. You're likely to encounter spam in the "Close Votes", "Low Quality Posts", "Help and Improvement", "First Posts, "Late Answers", or "Triage" queues, but only the last three let you flag it as spam. From the "Close Votes" queue, you can only vote to close it, from "Low Quality", you can only suggest deleting it, and from "Help and Improvement", you can...well, you can't really do anything.

| improve this answer | |
  • 4
    I especially agree when it comes to Low Quality Posts. When it comes to spam answers, choosing Recommend Deletion may push it towards review deletion, which allows the author to undelete the post and restore the blatant spam without a trace. This is something I raised in my prior chat with Lisa (the question author), and I think this might come up in a later post. – Sonic the K-Day Hedgehog Apr 24 at 2:38
13

Potential unclear wording in new Triage options

I do like your proposed changes to the review actions in Triage. In fact, I was going to propose the same thing as you, to split it apart into four actions, with separate buttons for where the post simply needs details from its author and for when it needs to be removed as spam or rude or abusive. As far as I understand, these are the buttons you're proposing:

  1. Looks OK - for questions that are good, on-topic, and don't need proofing
  2. Improve readability - for questions that need some proofreading, but are otherwise good, on-topic (send to Help and Improvement)
  3. Add clarity - for questions that need more details from their authors, or should be closed (vote or flag for closure)
  4. Unsalvageable - for questions that are spam, rude or abusive, or need action by a moderator

Again, I like the categories, and those categories are exactly what I was thinking of before.

However, there is a small issue with the wording you're proposing: it's not entirely clear what action to take when reviewing a question that is fundamentally unsalvageable and needs closing (e.g. blatantly off-topic, a clear duplicate, etc.) but aren't spam or abusive. Your proposed wording for the "Add clarity" choice would imply that it's only for questions that could be improved by their authors, not for these types of questions that need closing but not red flag deletion.

I'd recommend changing the name of that choice to "Needs improvement", the same words used in the flag dialog, so it's clear that the closing option should be used for such questions.

| improve this answer | |
  • I'd like to plug the suggestion I made in the Meta SO post here: put the two "Requires editing" options into a dialog similar to the flagging options, with more detailed explanations of each. – HAEM Apr 29 at 7:15
13

Does it feel like any important actions or functions are missing from the review page?

For me yes. On Stack Overflow specifically, the H&I queue feels like it needs a "Close" button. Now, obviously you're trying to improve the workflow, and ideally posts that should be closed should never make it here in the first place, but unfortunately too many currently do. It'd be really nice if as a 3K user I could close it directly, or if there was some other action I could take to send it to the cv-queue.

Does having the original post, comments, and additional answers all on the page make it easier to do a review?

Yes, absolutely. I often find myself looking for this information anyway.

In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?

I have nothing particularly problematic to bring up, only I'm not sure this wording give quite as strong of a message as would be ideal. Perhaps there was some way to make clear that the OP needs to add clarity, and that can only done by the OP. On the other hand, improving readability can be done by anyone. It might be better in the long run if you can more strongly differentiate the two (i.e., what must be done by the OP vs other people).

But really, I think I'm just being nit-picky here.

In Help & Improvement, do you anticipate any issues with sending Very low quality posts to the Low Quality Posts queue?

There is a chance that people will mis-understand what is meant by "low-quality" here and might mistakenly push things that need to be closed into the low-quality queue. Not sure how major of an issue it will become, but something you might want to consider.

Do you anticipate any issues with removing the First Posts and Late Answers queues?

Not off the top of my head. If their intended functions can be better-served by other queues, then why not do that instead? (Obviously, you're already thinking of this; that was just rhetorical). No use having too much redundancy.


Oh, and BTW, I want to keep encouraging good things I see in you (the company) after the way 2019 ended. I want you to keep encouraging behavior I like, so I say this almost every time I give my feedback on meta. I say that because you might be tired of seeing me say stuff like this, but thanks again for asking. I see a lot of positive improvements for the queues here, and really appreciate that you have taken the time to ask us about it and listen to our feedback. So thanks again!

| improve this answer | |
9

The problem with the current queues is that they are about how the post got there, not about what needs to be done with it or decided about it. It seems like you are (finally) moving away from this and I appreciate this. If you do so, please be thorough and get rid of the historical baggage.

For example, consider the following streamlined system of review queues, all focussing on what needs to be done or decided:

  • A queue for providing guidance, editing, and checking for severe issues. This queue would be populated with first posts, late answers, posts with a bad quality heuristics, and possibly dedicated flags. The more reasons coincide, the more action a post needs to be de-queued. Possible actions would be like in the first-posts queue, maybe with a specific button for closure/needs to be deleted. Edits and comments from experienced users outside the queue can count towards de-queueing to some extent.

  • A queue for deciding whether a question should be closed. This queue would be populated through close votes and flags and the above queue. It would work like the current close queue.

  • A queue for deciding whether an answer should be deleted. This queue gets only populated by not-an-answer flags, a button in the first proposed queue, and possibly red flags. It only exists to decide about deletion.

  • The reopen queue and suggested edits queue can mostly stay as they are.

This is a riff on this suggestion of mine.

| improve this answer | |
7

I noticed a link to the timeline in the ... menu. Is that really useful? I never use the timeline while reviewing (probably because there was no direct link to it before) but I can't imagine it will be useful in more than 5% of the cases. I'd be inclined to leave the link where it normally is on the page, i.e. the clock icon under the downvote button (it's missing there right now).

| improve this answer | |
  • 6
    My idea about that was/is: either remove all UI actions for posts that distract from reviewing or make them available but then at the same location as they are in a regular view. – rene Apr 23 at 20:45
7

"Very low quality" option should send straight to the correct queues

I like that choosing "Very low quality" in the Help & Improvement queue won't sent the post right back into Triage, causing the endless loop issue that we talked about earlier.

However, sending those into the Low Quality Posts queue isn't the right thing to do in my opinion, because of how the queue works. The only actions that can be taken from that queue are to either mark them as looking OK, or send them to the Close Votes queue. There's no way to handle a post that needs a spam or abuse deletion (category 4 here), and it simply adds an extra step for questions needing to be closed (category 3 here). I think the proper thing to do is to open the flag dialog when clicking the "Very low quality" button, and have the reviewer choose a closing or flagging option, so that they get sent straight to the right spots.

Additionally, this change would bring questions into the same queue, which for many years on SO only, has consisted solely of answers. This brings me to another improvement suggestion.

| improve this answer | |
7

As you noted in Part II:

Most users learn how to be good reviewers by actually reviewing posts, but sometimes find themselves making wrong decisions.

Given this, I’d love to see this formalized into an explicit review training queue, similar to @PM-2Ring’s proposal, “Practice queues for review training”.

Review Training Queue

When a user first gains privileges for a queue, they will be presented with a set of curated audits which are hand-picked by moderators to be a representative sample of that queue’s review criteria.

When a user fails a training audit, there is no penalty. But they receive an explanation that educates them about the review criteria; e.g.,

Careful! While that answer may be incorrect, it is nevertheless an answer. Reviews should remove posts that are not answers; voting should be used to rank answers by usefulness.

Once the user has successfully completed each of the training audits, they will then receive a message:

Congratulations! You’ve finished the training period, and are now ready to review some actual posts. But be careful! After this point, unhelpful reviews may result in a review suspension. Still feeling unprepared? Be sure to read the FAQ.

Variations

  • Require the reviewer to correctly pass a percentage of training audits before being let loose on real audits. This would require having a larger pool of curated audits.
  • Require suspended reviewers to go back through the training audits before regaining their review privileges.
  • Prompt reviewers to retake the training audits after a failed audit—or even if their reviews are suspicious (e.g., based on frequency of Looks OK posts or minority reports).

Moderator Responsibility

This ties in neatly with something @Cody has raised on Stack Overflow:

This is why I (and several of the other moderators) are staunch advocates of revamping the audit system to allow us to nominate posts as audit candidates (both false-positives and false-negatives). We would select posts that are obvious and unmistakable, yet still represent corner-cases that often seem to trip up reviewers. This would be far more effective for pedagogical purposes, and I believe it would still scale adequately, as we could serve similar audits to all users without defeating the purpose.

I’m not advocating for anything so expansive here. But this would offer a simpler version of that, while prioritizing the moderator effort toward onboarding and training new reviewers.


The goal is to provide an interactive simulation of reviews as an easier way of training new reviewers—and a friendlier way of addressing initial mistakes.

| improve this answer | |
6

Thank you so much for making these great improvements. I have a few suggestions.

I think the button "Improve readability" should open an edit dialog instead of sending this to the H&I queue. The Help and Improvement queue should be removed altogether. The only real action in that queue is to edit. All users who have access to Triage can either suggest an edit or have full-editing privileges. There's no need to push this question into another queue.

I would also recommend renaming the Low quality posts queue. Put questions in Triage and Answers in LQP. LQP can be renamed Low Quality Answers.

As for the automatic heuristics that Shog9 has talked about I strongly agree to add more. I have recently spent some time researching NAA heuristics and I can tell there are few give away phrases/signs that make a post definitely NAA. Also, code-only answers are ok most of the time, only if it's more than a few lines of code it should be reviewed.

| improve this answer | |
  • 7
    "I think the button "Improve readability" should open an edit dialog instead of sending this to the H&I queue." I think I probably speak for a few people, but this would put me off Triage even more than I am now, I want to quickly review the quality of posts, I don't want to be slowed down having to edit them when that is the correct action, I'd rather mark them for editing – Nick Apr 24 at 0:18
  • Agree with Nick. Also, someone may recognize editing is needed, but is not able to do so (language, for example). – Cindy Meister Apr 30 at 16:09
6

Here is something that always bothered me with the Triage queue: how to handle duplicates?

Here is the thing: I can review questions and decide whether they are OK or not even if I'm not an expert. I can see that there is a good problem description, what is expected, what isn't. I can also read code in a language I don't use and figure out it's representative. Or I can spot if any of these is missing. Not always, as it might be something very specific that is outside my knowledge but in many cases I can say whether a question is OK or is missing something.

The problem: I don't know if there is a canonical duplicate for a given issue. Let's put aside other duplicates for the moment - canonicals are the ones that tag experts would be seeing all the time as it's a frequent question. Since I'm not a tag expert, I can't really say whether something is a really common question or not.

Right now, duplicate closure falls under "Unsalvageable". So, I often feel uncomfortable choosing "Looks OK" for a question I'm not an expert in when it might actually be a duplicate. I often skip these. Yet at the same time, it doesn't feel like I should do that, as I feel the question, even if a duplicate, can still be classified as a good question.

If I'm not mistaken, under the new review system the same thing would apply - duplicates fall under "Close this post" in Triage. So, I have the same set of concerns.

| improve this answer | |
  • I think you raise a valid point. But at the same time you need to remember that the post needs a few votes to leave the review so I think it is ok to mark the post as OK if it is to you. If it is not, someone else will mark it differently – Tomerikoo Apr 27 at 13:58
  • 1
    @Tomerikoo that's exactly my issue. I wouldn't like it if I choose "Looks OK" and then somebody comes in and marks it as a duplicate. Depending on the votes, one of us would be against the minority. It really matters little which one, as the core of the problem is that the review system is what puts as at odds, we can still agree otherwise that the question is fine as a question. It requires expertise to reach a common result. Dividing the votes because of required knowledge is not a good idea. – VLAZ Apr 27 at 14:02
  • @Tomerikoo: Wouldn’t that potentially trigger a review suspension? My understanding is that (automatic) review suspensions happen when you flag a post as Looks OK, but it is later closed. Or is there an exemption in place for Duplicates since, as VLAZ notes, an otherwise good post can be a duplicate? – Jeremy Caney Jun 10 at 23:04
5

Add better audits to Triage

I'd like to see better audits in the Triage queue. Right now, the only audits I see in the Triage queue are really good questions that should be reviewed "Looks OK", and blatant spam. There is no automated checking done to see if users are appropriately reviewing closeable questions as "Unsalvageable" (in the current model), which has led to a group of moderators handing out manual review bans in droves, to the extent that the majority of review bans resulting from Triage are manual.

Can we please add closeable questions to the review audit system for Triage, so that people can be prompted to pick the right choice ("Unsalvageable" in the current model, or "Add clarity" in your proposed new one)?

| improve this answer | |
  • 6
    Or, better yet, get rid of the current awful audit system and replace it with something that isn't designed purely as a trap. There's been countless audit fails where reviewers (including me) have gotten stuck wit ha bad audit that was mishandled originally. I use Samuel Liew's review queue helper userscript so I don't have to deal with garbage audits, and it honestly made reviewing a lot more pleasant. – Zoe The Lockdown Princess Apr 24 at 7:36
  • @ZoeTheLockdownPrincess: Cody has advocated for moderator-curated audits (reference) in order to minimize the number of false-positives and false-negatives—or, at least, audits which might not be especially useful from a pedagogical perspective. I really like that idea. – Jeremy Caney Jun 10 at 23:12
  • @JeremyCaney over on Stack Overflow, there's currently next to no mod flag handling. Adding more workload to the mods at this time won't work. – Zoe The Lockdown Princess Jun 10 at 23:32
  • 1
    @ZoeTheLockdownPrincess: That seems like a reasonable concern, though according to Cody: “This is why I (and several of the other moderators) are staunch advocates of revamping the audit system to allow us to nominate posts as audit candidates (both false-positives and false-negatives). We would select posts that are obvious and unmistakable, yet still represent corner-cases that often seem to trip up reviewers. This would be far more effective for pedagogical purposes, and I believe it would still scale adequately, as we could serve similar audits to all users without defeating the purpose.” – Jeremy Caney Jun 11 at 0:36
4

The instructions say

  • Consider using filters to find posts relevant to your interests and subject-matter expertise

Not all queues have them, and neither do all sites in the network. Are you planning to add them to more queues, or if not, will you remember to hide this bullet point if it's not applicable? There's little as frustrating as searching for promised functionality that doesn't exist.

| improve this answer | |
  • 1
    I think they are - it's mentioned near the end of the part II post. – Em C Apr 23 at 19:21
4

Adding more context to each queue

More context is certainly welcome, but I think that no matter what, people are going to be opening the question in a new tab. When triaging, and it is obvious that a question was well written this isn't as important, but certainly before closing or deleting a post I always like to get the full context of what was going on before making such a decision.

Perhaps you could just embed the original question in an iframe or make a plan around using HTML Portals when they come out.

| improve this answer | |
4

I see no mentioning of the Reopen queue.

The main thing that bothers me with that queue is that once a closed question has been edited, it enters the queue. This leads to many Leave closed clicks because a question that is closed as a duplicate shouldn't obviously get reopened after some grammar fixes...

One thing that could be useful is to show who actually did the edit (I couldn't find that at least in a convenient way). But another option is to take it one step further and either:

  • Don't allow people other than OP to edit a closed question.
  • Add a very visible message stating that this is a closed question and edits should be meaningful.

I think that allowing only OPs to edit closed question might sound extreme, but if you think about it, only they can really edit the question to make it on-topic. Whether the question is closed as a duplicate, needs details or focus, how can anyone else add details to make the question not a duplicate or more focused. If a question got closed, it probably means that only OP can save it!

Another, "softer", approach might be to allow edits from everyone, but only edits by the OP will send the question to the Reopen queue. Or, add a tick-box for anyone other than OP to mark if they believe the edit was significant enough to be sent to the Reopen queue.

| improve this answer | |
  • 1
    "Don't allow people other than OP to edit a closed question." I disagree with this because an edit can make a closed question reopenable, even if it's not from OP. Moreover, an edit can be worth it even if it doesn't reopen a question nor aim to. For example, a good duplicate can be cleaned up for presentation and kept as a pointer to the canonical. The idea is not to reopen the question but still to make it more presentable. I think the core of the problem is conflating "edit" and "reopen" concepts. – VLAZ Apr 28 at 5:39
  • 3
    From my experience (not that is very long), most reviews in the Reopen queue end up being Left Closed because most of them are minor edits which doesn't change anything. This makes me go less to that queue and I think that's a shame and pointing out that something should be done there as well – Tomerikoo Apr 28 at 6:05
  • 1
    That's also a reason I don't visit the review often. And also why I don't want "edit" to be the same as "reopen" - a lot of times edits are started before a question is closed and finished after. Either because it was a suggested edit that was approved later or a full edit where the question was closed in the meantime. Neither of those are done with the intention to reopen the question. Making a question uneditable when closed it soling an XY problem at best, if indeed "solving" it. – VLAZ Apr 28 at 6:09
  • So maybe offering 2 edit options on closed questions? Edit for readability, or edit to reopen? – Tomerikoo Apr 28 at 6:11
  • 1
    I'd personally make it a single edit option but then a separate deliberate action if you want to reopen. I don't like putting two edit buttons upfront. So, it would vaguely look like Edit -> Save -> (optionally) and reopen. Might be better implementations but I definitely think the reopen intention should be decoupled from the edit. So, pre-close suggested edits will not be reopening, pre-close started full edits will still ask you "do you want to nominate for reopening?" (in some fashion) after being done. – VLAZ Apr 28 at 6:15
  • +1. The overabundance of minor edits in the reopen queue also creates a vicious circle, in that it trains reviewers to always click "Leave Closed" unless the review is very obviously an audit. Which means that even actual reasonable reopen votes rarely pass review, which means that few people bother to cast them, which skews the review queue even further towards meaningless minor edits. – Ilmari Karonen Apr 29 at 18:35
  • 3
    … IMO, to get the close-and-reopen mechanic working on large sites like SO again, we'll need to keep the reopen queue short and containing only questions that someone has already deemed worth reopening. Which means either turning off the feature than sends edited closed questions to review entirely or moving those questions to a separate queue. – Ilmari Karonen Apr 29 at 18:36
  • 1
    @IlmariKaronen, I'd say that only an edit by the author should automatically put the question into the re-open queue. An edit by a third party should result in the editor being asked if the question should be placed in the re-open queue or not. – Mark May 8 at 1:23
  • 2
    @Mark: That would help some, but a lot of the minor edits that show up in the reopen queue are by the author. They just don't even try to fix the issues due to which the question was closed, probably because the author isn't even aware that making a random edit auto-submits the question for reopening. I guess one way to fix it would be to have a popup show up after editing your own closed question that says something like "If you've fixed all issues with your question, you can flag it for reopening. YOU CAN ONLY DO THIS ONCE!" with buttons to "Flag for reopening" and "Not yet". – Ilmari Karonen May 8 at 12:29
4

What did you wish was part of your Review Queue onboarding?

More emphasis on QoL features

Filtering is pretty much the best way that you can prevent new reviewers from getting discouraged with the review queues. This would make them feel more effective and confident of their decision. Make it part of the onboarding route to set up a filter for the queue you acquired privileges, then introduce the skipping and afterwards the rest.

| improve this answer | |
3

In Triage, do you anticipate any issues with sending Add clarity posts to the Close Votes queue?

I think this is exactly what is needed, but please can it be worded to clarify whether the editing needs to be done by the OP or others in the community? Also, the wording "Add clarity if the question is clear" seems very odd.

| improve this answer | |
3

What did you wish was part of your Review Queue onboarding?

More guidance. I have had access to reviews for so long I do not recall when I gained the privileges. However, I have rarely used them. Despite going through the onboarding guidance and trial reviews a couple times, I still do not know what it is looking for and have zero confidence I would review items correctly. I think some examples of how things should be reviewed would help. I'm not talking about easy cut and dried items either. The trial ones are great in that it's difficult to decide whether to mark for improvement or close. But as I recall there wasn't much guidance on why they should have been marked for closing.

| improve this answer | |
3

Regarding the Suggested Edits review queue, it is very disappointing to see no attention placed on the canned message choices available to the reviewer, or consideration of whether the review UI could offer some way of allowing for a reviewer to be more helpful in training new editors.

It feels ironic that the canned messages are so harsh in the face of recent campaigns for a more welcoming and friendly environment. Why do I say harsh? Well, at some point, I thought I could increase my contribution to the SE sites by beginning to edit to improve. I recall getting slammed down with a terse canned message on occasion and recall that the message associated with the rejection offered zero help in understanding why a well-intentioned edit (a significant one) was rejected with no apparent avenue for being reconsidered. It seemed that the reviewer missed really understanding what was behind the edit. The overly harsh, terse, rigid, and unhelpful canned messages with no place for the reviewer to "explain", or for me, as the editor, to "explain", successfully caused me to stop trying to improve posts.

Before today, I'd only done 9 reviews, and I'd forgotten what it was like to review edits in light of my experience, so I went through a days allowance of reviews today to (re-)experience for myself how a reviewer might feel when trying to be sensitive to people learning how to edit. The experience was illuminating and very disappointing.

It seems that at some point reviewers must get hardened to how their rejects are going to be received by the people whose edits are being rejected because the choice is either to not review or continue to use the square-peg-in-a-round-hole canned messages.

After today's experience, I better understand now why I felt slammed. The system actually promotes it. The messages are canned, and terse, and are (mostly) not written to allow for cases where a reviewer might be overly picky or where the editor was really not so far off the rails as to warrant being slapped down with a radio-button-selected canned message that was the best of all the poorly fitting one-shot messages possible.

Clearly some edits seemed to warrant, to some extent, the harsh, succinct canned messages, but a lot of the time, chose to skip because there was no way to tell the new (low rep) editor what it was about their otherwise okay edit made accept not viable. In some cases I edited and left the modified edit for someone else to review, but it wasn't uncommon that I wasn't really in a position to edit either - not necessarily being a subject matter expert, or not really having enough context supplied in the review UI.

The exercise revealed that what felt like to me a personal response was not in fact anything personal at all, but rather a canned message that offered no way for me as a reviewer to hint to the editor specifically why I could not in good faith accept the edit even though the edit contained noteworthy and helpful changes.

I did skip a LOT to avoid cases where I didn't know the material well enough to make an edit better (to avoid reject), or to avoid gnarly cases where an edit had a big enough problem to not warrant accept but did not warrant a response written as severely as the canned message options offered.

I saw over and over that while one fixed canned message was more appropriate than the rest, it was deficient in one way or another and that it was truly unfortunate not to be able to provide a low-rep editor some more pointed feedback right then and there with a text entry box as appears by Causes Harm.

In some cases, Causes Harm with its edit box was usable even if it required a bit of a loose interpretation of "harm". (What is harm by the way?) More often than not, though, I knew that having to radio-button-select a more appropriate canned message was going to be frustrating to new editors in the queue.

I think there is much more here to do than training reviewers. The queue tools themselves need to be much better geared toward helping new reviewers tell editors (nicely) what they are doing wrong so they aren't put off by harshly worded canned messages that primarily fit worst-case edits.

Regarding nicely, see Also: What’s the course of action when you think someone is being unjust?

It seems sad that the answer to the question here is that the new editor needs to cherry pick what to edit to avoid getting slammed by a reviewer until one reaches a (huge) threshold of 2K edits, and that the harshness of the canned messages isn't addressed by any answer. I suspect it may not be uncommon for people who feel unjustly slammed to stop contributing rather than do what this editor did (posting a question to meta).

On another note, there were some edits where it felt like it would be better to move the edited message to a poor-quality question/answer review queue instead of spending time reviewing formatting/spelling on a post that really needed content work by the original poster. Some edits were entirely legitimate for formatting/spelling improvement, but it was clear that the effort was wasted unless the OP improved the question. It seemed pointless to reject the edit, but also pointless to accept it. It wasn't immediately clear how a new reviewer would know how to handle this situation from the review UI. It seems like that could have been integrated rather than require the reviewer to find another way to do it (i.e. flag).

I don't really know if this is the place to register observations or not. This series of posts doesn't actually make it very clear where community contributions are to be placed.

| improve this answer | |
3

What did you wish was part of your Review Queue onboarding?

What would have been great is a short (10-15 minute) video explaining how the review process works. For the triage queue specifically, it could include two or three examples of questions for each category along with a discussion of why the question gets the category it does.

Following the intro video, the new reviewer could be given a quiz with 20-25 carefully curated questions - similar to the audit questions, but instead of suspending them for getting one wrong, they could be given the "correct" answer along with an explanation. Only when they get enough quiz answers correct would they be given access to the real Review Queues.

What feature/aspect was difficult to discover as a new reviewer?

I had no idea that review suspensions were even a thing until I discovered them the hard way.

| improve this answer | |
0

Edit as an option in Close Vote Queue should be removed.

People that have just started using queues will use it in a wrong way, and experienced users don't need such option nor guidance.

If the question can be sufficiently edited by a reviewer, they can always use Leave Open option and subsequently edit question.

I am not in position to comment how that option works on other sites, but on Stack Overflow, if the question is closeable, usually there is very little reviewers can do to put the question in shape. Situations where they can make question on topic are quite rare.

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .