24

This time last year, the Public Platform team announced an initiative to improve the review queues. Our goal was to explore new ways to better support our curator and moderator communities by revisiting the review queues used for community moderation.

With the help of the community, we have been able to accomplish a significant amount of work and made multiple releases including:

We’ve made a lot of progress with this project, but there is still work to be done. Today, we’re coming back to the community to discuss one of our last objectives: improving queue workflows and pathways. In parallel to our other releases, we’ve been taking more time to carefully consider what changes would make the most impact on the review queues. We’d like to share some new proposals and get your feedback.

The original plan

Our initial research helped us identify the following pain points in review queue workflows:

Problem #1: The Requires editing action in Triage was sending a surplus of unsalvageable posts to the Help & Improvement (H&I). H&I reviewers are then limited to either Skip or mark the task as Very low quality, having it return to the Triage queue and creating a frustrating endless loop.

Solution (implemented): We updated the actions available in the Triage queue and redirected tasks to where they may be better addressed.

  • Needs community edit → Help & improvement queue
  • Needs author edit → Close votes queues
  • New flag action for other problematic posts

Problem #2: First Posts and Late Answers queues and combining those tasks into Triage. These queues were intended to give good posts a head start toward success and filter out bad posts, but tasks could see better, faster intervention.

Solution (not implemented): Initially, we considered deprecating these queues. When we got into technical discovery, we discovered that we couldn't simply move First Posts and Late Answers tasks into Triage because the Triage queue only accommodates question tasks.

New considerations

We wanted to look at these problems again with fresh eyes. We revisited our internal conversations, reviewed some data, and reread the Meta feedback from previous posts to ideate on new solutions.

Deprecate Triage

  • Triage is a Stack Overflow exclusive queue and was created to quickly assess and provide feedback on questions that had been flagged low quality. This queue is often the root cause of other review queue issues. It lacks strong guidelines for reviewers to follow and assumes that they can identify good questions, both leading to inconsistent reviews.
  • Currently, the Low quality post queue on Stack Overflow only handles answers. By deprecating Triage, we’d like to adopt how the Low quality posts queues on other SE sites function and route all posts (questions and answers) that have either been flagged as low quality or fall below the system-generated quality score threshold to Low quality posts.

Deprecate Help & improvement

  • H&I hasn’t been successful in its intended purpose with issues from Triage diminishing the queue’s usefulness. Improvements to other queues could still satisfy the same need.
  • H&I is our least productive queue. Over the last 90 days, only 7 tasks went into the queue each day on average, compared with 2,458 for First Posts (our busiest queue) and 219 for Reopen Votes (our second-to-last busiest).

Create separate First posts - questions and First posts - answers queues

  • The First posts queue had good intentions in helping new users with their first posts, but the queue itself has no clear goal. Reviewers can take any or no action against a review task and is often not impactful or useful to the post.
  • We want to separate questions from answer tasks and identify specific actions that may help new posts fare better as well as maintain content quality on the site.

Keep the Late answers queue

  • The Late answers queue has been effective in removing low quality posts, repeat answers, and spam.
  • Similar to First posts, we want to identify specific actions that help in resolving problematic answers.

Other: Prioritize aging review tasks

  • Review tasks age out of the queues if they are not acted upon, so we’d like to ensure posts get the attention they need in a timely manner.

Next steps

We will not be conducting 1:1 user tests or interviews in this release. Instead, we would like to open this up for discussion on Meta to gather a number of opinions. Please share your thoughts until April 28, 2021. Afterwards, we’ll analyze your responses and share new designs. We’re excited to read your responses and look forward to learning from your insights.

8
  • Is the problem of "improving filtering and discovery tools" part of this step or an upcoming one? – TylerH Apr 21 at 19:56
  • 4
    For clarity, is this only for Stack Overflow or is this intended to be applied across all Stack Exchange sites? – KillingTime Apr 21 at 20:42
  • 1
    Like this very much. Frankly we could have done with (and still might find helpful) "triage" on worldbuilding.se since yahoo answers closed, I suspect a similar loading may have occurred on other sites. – A Rogue Ant. Apr 21 at 21:24
  • So first posts is becoming the new triage? – Luuklag Apr 21 at 21:35
  • 1
    @TylerH "improving filtering and discovery tools" is not part of this step – Lisa Park Apr 22 at 2:06
  • @KillingTime It's definitely Stack Overflow-centric, but we're open and interested in changes that can effect the entire network (ie Triage for SE?) – Lisa Park Apr 22 at 2:07
  • 2
  • 2
    Unfortunately I've only just been pointed to this question today (!2 May) so I'm probably way too late. The extra clicks now mean that I've basically given up on handling reviews. The idea might be good but the implementation definitely isn't there (for me) – roaima May 12 at 15:19
21

I would like to reiterate something I brought up in Improving Review Queues - Design overview I: Onboarding and updating workflows regarding the workflow of the Reopen Votes queue.

The main problem with it currently is that any1 edit on a closed question sends it to the reopen queue. This means that "meaningless"2 grammatical/formatting corrections which obviously doesn't make the question reopenable - will send it to the queue. The biggest pitfall with that fact is: the question only gets one pass to the queue so if a meaningful edit comes a long, the question is less likely to get reopened. The collateral damage is simply wasting reviewers time.

Some possible changes (in no particular order) are:

  • Grammatical/formatting edits do make the question better - we still want to encourage them - but not reopenable. Most chances are that only the OP can really edit the question to bring it to a reopenable state - only OP edits should push the question to the queue. This will actually align better with the new Triage's "Needs OP edit". It currently flags the question for closure expecting the OP to edit, only to be sent back to the reopen queue by any random edit.

  • Add a (very) visible warning (similar to the new onboarding modals) when editing a closed question that this edit will put it in the reopen queue, so the edit should be really meaningful. Bonus: add the same warning to the review of edits of closed questions.

  • Add a meaningful edit or edit and reopen checkbox to the edit screen that editors can mark if they believe the edit changes the state of the question and it should get to the queue. By default, of course - it will not.

  • Make suggested edits on closed questions require 3 (or more) approved reviews instead of 2.


You mentioned that the Reopen Votes queue is the second to last busiest, but I still believe that making its items the most relevant possible will make it more focused and will encourage more people to use it and get some questions reopened! Personally, I got tired of just leaving closed due to meaningless edits and barely use the queue...


Related discussions:


1. Not really any edit, but most. See Which edits push closed questions to the reopen review queue?

2. Meaningless in the sense of making a question reopenable. I'm not saying that grammatical/formatting edits are meaningless in general.

4
  • 6
    The fact that non-OP edits send posts to the reopen queue is contradictory to how Triage works– in Triage, "Needs OP edit" flags/ votes to close the question. Why is it possible for a question to get closed in Triage for needing an OP edit, and then later sent to Reopen queue by a non-OP edit? Something needs to change here, and I really really like the first idea you list. – zcoop98 Apr 21 at 22:41
  • 1
    @zcoop98 That's a very good point. I hope you don't mind I added it to the list because it's really a strong claim about queues consistency – Tomerikoo Apr 21 at 22:51
  • 1
    I remember Wikipedia used to have a "minor edit" checkbox (though it seems to have disappeared); perhaps there should be one for formatting fixer-upper edits to mark them as not changing the substance? – tripleee Apr 23 at 7:50
  • @tripleee I suggested something similar in the third bullet point but with the reversed logic. This is assuming most people won't bother to mark their edit as minor and we will have the same problem. On the other hand, people that are aware of the system will know to mark it as meaningful – Tomerikoo Apr 23 at 9:13
10

One big problem I have with First Posts queue is that it has both questions and answers. This often leads to confusion when users see a question posted as an answer. They edit the post instead of flagging it as NAA. Separating the queue is definitely a good idea, but the question is whether we need a specialized queue like this. Could we not have "Low-quality answers" and "Low-quality questions" queue instead?

3
  • 5
    Are you saying that first posts would go into low quality? My concern if we did that, is that I'd want to rename the queue because "low quality" is not equivalent to "first". Someone's post being their first doesn't make it low quality and we're priming people using the queues to assume the post is "low quality" by calling it that. If we're only throwing stuff that is very short, flagged as such, or downvoted into the queue, the naming makes sense but it doesn't make sense for first posts or late answers. – Catija Apr 21 at 21:04
  • 3
    @Catija True, but even now, posts are added into LQP despite being perfectly fine. I doubt normal users are flagging them, which means that they are put there automatically. The actions available are already what we expect them to do with first/late posts. Vote, flag for deletion or edit. – Dharman Apr 21 at 21:07
  • 3
    Think of it from the pov of someone whose post was in the LQP queue - "your post was marked by the system as low quality" - that's just... not cool. And maybe it's "not cool" by current standards, either, but at least there was something actually about the post that our metrics caught as being concerning - or that a user flagged (such as NAA) - That doesn't guarantee that it's actually "low quality" but it's at least more of an indication than just being a first post by a new person or a late answer to a question. – Catija Apr 21 at 22:15
7

When thinking about this, and the Triage queue especially, I always recall an answer from our beloved Shog9.

Shog presented the following numbers:

So what good is Triage?

Ok, now we can get to some numbers. From my perspective, it's still a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. But that doesn't directly help folks just trying to browse the site without bumping into lousy questions.

So let's focus on views. In May of 2018, 256,414 questions were posted on Stack Overflow. Of those, 31586 entered Triage, and 25508 were reviewed completely (that is a consensus was reached). The average view count across all questions posted in May was 89; the average view count for questions that entered Triage was 64; the average view count for questions that completed Triage was 67.

Broken down by Triage consensus:

Triage result Questions Avg Views
Looks Good 11379 76
Should Be Improved 9825 66
Unsalvageable 4299 48

And drew the following conclusion:

....

However, Triage is not unsuccessful. Beyond its utility as a tool for moderation, it does at least partially fulfill the original goal: ensuring that fewer people have to look at terrible questions.

As we can see from the table Unsalvageable posts get about 2/3th of the views Looks Good posts do. So in all fairness, Triage does do a job at getting rid of garbage.

Now why does it do that job, because questions don't face any delays into getting into triage. They are added as soon as they are posted, and meet the criteria.

The flowchart of the review queue's shows how interrelated the Triage and Low Quality Posts(LQP) are. They essentially serve the same purpose. So combining them could make sense, when the implementation details are good enough.

Flowchart

What details should we get right then if we want to merge Triage into LQP?

  • Set an appropriate reputation threshold
    Currently the Triage queue requires one to have 500 reputation. It's a fair level I think, as it helps you ease into reviewing with very little you can seriously break. LQP only becomes available at 2K reputation, which requires way longer to achieve. So simply moving the Triage posts into the LQP would greatly reduce the number of available reviewers, making the queue's even more backlogged.

Are there any other options?

  • Yes, just keep triage as it currently is.
    The most major problem we had with it, the infinite loop between Triage and LQP has been cut off. Also I think that it is really good at flagging posts, also something that we need to learn new users to do effectively. We could use the review system to provide feedback on declined flags that came from the Triage queue, and use that to educate users on when it is appropriate to use what flag.

Disclaimer: I set out to write a coherent post, but I kinda feel I failed in the end...

1
  • I think the missing piece here is that reviewing questions is vastly different from reviewing answers. Lumping them together encourages context switching while removing the ability to distinguish fairly benign actions (recommending closure) from fairly severe ones (recommending deletion). – Shog9 Apr 23 at 1:07
6

Triage is a Stack Overflow exclusive queue and was created to quickly assess and provide feedback on questions that had been flagged low quality. This queue is often the root cause of other review queue issues. It lacks strong guidelines for reviewers to follow and assumes that they can identify good questions, both leading to inconsistent reviews.

By deprecating Triage, we’d like to adopt how the Low quality posts queues on other SE sites function and route all posts (questions and answers) that have either been flagged as low quality or fall below the system-generated quality score threshold to Low quality posts.

I'm of the opinion that the low quality posts queue isn't supposed to be used for questions, period. To get them closed or have the community make the edits necessary to keep it open, you use the close-vote queue. The close-vote queue already asks community members to edit if that means a question can stay open, there's no need to ask the same question in a different queue.

You mention it in this post too: Triage was used to send questions where people picked "Needs author edit" to the close votes queue. Not 'the low quality queues', where people would be making the same decision (leave open, edit, close). Deprecating Triage and just sending more questions to the low quality posts queue is just moving a problem around.

If you're going to put questions in a low quality queue, make the queue about whether or not a closed question should be manually deleted, not about whether or not the question should be closed, that queue already exist.

Of course, such an implementation of a low quality queue (delete or not delete) comes with a few risks:

  • Closed questions that are deleted before the OP has had a chance to make those edits only an OP can make.
  • Every closed question ending up being flagged into that queue, even the ones that don't need much intervention because they will be cleaned up by the Roomba process.

Questions right now can't be deleted by the community unless they are closed first, and then they need to be closed for >48 hours, or closed and having a score of -3 or less. to be eligible for 'fast deletion'. So any question that can be flagged into this queue should probably meet the 48 hour criteria at least, so the author had a chance to make any edits.

The point about questions that would be cleaned up by Roomba is a tricky one. Just saying 'wait the entire 9 days before anything can be flagged into this queue' is probably not the best solution as it takes away some of the power of the fast-deletion mentioned in the previous paragraph.

Then again, you also don't want people flagging everything that has been closed for 48 hours into that queue. The queue would be most helpful if only posts that won't otherwise get deleted or really need fast deletion end up in it, for example closed, low quality questions that can't go through the Roomba process because they have a FGITW low quality accepted answer, or blatantly off-topic posts.

So, in my opinion, if implemented correctly, a low quality queue for questions:

  • Doesn't duplicate the workflow of the close-vote queue.
  • Would still give a post's author the time to make necessary edits, before their post ends up deleted.
  • Should offer a bit more visibility to people into what should be candidates for deletion than the 10k tools.
  • Would encourage only doing useful review work, and not just be a manual replacement of the Roomba process.

This would mean a lot more work than just dropping questions in the current low quality posts queue, but it would improve workflows a lot.

2
  • For users who don't have 10k on any site it's hard relating to this post. I've found the queues are always different from what I had imagined when I finally get there. – bad_coder Apr 22 at 20:01
  • Well @bad_coder the 10k tools aren't a queue at all. The access to them is 'hidden' behind the review queue icon, and what it does is basically give you a list of posts with pending (un)delete votes. It just isn't as visible as the review queues, and there are e.g. no red dots to indicate that there's new stuff you might want to judge. – Tinkeringbell Apr 23 at 9:25
5

Create separate First posts - questions and First posts - answers queues

I think it's a good idea to emphasize different actions for different post types, but are separate queues really necessary? Right now, the Low Quality Posts queue has different actions depending on the post type as well. (I'm ignoring the fact that the use of the Low Quality Posts queue for questions is not universally accepted.)

If you still go forward with separate queues, what will happen with past reviews? Will they be split as well? What would happen to a Steward badge awarded to a user reviewing 600 first answers and 600 first questions?

5

I think of the 2 arguments one isn't precise and the other isn't accurate.

  • Deprecate Triage

    This queue is often the root cause of other review queue issues. It lacks strong guidelines for reviewers to follow and assumes that they can identify good questions, both leading to inconsistent reviews.

In other words, the root cause might be that users with 500 rep get access to Triage, if there's any correlation between reputation and experience (a few like to argue there's no correlation) it's obvious a 500 rep user is unlikely to have enough experience with site moderation to be effective at flagging and duplicate finding.

There weren't tag filters just a couple of months ago, users had to click through every technology under the sun and make an unreasonable number of evaluations regardless if they had expertise or experience.

I don't think the elapsed time since introduction of the tag filters allows for any significant data about changes in the quality of Triage reviews. If such data exists it hasn't been presented. As such a major change is being proposed based on the former business rules that have themselves been deprecated. (I can only read this as a "lesson learned" that inexperienced users aren't on average accurate enough in reviewing.)



Deprecate Help & improvement

H&I hasn’t been successful in its intended purpose with issues from Triage diminishing the queue’s usefulness.

There are less than 90 stewards from H&I and I'm one of them. Most of the arguments I heard against H&I during that time did not come from someone with proven experience in that queue.

(...) and look forward to learning from your insights.

All the H&I stewards I've read commenting about the queue liked the experience and thought it added value. The art of copy-editing seems to be disdained. Besides, I think H&I is a great copy-editor school because you risk getting review suspended for not editing right.

Talking from experience, I see a ton of posts daily (perhaps more than not) that would benefit from going through H&I. The only problem is: the posts aren't being sent to the queue.

Deprecate Help & improvement

H&I is our least productive queue.

Yes but it's also the most effective. You don't see any pending review items in that queue because users compete to get their cut of the action (folks want the extra custodian/reviewer badge to complete their collection).

Over the last 90 days, only 7 tasks

Which amounts to 630 edits that might not have been done otherwise. (I fail to see how this doesn't improve "the product" - or "quality" if you prefer.)

Create separate First posts - questions and First posts - answers queues

I read this last point as the TL;DR. Raise the reputation bar for questions (because that requires more skill) and turn the "Answers FP" pretty much into what Triage queue is now.



H&I could be integrated effectively into the existing review queue business logic by automatically dispatching items to it that fulfill a number of criteria (a lot of the bots on the network already check for this):

  • Fluff in post, like "Thank you", "Please help ASAP", etc... Automatic dispatch to H&I if it passes initial Triage with "Looks OK".

  • 1 single tag on post, send to H&I for double check that tagging is accurate.

  • 2 tagged keywords in title?! -> H&I...

  • Long stretch of what appears to be code without a source fence, H&I...

  • Several "lemma" programmatic keywords (like the syntax highlight is using) without inline literals?! -> H&I...

  • Post that is 1 long quote and nothing else (poster inadvertently pressed quote button) -> H&I...

  • Image links that aren't rendering inline? -> H&I...

(And have the system generate a digest of the criteria for potential H&I reviewers to get the hints.)

If 2 or 3 reviewers in H&I say: "No edit needed" then it probably doesn't need it. (One of the reviewers is sure to find something that can be improved in the post, there always is some detail...)

2
  • 1
    "if there's any correlation between reputation and experience (a few like to argue there's no correlation)" There's certainly and objectively a clear correlation between reputation and experience; there isn't necessarily a causal link between the two. – TylerH Apr 22 at 13:31
  • 3
    I think what they are trying to say is there is no clear correlation between high reputation and actual competence with the site. There are many high-rep users who got there by ignoring or even actively subverting the site's quality guidelines. – tripleee Apr 23 at 7:53
2

In addition to creating all these new queues, please also provide a place were one can access posts from all queues at one place.

Why? If one filters the review queues for a narrow set of tags one is active in, it is hard to notice when there are items to review. The red status dot is no help, because it does not take into account one's filters. So the only possibility is to visit the queue from time to time and with good luck, it will have something to review. For the close review that works okish, but for all other queues the probability that there will be something to review is so low that it is easy to give up on them altogether.

I think this would help to bring the users who are actually most familiar with a technology together with review items interesting for them.

2
  • 4
    We've generally avoided this because of the concern that someone would be constantly context switching... first I get a close review, than a reopen review, then a suggested edit, then another close then a low quality post - My concern would be that putting all the queues in one bucket would mean that someone would be more likely to err in their action. As an example - If a reopen review was immediately after a close review, one might think that they were being asked to judge whether it should be closed and accidentally reopen it. Is this a concern you have? – Catija Apr 21 at 21:00
  • 2
    @Catija I would rather appreciate the constant context switching - this would make reviewing less monotone and keep me more alert – samcarter_is_at_topanswers.xyz Apr 21 at 21:07
2

H&I is our least productive queue. Over the last 90 days, only 7 tasks went into the queue each day on average, compared with 2,458 for First Posts (our busiest queue) and 219 for Reopen Votes (our second-to-last busiest).

I would be very cautious about this one. In fact, I would instead investigate how many "looks ok" where later edited, which would be candidates for the H&I queue. Remember that folks usually see things in the extremes, when many things are actually in the middle. Triage right now is a "flag/no flag" queue, which of course would reduce significantly the volume of questions that are sent to H&I.

By deprecating Triage, we’d like to adopt how the Low quality posts queues on other SE sites function and route all posts (questions and answers) that have either been flagged as low quality or fall below the system-generated quality score threshold to Low quality posts.

I would recommend that you review previous documents about why was this created in the first place. The Plan™ was actually the other way: remove first post and keep triage and H&I, because first post was a very suboptimal queue: post marked as "no action needed" when they definitively did need some action, which makes the entire queue a waste of human resources, since it doesn't add anything substantial towards the goal of achieving quality. Same thing as triage, look at how many First Post were edited/closed after the review task was completed and which kind of action was it completed with.

2
  • 1
    We understand that first posts isn't awesome - that's why we want to improve it - but I don't think that keeping Triage is the right answer to that. First posts can end up looking a lot more like Triage but that doesn't mean we need to keep Triage or H&I to do that. Additionally, even if we remove first posts, we also have to keep it because first posts is for questions and answers, and Triage is for questions only. It makes more sense to me to get rid of these two and improve first posts. – Catija Apr 21 at 22:10
  • @Catija I would then evaluate what value a queue for first answer add to the overarching goal. Remember that triage doesn't catch every new question either and that was fine. I would first focus what are the overarching pain points and see how the current workflow does address them. I would rather have useful queues, than just have queues. Low quality and late answers also address specific answers, rather than all answers. Those I feel are more useful. – Braiam Apr 21 at 22:27
-1

"Triage is a Stack Overflow exclusive queue and was created to quickly assess and provide feedback on questions that had been flagged low quality. ..."

On sites without a Triage queue we sometimes get questions that aren't a clear single question, or two related.

When there's too many questions in one we can use "Needs more focus", and when an answer doesn't answer the question we have an 'answer flag' called "Not an answer" - but currently we don't have a "Not a question" flag, leading to the use of "Needs details or clarity"; which is fairly unsatisfactory if it's clear that the question won't be fixed by editing.

In the spirit of the 'answer flag' "Not an answer" we need a question flag called:

◯ not an question
This was posted as a question, but it does not attempt to ask an answerable question. It should possibly be edited, but most likely deleted altogether.

The wording was kept as close to the other flag's description, with changes to accomodate the difference; I'm not opposed to different wording. It should be clear that like NAA this NAQ should either be headed to editing or deletion.

Other questions involving this issue:

Note: In the Charcoal HQ chat there was a hope that one-day we would have a NAQ (or this noisier search term) flag someday, for questions that aren't spam or rude but also are not a question; and are most unlikely to be saved by an edit, leaving deletion as the most likely outcome.

That could be resolved by a queue for deletion of the question, where it's survival is determined similarly to the NAA reviews. Once it enters the queue the "Doesn't seek input or discussion" result is appropriate if it doesn't appear editable.

8
  • 3
    Please don't bring Charcoal into this to support this request. The discussions in Charcoal about "Not a Question" have been about having an additional feedback response to SmokeDetector reports (indicating that the user would vote to close), not about a "not a question" flag. – Makyen Apr 21 at 23:48
  • One can see from the link to the search that isn't the case, only a couple of people have restricted their comment to Charcoal feedback; and exactly what would it connect to on the other end. We've had this discussion recently. – Rob Apr 21 at 23:57
  • What I see is that most of the people are talking about SmokeDetector/metasmoke feedback, with a few people, primarily you, mentioning that they would like a NAQ flag. – Makyen Apr 22 at 0:08
  • 1
    My issue with mentioning Charcoal here is that the way it's mentioned appears to imply the Charcoal project desires a "Not a Question" flag. That is not the case. Individuals may or may not desire such a flag, but the Charcoal project has definitely not taken a position on SE having a NAQ flag. In fact, the position we've taken about NAQ is that having such an option for feedback to SmokeDetector/metasmoke is too much work, for no real benefit to the goals of the project (i.e. the project exists to identify/remove spam/rude/abusive content, not to categorize questions as off-topic). – Makyen Apr 22 at 0:20
  • Your "not a question" flag is very similar to the already existing "Very Low Quality" flag, which, to a significant extent, already effectively does what you're asking. The difference is that the VLQ flag (on SO) puts questions into the Triage queue (on SO) or the Low Quality Posts queue on other sites. In those queues, people are already asked to close/delete the post. How is your proposed "Not a Question" flag different from a "Very Low Quality" flag? – Makyen Apr 22 at 0:35
  • @MetaAndrewT. That's a relevant tag not-a-real-question, but it looks like the last active question was not after the seperation of MSE/MSO. – Rob Apr 22 at 3:50
  • 1
    Something I think is worth adding here, with the mention of VLQ flags on questions, is that this flag is historically handled poorly and ambiguously defined, and SO doesn't handle it very well. This is despite the fact that it may otherwise sound like the perfect flag for a "not a question" question. – zcoop98 Apr 22 at 14:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .