Let's have an explicit triage system for questions from new users

Question

This is a follow-up feature-request based on: How can we make First Posts review actually useful? - big thanks to everyone who participated there, I've tried to incorporate the feedback into the design below.

I've spent the past few weeks thinking about how we can do a better job of handling questions from new users (folks with no past history to speak of on our sites). This is challenging, because there are two goals that don't cleanly align:

• Triage new posts - eliminate spam and other forms of abuse, queue up low-quality posts for further review, mark good posts as worthy of further attention. Ideally, this should happen as quickly as possible after the post is created, so that both the author and the rest of the site can benefit from the actions that follow.

• Welcome wagon - help new users settle in, provide them with advice and assistance as they learn to use the site. It's nice if this happens reasonably quickly, but not critical.

Right now, the system fails to fully achieve either of these goals:

• Reviews take too long - there's a 15-minute waiting period before posts even enter review.
• Reviewers are not encouraged to take their time when welcoming new users, as tasks are quickly snatched up by others. A post might be dismissed as needing no action by one reviewer when another would wish to take the time to offer useful guidance.
• The outcome of review is insufficient to properly direct the majority of new posts: on the top Stack Exchange sites, nearly 60% of reviews end with "no action needed", and under 20% are voted on.

Therefore, I recommend the existing system be scrapped, and replaced with the one described below. Note that for brevity, I'm focused entirely on questions in this post - a corresponding review queue for answers should also be created, with more or less identical functionality.

Two-level review: mandatory triage, optional welcome

The strategy I'm aiming for here is to first collect as much signal as possible in the shortest amount of time, and then allow conscientious reviewers an opportunity to do a bit of hand-holding when and where they feel it is useful to do so.

Rather than trying to make the normal UI work for these two different purposes, I'm just using two different, consecutive interfaces: first triage, then welcome, with guidance and restrictions appropriate for each.

Note: all screenshots are fake. So is the reviewer guidance in them. I'll write less boring text before this is done - suggestions welcome!

Triage review

This should be very similar to the existing Low Quality review: only three possible actions (choosing "Abuse" should present the flagging dialogue with three additional options: spam, offensive and 'other'). Unlike Low Quality, there's no direct way to recommend closure or deletion here - the goal is merely to capture enough information to decide whether or not that should be the next step. Basic functional requirements:

1. Only the first 1 post(s) scoring 0 from users with less than 10 reputation should enter the queue (values in bold should be site configurable, however). Whenever possible, they should enter the queue immediately upon being posted (without requiring a waiting period or queue synchronization) and be removed when they no longer meet these criteria.
2. Each review task should be shown to only one reviewer at a time, no matter how many reviews are ultimately required to complete the task. So far as possible, the system should avoid encouraging reviewers to "race" to get credit - reviews should be fast enough without this.
3. Posts should be removed from the queue after receiving FP_REVIEWS_REQUIRED reviews in a single category ("Looks OK", "Needs improvement", "Abuse"), where FP_REVIEWS_REQUIRED is initially 3 on Stack Overflow and 2 everywhere else. (As noted above, posts that are no longer eligible per #1 should be removed as well.)
4. Posts that attract FP_REVIEWS_REQUIRED "Looks OK" reviews should be bumped to the front page by Community.
5. Posts that attract FP_REVIEWS_REQUIRED "Needs improvement" reviews should enter the Low Quality queue.

All auditing should take place at this stage, with a mixture of known-good, known-bad and known-spam posts used to evaluate the reviewer's ability to correctly evaluate the essential nature of a given question.

Reviewers who choose "Abuse" will be required to choose a flag type and will then progress on to the next review. Reviewers who choose either "Looks OK" or "Needs improvement" will progress to...

Welcoming review

At this point, further action on the part of the reviewer is optional. For those that wish to aid the asker in some fashion, guidance will be provided based on their triage review. For "Looks OK",

...and for "Needs improvement",

All the usual post tools become available at this stage, just as in the current First Posts review. All are optional - the reviewer can click "I'm Done" immediately without further action. Certain actions taken as part of the "welcome" review may remove the need for further triage reviews, however.

Basic functional requirements:

1. Action taken should be recorded along with the review (but not displayed publicly - as is done now).
2. If a flag, close or edit action is taken, the post may be dequeued immediately without waiting for further reviews. Voting may implicitly do this as well, if it causes a post to become ineligible.

Concerns

• There may not be enough feedback for reviewers in the Triage stage. gnat suggests that more audits may help here - perhaps we should increase the frequency of audits for new reviewers?

• There's really no extrinsic motivation for doing anything in the Welcome stage. I'm mostly fine with that; I think this is too complex, too personal for extrinsic motivators to have any positive effects. But there's a possibility that we'll see interaction with first-time users drop even further.

• It's kinda complicated. There were already more moving pieces in the First Posts review than in most other queues; this mostly just makes it worse. I have complete confidence in Geoff not to create any actual bugs while implementing this, but it's possible there might be a few more unexpected features.

• ???

Stack Exchange Quality Improvement Project

• Allow users to optionally filter out low-quality questions
• Feedback requested: New “recommended” homepage, phase 1
• What should the system be deleting automatically that it already isn't?
• Let's have an explicit triage system for questions from new users
• Breaking down question blocks - let's talk about rate limits

Answer 1

This is almost but not quite what I was imagining. Here's a flowchart of what I was picturing:

1. Every question gets evaluated for two things: Quality Score and User Heuristics. Quality Score is based on content, length, code ratio, etc. User Heuristics are things like "Is this a new user?" "Do they have a track record of posting questionable stuff?". We combine both of those things to give an immediate verdict: "Probably Good" or "Maybe Bad". The good stuff goes immediately to the homepage.
2. The "Maybe Bad" stuff now needs a human to look more closely at it. This is the Triage Queue. This queue focuses on a quick triage: is this question okay, is it salvageable, or is it hopeless? This queue has to be fast (<5 minute turnaround) so that the okay stuff goes on to the homepage very quickly.
3. The "Could be Saved" stuff goes to a third place, which is what you're calling the "Welcome Wagon" and I'm calling the "Feedback & Improvement Queue". Here, the emphasis is on helping people. You don't come here to delete junk, you come here to salvage borderline questions and train new users. Your main actions here are Edit and Comment. Stuff either gets fixed (yay, it can go to the homepage), rejected (actually, we made a mistake) or it ages out (we left a helpful comment, but the user never showed up to fix it).

I think it's important to separate out the "Fix this post" from Triage. Fixing a post is orders of magnitude more difficult than a simple triage, and we need the triage to be fast.

Note that this also replaces the Low Quality queue: it's just one process now, since we're taking "This user is new around here" as a signal that a human should triage this post.

Answer 2

New system doesn't match people who need help with the people who want to give it

Preface: this answer is from the point of view of one veteran Stack Overflow user.

I've spent the past few weeks thinking about how we can do a better job of handling questions from new users (folks with no past history to speak of on our sites). This is challenging, because there are two goals which don't cleanly align:

...etc...

• Welcome wagon - help new users settle in, provide them with advice and assistance as they learn to use the site. It's nice if this happens reasonably quickly, but not critical.

It doesn't really seem to me that this new system is much better than the current First Posts queue in terms of helping newbies learn how to thrive on the site. More often than not, veteran users are going to be clicking on "Needs Improvement" rather than "Looks OK" on new posts (especially questions).

"Needs Improvement" Actions

Bold emphasis mine,

If this question Needs improvement, please edit it to improve.
If you can't, consider downvoting, flagging or closing - and leave a comment for the author to let them know what they should fix!

There are a few problematic things with the 2^nd line:

1. Reviewers are encouraged to downvote, flag, and close, and optionally leave a comment...instead of leaving a comment, and optionally downvoting, flagging, and closing. There are some veteran users who would rather use a softer hand towards new users, and give them benefit of the doubt, by engaging with them a little before scaring them away with close and downvotes:

   Other close reasons (e.g. unclear what you ask, too broad) should not be instant in my opinion, better leave a comment first asking for details or asking to improve quality, and only after a while vote to close, if the OP can't be bothered.

   So as written, the reviewer advice basically says to go right ahead and immediately fire off close and downvotes, even without a comment as to how the post can be improved.

   Which brings me to the next problem...

2. Many veteran users still won't leave a comment. For many reasons,

   • Comments take too much time to type.
   • Reviewers want to avoid getting into conflicts with the users they're close/downvoting.
   • Etc?

   This doesn't really improve the perceived friendliness and helpfulness of the site for new users, who may have just gotten off on the wrong foot.

What happens to posts in the 2^nd review phase

Bold emphasis mine,

If a flag, close or edit action is taken, the post may be dequeued immediately without waiting for further reviews. Voting may implicitly do this as well, if it causes a post to become ineligible.

Some veteran users just don't want to hand-hold. Probably a lot, actually. These are the users who will just be firing off close and downvotes, usually with no comment to the OP. This is no different from the way things currently are...and because these actions take the post of the queue, they no longer have anyone else reviewing them.

As I've already pointed out in the previous section, however, there are some veterans who are indeed willing to hand-hold newbies. If most veterans are going to be dequeuing posts that need improvement with close/downvotes and without comments to help the OP, then these other more welcoming users are less likely to be able to find these new users who could use their help.

Basically, the new system does not match up new users with veteran users who are willing to hand-hold them. The kind of veteran who will leave a comment like:

Welcome to Stack Overflow! I see that your post is a little unclear, could you please provide more explanation as to how you're trying to use your foo to bar the baz so that the shazbot makes it to the derp?

instead of just a bunch of close and downvotes with silence. Which I think you kind of already mentioned here,

There's really no extrinsic motivation for doing anything in the Welcome stage. I'm mostly fine with that; I think this is too complex, too personal for extrinsic motivators to have any positive effects. But there's a possibility that we'll see interaction with first-time users drop even further.

Answer 3

On the whole, there's something to be done about first posts review. This is something. But not the right thing.

Give me the opportunity to edit before I do anything else. Often newcomers don't know how to tag and format their post, and the post is unreadable as is. I want to be able to edit a post, and then decide whether to upvote it or to send it into the low quality process.

Note that for answers, the context (question, other answers, sometimes even the author) is often important to detect spam. It's also important to distinguish non-answers: what is a good answer to one question may on another question be a repeat answer, a comment, or so wildly off-topic that it should just be deleted.

Generally speaking, giving us a very limited set of possibilities before making a decision is crippling, and I don't see a decisive advantage of doing things that way. There would be better ways to fix the defects you see in the current system.

Reviews take too long - there's a 15-minute waiting period before posts even enter review.

So make posts enter review immediately, or at least after the 5-minute grace window (because it's really annoying to have untracked edits while reviewing a post and find that in the two minutes it's taken you to review it, maybe edit it, the author has completely transformed it).

Reviewers are not encouraged to take their time when welcoming new users, as tasks are quickly snatched up by others. A post might be dismissed as needing no action by one reviewer when another would wish to take the time to offer useful guidance.

So introduce a lock (with a time limit, maybe 5 minutes). The lock is especially important while someone might be editing, so in your proposal, the second phase should still be locked.

The outcome of review is insufficient to properly direct the majority of new posts: on the top Stack Exchange sites, nearly 60% of reviews end with "no action needed", and under 20% are voted on.

That's a problem with abuse of “no action needed”. Make it not count for badges.

Answer 4

You have some dequeue conditions that shouldn't happen.

As a foreword: I'm one of those welcoming veterans who'll be giving the new users guidance. I'm a regular user of RPG.SE, which has a lot of users who provide new-user guidance of various forms.

As I currently understand it, the Triage and Welcome interface are both in one queue for new posts (or one for new questions and one for new answers). A user takes an action on a post in the Triage interface, then can optionally do extra stuff in the welcome interface.

Triage side

Only the first 1 posts scoring 0 from users with less than 10 reputation should enter the queue (values in bold should be site configurable however). Whenever possible, they should enter the queue immediately upon being posted (without requiring a waiting period or queue synchronization) and be removed when they no longer meet these criteria.

By "dequeueing," it sounds like you mean it'll leave the New Posts queue altogether when:

• The user gets one upvote on an answer or two upvotes on a question (user reputation > 10), or in fact
• The post gets any votes at all (score != 0)

However, even if someone gets a couple of votes, they should still be given an opportunity for guidance. If they have a downvote or two, they probably especially need someone to say: "Hey, welcome to the site. Your answer could do with {specific improvement}. Do you think you could {suggestion}?"

(On RPG.SE, we regularly encounter new users making a poor answer that gets downvoted. A comment gently nudging them as to how they can improve it works wonders.)

I suggest these posts stay in there 'til they get X reviews as well as some kind of action taken, not solely until they get X votes or not-X score. (But also not only until they get X reviews either: as you point out, those X reviews might result in no action at all.)

Welcoming Review

If a flag, close or edit action is taken, the post may be dequeued immediately without waiting for further reviews. Voting may implicitly do this as well, if it causes a post to become ineligible.

Same as above. Keep these in the queue. The first guy might downvote, flag, edit, or vote to close, but the next person might be the one to leave the welcoming comment with some guidance.

The important thing is to leave this in the queue long enough for some of those people to get their eye on it.

Consider having a button that says something like: We don't need to do anything else here, the user's thoroughly welcomed. Hopefully, it won't be clicked a bunch of times before those welcoming veterans see the post.

Answer 5

Use of the word triage reflects an overwhelming sense that there are too many poor quality posts coming through and there are not enough qualified folks to doctor them all. What if we were able to back up from the crisis and consider ways to avoid the problem instead?

A search on "low quality posts" generates a plethora of discussions on how to handle and manage low quality posts. A search in regard to "first posts" indicates that most effort has been focused on how to remedy the problem after the post has been made. The present system of inducting new members to essentials/expectations of this very unique forum involves introductory pages and prompts that has evidently left us wanting. Having experienced the present system as a new user and participated in it as an editor, I think that there is a better way.

My participation has mostly on BH-SE; thus, I leave it to others to discern how applicable this might be for other sites.

I suggest that first posts be not only prompted, but guided step by step in such a way that all the SE essentials are expected of the user in order to progress to the next step. It is easy to read through a page of site standards, then get enveloped in the Q&A and forget/neglect the standards. Learning is much more effective when practiced immediately or acquired in the process. In this way, new users know they are interacting with an established system of expectations and standards and choose to either comply or they simply do not progress.

Low quality posts we frequently encounter on BH-SE that other sites may share in common:

Questions:

1. Off topic
   Thus a prompt would inform the user of the forums acceptable subject matter (perhaps this one prompt could be tailorable for each site or linkable to each sites defining elements)
2. duplicates
   Thus, for a first step one would type in the question then be prompted to scroll through existing questions to see if their question has been answered already, before their screen advances to where they can enter the body of their question.
3. failure to do research.
   The next step would inform the user that there is an expectation that some research has already been done on the matter, and would instruct the user to post any research they have done. If they do not post anything then there is no way to advance to the next step.

Answers:

1. Lack of a thesis
   Step one would prompt the user to type in their thesis.
2. Lack of main points/logic
   Step two would prompt the user to enter any main points and connect the dots logically.
3. Lack of credible sources
   Step three would prompt the user to enter support for each main point with credible sources. (If this is skipped there is no way to advance).
4. Failure to answer the question. This mostly occurs because people come in to share their information or they are responding/reacting to other answers rather than the question itself.
   Step four would prompt the user to go back an read the original question and see if they actually answered the question. They would check the box (this directly answers the question or not progress).

It may be that there are not enough shared reasons for low quality posts to make this work on a system level. I have observed however, that within the general format/structure that all SE sites share, there seems to be room for each to enter their own site specific prompts and bullets. Perhaps a step by step progression through first posts could be programmed into the whole system in such a way that the individual prompts for each step could be tailored by each site, or even so that each site could opt in or out of the procedure.

In any case, I recommend we consider ways to prevent the problem rather than ways to remedy it.

Answer 6

I think that this workflow would be more difficult for many common review cases. Currently, I look at a first post and see that it should be edited, so I click a button that says "edit". Under this proposal, I'd click a button that says "Improvement Needed" and then get to a page where I can click "edit".

Why not stick with something more like the current workflow? I click "edit" which implies "Improvement Needed". When done with the edit, I continue on in the "Improvement Needed" workflow. Similarly, if I flag for something other than a "Abuse" reason, I go into the "Improvement Needed" workflow. If I downvote, it asks me to choose between "Improvement Needed" and "Abuse". If I upvote, it asks me to choose between "Looks Good" and "Improvement Needed".

If I choose to click any of the three options instead, it goes into the guided workflow and encourages me to upvote a "Looks Good"; flag, edit, or comment on an "Improvement Needed"; flag, downvote, and/or comment on an "Abuse". This gives the most help to the people who are showing the least understanding of the queue while still leaving it easy for a power user to move directly to the next step.

This preserves the best part of the current queue, that I can dive write into working on the post's problems, while still offering additional guidance to those who want or need it.

Another issue is brought up in Cupcake's answer: the current queue offers no real incentive to comment. So add a new queue for things that are voted "Needs Improvement" without comment. In this queue, we offer a privileged comment where someone can post a review of a post. If needed, the length can be longer (not sure what) and the comment should be displayed more prominently than a regular comment.

People with sufficient reputation to process that queue should then be able to vote on these comments (both up and down). This shouldn't cost reputation. An upvoted comment should get review credit in this queue while a downvoted comment should not.

The advantages of this are that it encourages handholding while not requiring it for the regular queue. If someone tries to fake it by posting a mediocre comment, then others can downvote it and take away the benefits of commenting.