This is a follow-up feature-request based on: How can we make First Posts review actually useful? - big thanks to everyone who participated there, I've tried to incorporate the feedback into the design below.
I've spent the past few weeks thinking about how we can do a better job of handling questions from new users (folks with no past history to speak of on our sites). This is challenging, because there are two goals that don't cleanly align:
Triage new posts - eliminate spam and other forms of abuse, queue up low-quality posts for further review, mark good posts as worthy of further attention. Ideally, this should happen as quickly as possible after the post is created, so that both the author and the rest of the site can benefit from the actions that follow.
Welcome wagon - help new users settle in, provide them with advice and assistance as they learn to use the site. It's nice if this happens reasonably quickly, but not critical.
Right now, the system fails to fully achieve either of these goals:
- Reviews take too long - there's a 15-minute waiting period before posts even enter review.
- Reviewers are not encouraged to take their time when welcoming new users, as tasks are quickly snatched up by others. A post might be dismissed as needing no action by one reviewer when another would wish to take the time to offer useful guidance.
- The outcome of review is insufficient to properly direct the majority of new posts: on the top Stack Exchange sites, nearly 60% of reviews end with "no action needed", and under 20% are voted on.
Therefore, I recommend the existing system be scrapped, and replaced with the one described below. Note that for brevity, I'm focused entirely on questions in this post - a corresponding review queue for answers should also be created, with more or less identical functionality.
Two-level review: mandatory triage, optional welcome
The strategy I'm aiming for here is to first collect as much signal as possible in the shortest amount of time, and then allow conscientious reviewers an opportunity to do a bit of hand-holding when and where they feel it is useful to do so.
Rather than trying to make the normal UI work for these two different purposes, I'm just using two different, consecutive interfaces: first triage, then welcome, with guidance and restrictions appropriate for each.
Note: all screenshots are fake. So is the reviewer guidance in them. I'll write less boring text before this is done - suggestions welcome!
This should be very similar to the existing Low Quality review: only three possible actions (choosing "Abuse" should present the flagging dialogue with three additional options: spam, offensive and 'other'). Unlike Low Quality, there's no direct way to recommend closure or deletion here - the goal is merely to capture enough information to decide whether or not that should be the next step. Basic functional requirements:
- Only the first 1 post(s) scoring 0 from users with less than 10 reputation should enter the queue (values in bold should be site configurable, however). Whenever possible, they should enter the queue immediately upon being posted (without requiring a waiting period or queue synchronization) and be removed when they no longer meet these criteria.
- Each review task should be shown to only one reviewer at a time, no matter how many reviews are ultimately required to complete the task. So far as possible, the system should avoid encouraging reviewers to "race" to get credit - reviews should be fast enough without this.
- Posts should be removed from the queue after receiving FP_REVIEWS_REQUIRED reviews in a single category ("Looks OK", "Needs improvement", "Abuse"), where FP_REVIEWS_REQUIRED is initially 3 on Stack Overflow and 2 everywhere else. (As noted above, posts that are no longer eligible per #1 should be removed as well.)
- Posts that attract FP_REVIEWS_REQUIRED "Looks OK" reviews should be bumped to the front page by Community.
- Posts that attract FP_REVIEWS_REQUIRED "Needs improvement" reviews should enter the Low Quality queue.
All auditing should take place at this stage, with a mixture of known-good, known-bad and known-spam posts used to evaluate the reviewer's ability to correctly evaluate the essential nature of a given question.
Reviewers who choose "Abuse" will be required to choose a flag type and will then progress on to the next review. Reviewers who choose either "Looks OK" or "Needs improvement" will progress to...
At this point, further action on the part of the reviewer is optional. For those that wish to aid the asker in some fashion, guidance will be provided based on their triage review. For "Looks OK",
...and for "Needs improvement",
All the usual post tools become available at this stage, just as in the current First Posts review. All are optional - the reviewer can click "I'm Done" immediately without further action. Certain actions taken as part of the "welcome" review may remove the need for further triage reviews, however.
Basic functional requirements:
- Action taken should be recorded along with the review (but not displayed publicly - as is done now).
- If a flag, close or edit action is taken, the post may be dequeued immediately without waiting for further reviews. Voting may implicitly do this as well, if it causes a post to become ineligible.
There may not be enough feedback for reviewers in the Triage stage. gnat suggests that more audits may help here - perhaps we should increase the frequency of audits for new reviewers?
There's really no extrinsic motivation for doing anything in the Welcome stage. I'm mostly fine with that; I think this is too complex, too personal for extrinsic motivators to have any positive effects. But there's a possibility that we'll see interaction with first-time users drop even further.
It's kinda complicated. There were already more moving pieces in the First Posts review than in most other queues; this mostly just makes it worse. I have complete confidence in Geoff not to create any actual bugs while implementing this, but it's possible there might be a few more unexpected features.
Stack Exchange Quality Improvement Project
- Allow users to optionally filter out low-quality questions
- Feedback requested: New “recommended” homepage, phase 1
- What should the system be deleting automatically that it already isn't?
- Let's have an explicit triage system for questions from new users
- Breaking down question blocks - let's talk about rate limits