How can we make First Posts review actually useful?

Question

There's now a feature-request based on this discussion; see: Let's have an explicit triage system for questions from new users

---

I'm about fed up with the First Posts review queue.

Oh, it was a nice enough idea in theory: give folks a chance to welcome new users to the site, without the heavy constraints that folks complained about with Low Quality review.

The problem is, the review system was designed around the idea of guiding folks to accomplish specific goals. And there's no clear goal for First Posts. Without a goal, folks tended to just upvote everything, until we "fixed" that by adding a "do nothing" button and strict audits.

Great fix! Now most folks reviewing first posts don't do anything at all, and those that do find that trying to automatically audit an open-ended system is fraught with danger. It's the same problem we had with the 10K flag queue, with the addition of a bit of robot-slapping.

So yeah, this review queue is terrible and useless

But before throwing the baby out with the bathwater, I figured I'd best check with y'all to see if any baby even lurks in these murky depths.

• What value do you see in having a First Posts review?
• What outputs do you see as valuable from such a queue?

I'm thinking we could probably make this work if we narrowed the focus considerably; less "help new users in some mostly-unspecified way" more "help filter the wheat from the chaff":

• Separate queues for questions and answers, with answers being sanity-checked for spam and questions requiring one vote - up or down. Essentially answer voting would be left to those with a specific interest in a given question (as it should be) while question voting would be used as a sort of peer-review system to draw more attention to questions that merit it.

• Allow flagging, commenting, comment voting, etc... But don't count those as a review action (unless we decide that the goal of reviewing these is to dramatically increase the number of comments or something, in which case we should probably drop that & try jSobriety).

• Send more things into the queue faster, remove them more quickly if they're "reviewed" organically. Less waiting for a new post to get attention, but less duplicate reviewing when something is getting attention without the need for review.

...Thoughts?

Answer 1

I agree with you that this is terrible on the trilogy sites, but it works very well on smaller sites that have a strong community core.

Whatever your decision on this queue, can you keep it in place for smaller ones? It allows us to welcome new community members, and gives us a steer to edit, where for an established user we may just chastise them :-)

Answer 2

Smaller sites live and die by reputation. Reputation is gained by having people vote for you. In order to vote, people need 15 rep themselves.

For most new people, to get 15 rep, that means getting 3 question up-votes.

Sites where there are a lot of drive-by people (ask a question, then move on), those askers will never be able to upvote answers on their own question, and will have moved on before they can... but.... what I see as being the primary purpose of the first-post queue, is User Retention, and is accomplished in 4 ways:

1. brilliant first posts get upvoted, and more eyes on. A nice comment makes people feel good too.
2. decent first posts get edited so they they become brilliant, and get upvoted, and more eyes on.
3. crap first posts get commented on, with instructions oh how to resolve the issues, so that the post can get edited, and then later upvoted.
4. off-topic posts that cannot possibly be salvaged by editing, should be voted-to close (duplicates, Gimme tez code, whatever), but a comment suggesting why may be warranted too (if it assists with user retention - and trains the user to be a positive contributor in the future).
5. spam/offensive stuff gets flagged and removed, and we don't want to retain those users.

The goal of the first post queue is to ensure that questions that succeed in the review are good enough for 3 upvotes, and answers are good for 2.

Then, the first poster can vote, and make everyone feel good, and they also feel good because they earned something.

User retained!

So, as a reviewer, your objective is to mark each post as great, or to make it great if it was not great to start with.

If you don't think you can find a way to make it worth 3 votes, then skip it, and leave it for someone who can.

Answer 3

It's critical to scrutinize first posts. They're by far the most likely source of spam, and the most likely to have other problems.

You say

Allow flagging, commenting, comment voting, etc... But don't count those as a review action

But I actually do think those should count as a review action. The point of the review is to give new posts more eyeballs, and encourage any actions that are necessary, whatever that may be.

Yeah, it's vague. That's kinda the point. Unlike all the other reviews (except Late Answers), the posts aren't the trigger for entering review - the user is. We have no idea what will be needed.

In fact, thinking about Late Answers, it should just be merged with First Posts, and renamed New Users. If you want to review just one type of post, that would be available through filters. The purpose of all three – First Questions, First Answers, Late Answers – is the same: increase views. "Reviewing" is just the easiest and most accountable way to do that. Any action one would take seeing the post elsewhere should be valid and allowed here.

In that vein, enough organic views should definitely remove it from the queue, since the purpose has been fulfilled.

Answer 4

The first posts queue is a very helpful tool to guide new users. It should be expanded, not removed.

First posts (or early posts, not necessarily just the very first post) warrant more attention for:

• guidance in comments, regarding how to improve a question which isn't suitable as posted, what extra information should be provided, what could improve the quality of an answer, etc.
• spam triage
• edits, as the user probably often hasn't mastered formatting — this is especially important on sites with MathJax
• making extra sure that helpful first posts are rewarded by an upvote (as rofl notes, it's especially important to get new contributors to the stage where they can upvote)

The first posts queue is all about guiding new users. If it's misused 50% of the time, this means that 50% of users are getting this extra chance of guidance. Removing the queue just because it isn't as effective as it should be would be silly.

So yeah, this review queue is terrible and useless

Wrong. Dead wrong.

I'm thinking we could probably make this work if we narrowed the focus considerably; less "help new users in some mostly-unspecified way" more "help filter the wheat from the chaff"

This seems completely backwards. Helping new users in some mostly-unspecified way is very much what the first posts queue is about! Yes, triage is also involved — there are users (especially spammers) that we don't want to help. But primarily this is about guidance, about helping.

Before the first posts queue in its present form, I used to pay special attention to posts on the front page marked “modified X time ago username 1”. I deliberately went to these posts to provide triage and guidance. This is obviously very approximate: it doesn't catch cases when someone else modifed another post on that page afterwards, sometimes brings me to already-deleted posts, etc. Essentially, what I wanted a list of first (or early) posts requiring special attention, preferably with some system so as to distribute said attention evenly. Which is the exactly what the first posts queue does.

Ideally, a new user's posts should go into the first posts queue as long as the user hasn't been determined to be good or bad. I'd like to see something like this (I haven't thought about the numbers much):

• A new account with 1 rep is considered new until further notice. All posts by a new account go through the first posts queue.
• When a new account reaches 2 posts with a positive score, it is presumed good; posts no longer enter a queue based on the poster.
• When a new account reaches 2 posts with a negative score, it is presumed bad; posts enter the low quality queue instead of the first posts queue.

Posts from new users should enter the first posts queue as soon as technically possible. We want to do triage and guidance as early as possible, ideally while the user hasn't closed their browser yet.

Each post should be seen by multiple pairs of eyes. I agree with the general idea that posts should exit the queue based on actions taken, though it's not as simple as having had a vote up or down. Some posts are just middling. I think the rules for number of reviews should be relatively complex, perhaps a sort of point system (actions grant points whether taken from the review queue or not):

• the first upvote or downvote gives 3 points
• subsequent votes give 1 point, but the second vote gives 0 point if it contradicts the first one
• a close vote or any flag gives 1 point
• an edit gives 2 points
• a comment gives 2 points
• a “no action needed” review gives 1 point
• a post stays in review until it's acquired at least 4 points, with always 1 mandatory review

I haven't thought much about all possible combinations of actions, this is just an idea of what I think the system should be. The idea is to ensure that feedback is provided to the user.

I don't see the point in separating questions and answers. This queue is about providing tailored guidance: you never know whether the next post will be good or bad, so why would it matter whether it's a question or an answer?

To combat the serial reviewer problem, we need to combat serial reviewing in itself. This isn't a problem of the first posts queue specifically. Making “no action needed” not count towards badges and rankings might help, though I don't like it much — it penalizes conscientious reviewers because of some bad apples. I had proposed minimum quotas of each action to get badges. The objection was that serial reviewers would turn to acting randomly and undoing their damage would become harder — but I'm not convinced: are we really undoing the damage of serial reviewers now? Alternatively, we can punish serial reviewers: there are now tools to suspend users from reviewing, and we should reset offenders' progress towards badges.

Audits aren't doing a good job of revealing serial reviewers because too many of them are litigious and the ones that aren't are trivial to avoid without really engaging one's brain. However, patterns of behavior can be used to good effect — a user who spends 5s on average and always selects the same action clearly isn't doing a good job. A minimum time per review (with some averaging going on because there is the occasional short review) could be enforced or verified (“you cannot submit more than 10 reviews in 5 minutes, come back in 1 hour”); there are ways to make reviewing more annoying for serial reviewers while not impacting conscientious reviewers, and we aren't doing that at all at the moment.

Answer 5

What value do you see in having a First Posts review?

On smaller sites, I have primarily used it to find good posts and add a "Great first answer and welcome to the site!" type of post.

Some of the smaller sites tend to do this more than Stack Overflow I've noticed. I think I've seen comments like that on Workplace, TeX, and Parenting.

Answer 6

I see the purpose of the first posts review list in "Help them learn to use the site.".

It's a fact that most first posters do not understand the workings of this site when they post their first post.

The idea is then that there is always at least someone who can check if all the points in How do I ask a good question? are followed to a reasonable extent. If this is done in a timely manner then it is a valuable for all of us.

Of course the whole system would also work without a review list. Everyone who searches for newest questions does probably the very same things.

But for me it feels like a step in the right direction. Newbies don't know the rules, so you have to help them (by improving or by flagging for deletion) as early as possible. What's wrong with this logic?

Answer 7

As a fairly active reviewer on SuperUser lately, and a really active one on ServerFault historically, I'm going to throw in my two cents' worth, which is that the First Posts queue is a valuable one, as-is, and shouldn't be tossed out. It could probably be improved (as can just about everything), but even without improvement, it serves some vaulable functions we'd miss if it was gone.

There are plenty of people who ignore it, or abuse it, but it really does allow higher-rep users of the sites to welcome and/or correct new users. Yes, most of the posts are mediocre, a good portion are utter crap, and only a few are really worth anything at all. And, as a result of most of the posts being "meh", sure, most of the actions taken will be... nothing. I think that's just fine, though.

Really, I think that focusing on the fact that the majority of new posts are really mediocre misses the point. Last I checked (and granted, it's been a while), StackExchange was aiming for most of its traffic to come from Google and other search engines. We have a very high number of drive-by users, who'll post something once, and never be heard from again. The point of the First Posts queue isn't those people who won't contribute, the point is to help people who do want to contribute. (Though throwing more eyeballs on the drive-by posts seems like a good idea to me.)

By bringing attention to new users as a whole, the First Posts queue lets the members of the community who want to guide potential new users of the site in how to use it, do so. The fact that the majority of site visitors or even site posters won't ever become members is irrelevant - the queue is there for those who do become members, to make it easier for them to contribute and get guidance right off the start... and frankly, I think the benefit of shining a spotlight on a class of post that's most likely to be spam or utter garbage, so it can be dealt with swiftly is a good thing™ as well.

When you get right down to it, StackExchange is a collection of user-generated content, which is its greatest strength on Google, so anything that helps improve that content (or the users generating it), seems like something we should be very wary of tossing out, even if it is difficult to do properly, and irritating at times. Having the First Posts queue, even as-is, is certainly better than nothing, and complaints about failed audits are probably a small price to pay for what the queue does for us.

Suggestion for improvement:

My suggestion for improving the First Posts queue would actually be to expand the queue. Instead of just first posts, it seems to me like it would be handy to throw a user's first several posts in the queue. After all, the queue text does say it's about helping new users learn to use the site... by restricting it to their first post only, it doesn't really let high rep users do that - it's not easy to provide feedback, or change based on a single data point, unless the data point is exceptional in some way. By expanding it encompass more posts from new users, it would allow more opportunity for feedback to those who decide to become members, or try the sites out, and I think it's hard to argue that this wouldn't be a good thing.

Answer 8

Why punish genuine reviewers for having robots in their midst by removing / limiting their tools? It's not their fault. Punish those pesky robots instead! Surely there are ways to statistically determine which members are compulsively pushing always the same buttons, no? So let's flag those, and auto-ban from further reviewing until a mod looks into it, and decides if the flag should stand or not. Frankly, I think that the First Post reviews do work on sites I'm most active, albeit those are smaller than the SO / SF / SU trilogy. If something needs changing, then please provide more feedback on actions that should be used less frequently, like comments.

Answer 9

Even for the trilogy sites, I still don't see what negative effect the queue has on the site.

What's bad about it, apart from the fact that trying to comment on an audit fails?

Whilst 50% of the reviews are NAN's...:

• 4,000 reviews are improving the quality of the site through edits
• 10,000 reviews are either flags or VtC's on bad content
• 15,000 of them are leaving a signal to the user about their first-post; be it an upvote, downvote or comment.

Given the recent talks about improving the quality of the site, removing one of the tools which audits quality seems an awful idea.

The review queue is almost always empty. People are happy to review through the queue. I really don't see the reason to remove it.

---

If you want a specific goals for the first post queue, I'd say:

1. To ensure new users get feedback about their new question.

   My gripe with the First Post queue is that it allows people to flag/ downvote/ VtC questions without giving feedback to the OP that there was anything wrong with the question.

2. To target posts which are highly likely to be sub-par

   My other gripe with the first post queue is that it only works for a users first post. Expand the queue to:

   1. A users first n posts
   2. A users next n posts after a post of theirs was deleted
   3. A users next n posts after a question of theirs was closed
   4. A users next n posts after a post of theirs received a negative (or < y) score.

The beneficial outputs for me is that:

1. New users get more feedback (and hopefully a more friendly experience) with their first actions on a site. People who purposefully go to a review queue are (hopefully) there to help and assist people. People who go to a question organically are there to answer them or get answers. They're less likely to give feedback to a user, short of casting a anonymous close/ downvote.

2. We get to remove crap content quicker

Answer 10

First Posts review queue should be about the User, not the Post.

At first glance, that may seem contrary to what SE stands for, being about content, but not the user. But the very nature of the First Posts queue is one that targets a User. It's not looking for "potential low quality posts", it is looking for "posts by new users".

I know SE doesn't want to be a social media site, however don't lose sight of the fact these sites are built and run by a community of volunteers, and to keep the site healthy we need to keep recruiting new members to replace older members that move on.

The First Posts queue is the perfect place to do that.

So I think we should expand this queue and make it clear that it is a queue about the User, and not the post.

Some ideas I had for that

• Show the new user the result of their review. If it was "Post looks fine", then show them that. If it was "Something is wrong", show them that too along with links to appropriate resources on how to ask/answer.

• Textual feedback should be required here. I don't care if its canned responses, or something as simple as "Post looks fine", but we need some kind of textual feedback to give the user that explains what the result of their review was.

• Reviewers should not feel the need to welcome new users using this queue. It should be right there in the review text for the new user

  Welcome to Something.SE! Because you are a new user, your post has been reviewed by other community members, and the following feedback was left for you:

• It should be clear to regular SE users that this is not a queue you go to when you want to evaluate posts. It is a queue you go to when you want to help out new users who are starting to contribute to the site for the first time. Reviewing posts is a part of that duty, however it is not the primary purpose.

• Change the text on the queue to something that more accurately describes what you want users to do in the queue, like "Help guide these new users".

  Right now it says "Review first posts from new users", so its no wonder that many people think it is just a queue of posts that have a higher chance of being "bad", and don't understand that the original purpose of the queue is to welcome and help new users (I didn't know that until I read the meta post you linked in your question).

This post makes me think of a question I asked a short while ago on Meta.SO, where I asked for suggestions on how we could promote new user guidance and/or mentorship in how to use the SE sites, in order to maintain a healthy user base.

I think updating the First Posts queue to make it focused on giving the new user feedback would be a great step in that direction.

Answer 11

Having been poking at the 10k view of the first posts queue for a bit, there are two goals... to give the good posts some attention and to lynch provide prompt feedback for poor posts.

Yes, there's about a 1 in 5 chance that the post you're looking at will be utter crap, or at very least something you'll want to down-vote. But it is just as important to provide some sort of feedback to the new authors whose posts aren't terrible - whether that's an up-vote, or a comment, or even a small edit that puts a bit more of a shine on their already-useful contribution.

(from New Review Queue: First Posts)

The thing is that these actions are a tug of war to an extent - pulling the review in different directions with a big hazy gray area in the middle.

One idea would be to break this into two queues with clearly defined aspects and triggers for inserting into the queue or removal from the queue.

There's the new low quality queue. Posts that need a firm push that is triggered either by a close vote or a down vote (or more down votes than up votes?). For this, the available actions are only corrective - similar to the existing low quality queue. You can flag, close, comment and edit. If the post gets these organically, the post is removed (don't overwhelm a new user with half a dozen edits or comments from people trying to complete a review).

As I write this, this sounds quite a bit like the existing low quality queue... but that doesn't see too much action from new posts. But here's the kicker... you need to do two actions to complete the review. Its a "you are on the path for writing bad questions, you need some corrective action and we will show you how to go down that path.

Then there's the new user encouragement queue. As mentioned in What happens to new users?, if we encourage users with some interaction they are more likely to come back. This queue is only for posts with 0 or 1 votes and no comments. The possible actions are: vote, comment, edit, flag/close (because its still possible it needs that interaction). If a question organically receives this interaction outside of the queue, the review is removed. As with my idea for the new low quality queue, this would require two actions to complete a review.

If the queue is about guiding a user along the path to being a good SE user, the two queues represent a minor correction and a major correction to what is needed in a post.

Furthermore, the multiple necessary actions for completing a review would serve as a "this queue takes work, its not a simple 'no action needed' or 'up vote' and do the next review."

Answer 12

On sites of a certain size (too big to read every post without the risk of missing one or two, but not so big as to have the problems you mentioned), the first post queue is invaluable for finding promising new participants. Some communities have developed organic welcoming committees who patrol that queue. So given the choice between eliminating or fixing First Posts, I'd like to see it be a site optional queue.

But if we really want to fix the queue, we need to figure out why people misuse it. The cynical side of me says that people who actively are annoyed by new users use the queue to punish them. More likely most reviewers just don't care. Likely they visit the queue in order to get a badge or see get smaller. If that's the case, we could try disconnecting the FP queue from the numbers that might be driving no action reviewers.

I was going to suggest some incentives for being helpful, but every scheme I could think of amounts to giving out lucite plaques. Helpful people like being helpful for it's own sake.

Answer 13

One useful thing First Posts accomplishes, in smaller tags, is it gets really really bad things closed or removed very quickly. Due to the CV queue overload, it's hard to get questions closed in tags with very few close-eligible active followers; so often questions just never close.

It also introduces some viewpoints from higher rep users not following these smaller tags, which can be a double edged sword (sometimes you have things closed/edited/commented on that shouldn't really be because the higher rep users don't really know the tag's specifics); but overall it's helpful getting additional points of view to tags that might only have one or two higher rep users commonly helping herd the sheep.

So, I would try to keep it in some fashion - perhaps keep it as is, but remove the audits. I don't feel like audits are needed for FP; the point is for people to look at questions, which they could do anyway just by looking at questions on the main page. Remove the badges, if that's the problem with the queue. I think enough people care about the site that FP would be read no matter whether a badge exists or not.

Answer 14

What value do you see in having a First Posts review?

1. For communities about programming it is often code formatting. Because some new members often seem to don't know about preview of posts, which makes their posts almost unreadable. Sometimes nobody even don't know how OP supposed his post to look like.

2. Sometimes new members don't care about punctuation, capital letters, because they think that it is no rules as in Facebook or Twitter (however sometimes I meet very good-looking and formatted questions from 1 rep users). So their posts look like:

hey guys something is wrong with my code i dont know i am new please help this is my code:

{some bad formatted code}

whats wrong please help

---

What outputs do you see as valuable from such a queue?

Less bad formatted questions and unclear questions from new users (I understand that it is low-qua queue for some of such posts, but anyway...).

Answer 15

I've always thought that First Posts (and possibly Late Answers) should be reviewed by more than one person each. Somewhere between three and five people should be looking at each of them to provide feedback. This provides more of an opportunity for the user to actually get some feedback rather than No Action Needed.

Another option is to remove the badges from this review queue. But that opens up the question of whether anyone will review posts without the prospect of gaining a shiny gold badge in three or four years.

Answer 16

I would consider it a valuable output if review presented a reasonably accurate estimate / prediction of how a post is going to be received in a future.

To keep it practically scoped, I would limit "future" to one month after review completion (maybe 2-3 months if recently proposed longer term auto-deletion gets implemented).

In this case, usefulness of review could be analyzed by checking post status (positive, negative, neutral, deleted, edited) a month after review and comparing it against review outcome.

• ^{Somewhat more ambitious goal of review could be to estimate how a poster is going to be received, analysis for this could then involve "aggregate" of first, 2nd, 3rd etc posts and comparing it against review outcome. For the sake of simplicity I'll keep that out of my answer.}

Results of retrospective analysis can be further used to adjust reviewers behavior if it deviates too much from observed data. System can plant more "known bad" audits into the queue if reviewers tend to be too optimistic and more "known good" ones for the opposite outcome.

---

Note how similar approach can be used to analyze and adjust behavior of individual reviewers.

System can check and analyze later state of the posts they reviewed and present them an individually tuned desired mix of audits. It can even report particularly gross individual deviations to moderators for further manual analysis.

Difference from "aggregate" case is that for "individual" analysis and corrections to be timely, a shorter (and thus less accurate) retrospective time frame is probably needed, a week or two instead of a month.

---

Speaking of review audits, their primary goal is said to help hone moderation skills and per my observations of FP queue, audits there are too infrequent to accomplish that goal. You simply don't show folks enough "known good" / "known bad" examples to learn from.

• ^{Note by the way that more "known bad" audits would help decrease the problem you seem to worry about, one about folks who "don't do anything at all", simply because such audits fail at No Action Needed.}

Teaching reviewers to appropriately use Skip action would also help getting better quality, more relevant reviews.

Answer 17

I'm speaking as a Stack Overflow moderator, and this is based on my experience there, but I see value in splitting answers out of the First Posts queue and rolling reviews of them into the Late Answers queue in a new form. We review answers and questions differently, and lumping both into the First Posts queue doesn't work too well on SO.

The primary purpose for reviewing both Late Answers and answers within the First Posts queue is to find spam and non-answers. Very rarely do we see spam in questions, and that's usually caught because questions appear on the front page of the site. First Posts review of answers can catch spam or non-answers that appear on newer questions (where Late Answers doesn't trigger).

Reviews of questions are different, in that you're typically looking either for inappropriate questions to close, or for new posters who could benefit from a little guidance. The latter is very important on smaller sites (as others have stated), but doesn't scale to the Trilogy. People leaving answers generally don't need such guidance, and they are more likely to be positively received.

I know that when looking through a suspect reviewer's recent history, I skip question reviews in First Posts and only focus on if they approved bad answers. Question reviews aren't all that meaningful on Stack Overflow, but the answer reviews in First Posts are.

I'd be all for a Late and New User Answers review queue, which would be very useful on Stack Overflow, with a separate Questions By New Users review queue. That queue could be enabled on the smaller sites, and potentially disabled on SO, if we thought it didn't bring any benefits.

Answer 18

Speaking from experience with reviews on GIS.SE, I'd say the process is flawed:

The review menu item is always live, even when there is nothing for me to review, which is most of the time. Thus I'm reluctant to waste my time looking.

The few times there is something there to review, I'm unsure what I'm supposed to do or how to do it -- especially since I already have powers to comment, vote or edit outside of this "review" process (and I do exercise those powers).

If first posts are withheld until they pass "review" (is that the case; I don't know?) then the process isn't working because I see many low quality posts and non-upvoted good posts outside of "review".

[Forgive me if this bit is off topic.] The edit review process is also flawed because I'd sometimes like to see the full Q&A with comments before I judge a suggested edit but I cannot do that inside the "review".

Answer 19

I did not read through every single answer already given here, so i apologize if i'm duplicating something already said.

I think that First Posts shouldn't be limited to just a user's first post. Until they show that they know how to use the site, they will be sent back to the queue. Once the user has posted something with a positive score, we'll leave them out of this queue. But if they post a question that gets -1, their next question will also go into review. And if that one is negative, their next post also, until they get something positively scored.