115

There's now a feature-request based on this discussion; see: Let's have an explicit triage system for questions from new users


I'm about fed up with the First Posts review queue.

Oh, it was a nice enough idea in theory: give folks a chance to welcome new users to the site, without the heavy constraints that folks complained about with Low Quality review.

The problem is, the review system was designed around the idea of guiding folks to accomplish specific goals. And there's no clear goal for First Posts. Without a goal, folks tended to just upvote everything, until we "fixed" that by adding a "do nothing" button and strict audits.

Great fix! Now most folks reviewing first posts don't do anything at all, and those that do find that trying to automatically audit an open-ended system is fraught with danger. It's the same problem we had with the 10K flag queue, with the addition of a bit of robot-slapping.

So yeah, this review queue is terrible and useless

But before throwing the baby out with the bathwater, I figured I'd best check with y'all to see if any baby even lurks in these murky depths.

  • What value do you see in having a First Posts review?
  • What outputs do you see as valuable from such a queue?

I'm thinking we could probably make this work if we narrowed the focus considerably; less "help new users in some mostly-unspecified way" more "help filter the wheat from the chaff":

  • Separate queues for questions and answers, with answers being sanity-checked for spam and questions requiring one vote - up or down. Essentially answer voting would be left to those with a specific interest in a given question (as it should be) while question voting would be used as a sort of peer-review system to draw more attention to questions that merit it.

  • Allow flagging, commenting, comment voting, etc... But don't count those as a review action (unless we decide that the goal of reviewing these is to dramatically increase the number of comments or something, in which case we should probably drop that & try jSobriety).

  • Send more things into the queue faster, remove them more quickly if they're "reviewed" organically. Less waiting for a new post to get attention, but less duplicate reviewing when something is getting attention without the need for review.

...Thoughts?

16
  • 11
    It's useful if you're not a robo-reviewer. I personally think that my comments from that queue could be useful to the person. Jun 17, 2014 at 15:48
  • I think the usefulness of a first post review boils down to the question if the people posting "actionworthy" first questions are willing to learn from those actions we take.
    – PlasmaHH
    Jun 17, 2014 at 15:49
  • 12
    I like the idea of separate question and answer queues. As much as it might introduce fatigue, ("omigosh, not ANOTHER queue!",) it would reduce the necessary mental effort to process what is going on. (A very short answer could easily require different treatment from a very short question, for instance.) That being said, forced voting in the question queue is... questionable. (I'm sorry. Just kidding. I'm not really.) I would probably avoid the question queue were that implemented. Jun 17, 2014 at 15:50
  • 42
    Personally I was not actually aware that the first posts queue was to welcome users to the site. Although I do often post comments when reviewing that queue I thought it was in primarily in place because new users are much more likely to make posts which need either major editing, flagging or downvoting to note that they are "not worth the time".
    – Amicable
    Jun 17, 2014 at 16:00
  • 2
    I think this review queue is very useful to catch spam and low-quality posts. Jun 17, 2014 at 16:59
  • 11
    For Stackoverflow: I think the first review queue would PROBABLY be actually useful if first questions would actually require that at least one reviewer has hit the "do nothing" button. otherwise it shouldn't show up on the site at all! (and if there are more downvotes than upvotes from the reviewers it shouldn't show up on the site neither, of course.) Reviewing questions after they are already visible on the front page is totally useless, in my opinion.
    – Michael
    Jun 17, 2014 at 19:10
  • Could we just remove all badges related to reviewing in the First Post review queue (and maybe the other ones as well)? We'd eliminate robo-reviewers and have less of a need for review auditing, while still keeping the benefits it brings.
    – Savanna
    Jun 17, 2014 at 19:22
  • Robo-reviewers aren't currently that much of a problem, @Savanna. Fixing that is actually pretty easy; making the queue do something useful is harder.
    – Shog9
    Jun 17, 2014 at 19:24
  • 19
    "Now most folks reviewing first posts don't do anything at all" - One third of those reviews downvoted, flagged, and/or commented on the post. That sounds like a success to me.
    – BlueRaja
    Jun 17, 2014 at 21:59
  • 6
    @Shog9 So catching most of the spam/bad questions as they come in isn't useful? Making sure good Q's and A's get good feedback isn't useful? I thought the first post queue was in place so we could check all the new user posts as they come in, flag, edit, comment, close and vote as needed. Without the queue the amount of that that gets done will go down drastically will it not? Before we decide to pull it I'd really like to see some stats on that. I like your third bullet though. If a question is getting lots of attention already, no need for more people to review it (esp. on bigger sites).
    – ɥʇǝS
    Jun 17, 2014 at 23:52
  • After following some of the links now, I see some stats. Are those recent?
    – ɥʇǝS
    Jun 17, 2014 at 23:53
  • 1
    As side-reference, there are some unresolved issues with the First Post review queue... For what it's worth, a vote-requirement would not really work on TeX - LaTeX, as we are a bunch of up-voting folk; as such, I think it will point to robo-reviewing by up-voting.
    – Werner
    Jun 18, 2014 at 6:05
  • 1
    Btw. my impression with the first post review queue is that it is empty most of the time but also that the posts are already some minutes old if I get some. Is there any delay in the system so that they are not shown right away? Time is of the essence for such reviews I would say. Jun 18, 2014 at 12:19
  • 2
    @Shog any updates? Something going to happen soon? Jun 18, 2014 at 18:38
  • 1

19 Answers 19

76

I agree with you that this is terrible on the trilogy sites, but it works very well on smaller sites that have a strong community core.

Whatever your decision on this queue, can you keep it in place for smaller ones? It allows us to welcome new community members, and gives us a steer to edit, where for an established user we may just chastise them :-)

5
  • 27
    It works reasonably when people approach the queue with the intent of helping users. It works horribly and completely misses its goal when people are not trying to help ( i.stack.imgur.com/IxqNK.png ).
    – user213963
    Jun 17, 2014 at 15:53
  • 3
    I agree with this. The queue seems to be useful on Physics, after its introduction we had a nice uptick in bad posts getting caught. Jun 17, 2014 at 16:28
  • 6
    @MichaelT That is exactly the problem with larger sites. Users who use the queue usually don't have the intention of welcoming the user and helping them ask their questions "the right way". Instead, they're more interested in clearing the queue, or downvoting/closing bad stuff. We need more emphasis on the purpose of the First Posts queue. Perhaps we should make it so the reviewee always sees a result from the reviewer, and improve our guidelines to reviewers for this queue.
    – Rachel
    Jun 17, 2014 at 20:32
  • 1
    @Rachel it is a problem on smaller sites too. And there, individual poor reviewers can have a much larger impact on the impact of even larger a percentage of the site's new users. The screen shot above is from one person's reviews back on May 31 on P.SE and you can look at them easily there too and draw your own conclusions of motivation. I don't necessarily agree with your initial analysis of motivation. I don't believe that showing a 'no action needed' or 'you got an up vote (or down vote)' would help the user any more than they are currently getting out of the review queue.
    – user213963
    Jun 18, 2014 at 0:39
  • 4
    I think we can rig this so that it works better on larger sites and smaller ones, Rory. A good parallel might be the suggested edits queue, which has a somewhat higher threshold on SO than elsewhere but otherwise behaves largely the same.
    – Shog9
    Jun 19, 2014 at 23:58
68

Smaller sites live and die by reputation. Reputation is gained by having people vote for you. In order to vote, people need 15 rep themselves.

For most new people, to get 15 rep, that means getting 3 question up-votes.

Sites where there are a lot of drive-by people (ask a question, then move on), those askers will never be able to upvote answers on their own question, and will have moved on before they can... but.... what I see as being the primary purpose of the first-post queue, is User Retention, and is accomplished in 4 ways:

  1. brilliant first posts get upvoted, and more eyes on. A nice comment makes people feel good too.
  2. decent first posts get edited so they they become brilliant, and get upvoted, and more eyes on.
  3. crap first posts get commented on, with instructions oh how to resolve the issues, so that the post can get edited, and then later upvoted.
  4. off-topic posts that cannot possibly be salvaged by editing, should be voted-to close (duplicates, Gimme tez code, whatever), but a comment suggesting why may be warranted too (if it assists with user retention - and trains the user to be a positive contributor in the future).
  5. spam/offensive stuff gets flagged and removed, and we don't want to retain those users.

The goal of the first post queue is to ensure that questions that succeed in the review are good enough for 3 upvotes, and answers are good for 2.

Then, the first poster can vote, and make everyone feel good, and they also feel good because they earned something.

User retained!

So, as a reviewer, your objective is to mark each post as great, or to make it great if it was not great to start with.

If you don't think you can find a way to make it worth 3 votes, then skip it, and leave it for someone who can.

9
  • What if you can't make it great?
    – user213963
    Jun 17, 2014 at 16:39
  • 17
    @MichaelT then skip it, and let someone at it who can
    – rolfl
    Jun 17, 2014 at 16:42
  • 1
    There are questions that are horribly off topic, cut and paste homework assignments, under specified problems, and other such 'gems'. While your job is indeed to make it great that often can only be done by the OP (and up voting these questions only makes it harder to take those corrective actions).
    – user213963
    Jun 17, 2014 at 16:54
  • 12
    @MichaelT _ I am not suggesting you blindly upvote questions that are bad. I am suggesting you make them good, and then upvote. If there is no way to make it good with the information at hand, then leave a comment with suggestions as to what should be done (and do not upvote) (my option 3 above).
    – rolfl
    Jun 17, 2014 at 16:56
  • 2
    So, when we vote to close?
    – Braiam
    Jun 17, 2014 at 17:12
  • 1
    Or (quite importantly) vote to close as a dup? Or flag as spam?
    – user213963
    Jun 17, 2014 at 17:26
  • 15
    When you're supposed to, obviously. I don't see how integrating this with existing VTC best practises is hard: make the internet better, skip it if you can't. Making the internet better can be polishing a gem or trashing a turd. Do we really need a Grand Unified Theory to integrate this answer's advice with what we already know? Are the gems and turds so hard to distinguish? If we do and they are, we don't actually deserve the rep by our names, do we. Jun 17, 2014 at 18:45
  • @MichaelT - You are right, I did not explain all the possible options, but I have added some more based on your input. If you can think of other things that help with user retention, then feel free to edit as well.
    – rolfl
    Jun 17, 2014 at 19:05
  • 1
    SevenSidEddie - what you say is true, this is common sense, however the process still fails on the huge sites, so i think rolfl's answer is spot on.
    – Rory Alsop
    Jun 18, 2014 at 6:02
45

It's critical to scrutinize first posts. They're by far the most likely source of spam, and the most likely to have other problems.

You say

Allow flagging, commenting, comment voting, etc... But don't count those as a review action

But I actually do think those should count as a review action. The point of the review is to give new posts more eyeballs, and encourage any actions that are necessary, whatever that may be.

Yeah, it's vague. That's kinda the point. Unlike all the other reviews (except Late Answers), the posts aren't the trigger for entering review - the user is. We have no idea what will be needed.

In fact, thinking about Late Answers, it should just be merged with First Posts, and renamed New Users. If you want to review just one type of post, that would be available through filters. The purpose of all three – First Questions, First Answers, Late Answers – is the same: increase views. "Reviewing" is just the easiest and most accountable way to do that. Any action one would take seeing the post elsewhere should be valid and allowed here.

In that vein, enough organic views should definitely remove it from the queue, since the purpose has been fulfilled.

6
  • 1
    I complete +1 from me. Jun 17, 2014 at 17:53
  • 1
    Personally I like the idea of splitting the question and answer queues. The tasks are different, so in terms of productivity it should be better to split them. (I have a similar viewpoint on daylight savings time, but we'll skip that) :-)
    – Elder Geek
    Jun 17, 2014 at 18:53
  • 2
    Well, I think they're similar enough that the separation could be better suited to filtering rather than different reviews, just as the current First Posts can be split to Questions and Answers
    – mhlester
    Jun 17, 2014 at 18:54
  • 1
    New User Questions and New User Answers? That's probably a more reasonable split.
    – Shog9
    Jun 20, 2014 at 0:00
  • @Shog I'd be fine either way. So long as both kinds of answers are combined, it doesn't make too much difference whether questions get separate queues or it's handled by filtering. I lean towards filtering over separate, but I'm also the type who'd spend some time in a "pot luck" review where you don't know what queue your next review comes from.
    – mhlester
    Jun 20, 2014 at 0:12
  • Thanks for the feedback - see feature request here: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:56
33

The first posts queue is a very helpful tool to guide new users. It should be expanded, not removed.

First posts (or early posts, not necessarily just the very first post) warrant more attention for:

  • guidance in comments, regarding how to improve a question which isn't suitable as posted, what extra information should be provided, what could improve the quality of an answer, etc.
  • spam triage
  • edits, as the user probably often hasn't mastered formatting — this is especially important on sites with MathJax
  • making extra sure that helpful first posts are rewarded by an upvote (as rofl notes, it's especially important to get new contributors to the stage where they can upvote)

The first posts queue is all about guiding new users. If it's misused 50% of the time, this means that 50% of users are getting this extra chance of guidance. Removing the queue just because it isn't as effective as it should be would be silly.

So yeah, this review queue is terrible and useless

Wrong. Dead wrong.

I'm thinking we could probably make this work if we narrowed the focus considerably; less "help new users in some mostly-unspecified way" more "help filter the wheat from the chaff"

This seems completely backwards. Helping new users in some mostly-unspecified way is very much what the first posts queue is about! Yes, triage is also involved — there are users (especially spammers) that we don't want to help. But primarily this is about guidance, about helping.

Before the first posts queue in its present form, I used to pay special attention to posts on the front page marked “modified X time ago username 1”. I deliberately went to these posts to provide triage and guidance. This is obviously very approximate: it doesn't catch cases when someone else modifed another post on that page afterwards, sometimes brings me to already-deleted posts, etc. Essentially, what I wanted a list of first (or early) posts requiring special attention, preferably with some system so as to distribute said attention evenly. Which is the exactly what the first posts queue does.

Ideally, a new user's posts should go into the first posts queue as long as the user hasn't been determined to be good or bad. I'd like to see something like this (I haven't thought about the numbers much):

  • A new account with 1 rep is considered new until further notice. All posts by a new account go through the first posts queue.
  • When a new account reaches 2 posts with a positive score, it is presumed good; posts no longer enter a queue based on the poster.
  • When a new account reaches 2 posts with a negative score, it is presumed bad; posts enter the low quality queue instead of the first posts queue.

Posts from new users should enter the first posts queue as soon as technically possible. We want to do triage and guidance as early as possible, ideally while the user hasn't closed their browser yet.

Each post should be seen by multiple pairs of eyes. I agree with the general idea that posts should exit the queue based on actions taken, though it's not as simple as having had a vote up or down. Some posts are just middling. I think the rules for number of reviews should be relatively complex, perhaps a sort of point system (actions grant points whether taken from the review queue or not):

  • the first upvote or downvote gives 3 points
  • subsequent votes give 1 point, but the second vote gives 0 point if it contradicts the first one
  • a close vote or any flag gives 1 point
  • an edit gives 2 points
  • a comment gives 2 points
  • a “no action needed” review gives 1 point
  • a post stays in review until it's acquired at least 4 points, with always 1 mandatory review

I haven't thought much about all possible combinations of actions, this is just an idea of what I think the system should be. The idea is to ensure that feedback is provided to the user.

I don't see the point in separating questions and answers. This queue is about providing tailored guidance: you never know whether the next post will be good or bad, so why would it matter whether it's a question or an answer?

To combat the serial reviewer problem, we need to combat serial reviewing in itself. This isn't a problem of the first posts queue specifically. Making “no action needed” not count towards badges and rankings might help, though I don't like it much — it penalizes conscientious reviewers because of some bad apples. I had proposed minimum quotas of each action to get badges. The objection was that serial reviewers would turn to acting randomly and undoing their damage would become harder — but I'm not convinced: are we really undoing the damage of serial reviewers now? Alternatively, we can punish serial reviewers: there are now tools to suspend users from reviewing, and we should reset offenders' progress towards badges.

Audits aren't doing a good job of revealing serial reviewers because too many of them are litigious and the ones that aren't are trivial to avoid without really engaging one's brain. However, patterns of behavior can be used to good effect — a user who spends 5s on average and always selects the same action clearly isn't doing a good job. A minimum time per review (with some averaging going on because there is the occasional short review) could be enforced or verified (“you cannot submit more than 10 reviews in 5 minutes, come back in 1 hour”); there are ways to make reviewing more annoying for serial reviewers while not impacting conscientious reviewers, and we aren't doing that at all at the moment.

4
  • 3
    "To combat the serial reviewer problem, we need to combat serial reviewing in itself." Yes! Yes! Yes! And good suggestions for how to do that too. Jun 18, 2014 at 16:39
  • I expanded on this in my answer, but I think there is a good argument to be made for splitting answers and questions apart in the review queues (potentially pulling the First Posts answers into the Late Answers queues). We review answers and questions differently, and I'd rather have all potentially troublesome answers in one review queue and questions from new users who may need guidance in another. On Stack Overflow at least I can see significant benefits in splitting reviews this way. Jun 18, 2014 at 19:15
  • 1
    The second two suggestions for determining which posts go into the queue are good, though I would prefer "two downvoted posts" rather than "two downvotes". This would allow folks to be given attention up to the point where they've demonstrated they're not learning anything from it.
    – Shog9
    Jun 20, 2014 at 17:07
  • 1
    Thanks for the feedback - see feature request here: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:52
28

What value do you see in having a First Posts review?

On smaller sites, I have primarily used it to find good posts and add a "Great first answer and welcome to the site!" type of post.

Some of the smaller sites tend to do this more than Stack Overflow I've noticed. I think I've seen comments like that on Workplace, TeX, and Parenting.

7
  • 15
    Sounds like noisy fluff to me. :/
    – Wooble
    Jun 17, 2014 at 15:50
  • 49
    @Wooble for smaller sites, this sort of thing helps contribute to community. Bigger sites like SO don't "care" about community the way a small site has to.
    – enderland
    Jun 17, 2014 at 15:52
  • 4
    And on Code Review ... we do this a lot
    – rolfl
    Jun 17, 2014 at 16:28
  • 1
    I believe that attitudes shouldn't change no matter how big or small the community/site is. If you don't welcome on Stack Overflow all the time same should be the attitude on other sites. Jun 17, 2014 at 16:41
  • 20
    @BleedingFingers there's a reason people always say SO is so hostile to new users... most smaller communities don't have this stigma or problem.
    – enderland
    Jun 17, 2014 at 16:43
  • 4
    Despise rudeness and hostility toward new users. If you can't handle the heat, then don't! New users need to know about phenomenons like meta, on-topic, close, opinion-based, unclear etc. One shouldn't slam the door on them. If you can spend countless hours chatting why can't you spend a few being nice and welcoming. All it takes is Hello and welcome to [so]. Request for clarification/reason to close/etc. Jun 17, 2014 at 16:51
  • @BleedingFingers Stack Overflow is not a model community. If anything, it should be the other way around: SO's community could do with learning from the smaller sites and imitating their behaviour. Jun 20, 2014 at 0:30
23

I see the purpose of the first posts review list in "Help them learn to use the site.".

It's a fact that most first posters do not understand the workings of this site when they post their first post.

The idea is then that there is always at least someone who can check if all the points in How do I ask a good question? are followed to a reasonable extent. If this is done in a timely manner then it is a valuable for all of us.

Of course the whole system would also work without a review list. Everyone who searches for newest questions does probably the very same things.

But for me it feels like a step in the right direction. Newbies don't know the rules, so you have to help them (by improving or by flagging for deletion) as early as possible. What's wrong with this logic?

2
  • 2
    I agree that "the purpose of the first posts review list in "Help them learn to use the site." I received plenty of valuable guidance on askubuntu.com. I like to think that resulted in my becoming a valuable member of the team. Without that guidance I would have likely found another way to be useful elsewhere.
    – Elder Geek
    Jun 17, 2014 at 19:00
  • 1
    +1 I think many of the regulars have been on SE so long they've forgotten that the rest of internet doesn't work like SE does. StackOverflow is an exception because it's so absurdly massive in its industry (it IS now the standard for how the internet works for programming questions), but for every other site, people need to learn somehow. Jun 18, 2014 at 9:49
19

As a fairly active reviewer on SuperUser lately, and a really active one on ServerFault historically, I'm going to throw in my two cents' worth, which is that the First Posts queue is a valuable one, as-is, and shouldn't be tossed out. It could probably be improved (as can just about everything), but even without improvement, it serves some vaulable functions we'd miss if it was gone.

There are plenty of people who ignore it, or abuse it, but it really does allow higher-rep users of the sites to welcome and/or correct new users. Yes, most of the posts are mediocre, a good portion are utter crap, and only a few are really worth anything at all. And, as a result of most of the posts being "meh", sure, most of the actions taken will be... nothing. I think that's just fine, though.

Really, I think that focusing on the fact that the majority of new posts are really mediocre misses the point. Last I checked (and granted, it's been a while), StackExchange was aiming for most of its traffic to come from Google and other search engines. We have a very high number of drive-by users, who'll post something once, and never be heard from again. The point of the First Posts queue isn't those people who won't contribute, the point is to help people who do want to contribute. (Though throwing more eyeballs on the drive-by posts seems like a good idea to me.)

By bringing attention to new users as a whole, the First Posts queue lets the members of the community who want to guide potential new users of the site in how to use it, do so. The fact that the majority of site visitors or even site posters won't ever become members is irrelevant - the queue is there for those who do become members, to make it easier for them to contribute and get guidance right off the start... and frankly, I think the benefit of shining a spotlight on a class of post that's most likely to be spam or utter garbage, so it can be dealt with swiftly is a good thing™ as well.

When you get right down to it, StackExchange is a collection of user-generated content, which is its greatest strength on Google, so anything that helps improve that content (or the users generating it), seems like something we should be very wary of tossing out, even if it is difficult to do properly, and irritating at times. Having the First Posts queue, even as-is, is certainly better than nothing, and complaints about failed audits are probably a small price to pay for what the queue does for us.

Suggestion for improvement:

My suggestion for improving the First Posts queue would actually be to expand the queue. Instead of just first posts, it seems to me like it would be handy to throw a user's first several posts in the queue. After all, the queue text does say it's about helping new users learn to use the site... by restricting it to their first post only, it doesn't really let high rep users do that - it's not easy to provide feedback, or change based on a single data point, unless the data point is exceptional in some way. By expanding it encompass more posts from new users, it would allow more opportunity for feedback to those who decide to become members, or try the sites out, and I think it's hard to argue that this wouldn't be a good thing.

5
  • 13
    +1, ...by restricting it to their first post only, it doesn't really let high rep users do that... absolutely. This should go for our 101 rep neighbors as well.
    – MDMoore313
    Jun 17, 2014 at 18:31
  • 12
    @BigHomie Absolutely. It should be per-site, and not let folks with association bonus slip under the radar. I may know how the sites I participate in work, but that doesn't mean I have the first clue about what's expected of me on, say, academics. Of course, I'd read the FAQ and such... but not everyone who joins a new SE site does. Jun 17, 2014 at 18:35
  • 2
    likewise, and the level of questions vary dramatically from site to site. SF doesn't take kindly to low researched questions, but other sites do, for example.
    – MDMoore313
    Jun 17, 2014 at 18:40
  • 3
    @BigHomie User with the associations bonus are already being put in the First post review see this post meta.stackexchange.com/questions/128255/….
    – Marc-Andre
    Jun 17, 2014 at 19:55
  • @Marc-Andre that's good to know, I wonder if we can get that changed like Hopeless is suggesting.
    – MDMoore313
    Jun 17, 2014 at 20:48
17

Why punish genuine reviewers for having robots in their midst by removing / limiting their tools? It's not their fault. Punish those pesky robots instead! Surely there are ways to statistically determine which members are compulsively pushing always the same buttons, no? So let's flag those, and auto-ban from further reviewing until a mod looks into it, and decides if the flag should stand or not. Frankly, I think that the First Post reviews do work on sites I'm most active, albeit those are smaller than the SO / SF / SU trilogy. If something needs changing, then please provide more feedback on actions that should be used less frequently, like comments.

1
  • You're not the only one here to think I was talking about "robots", but since that's the primary focus of your post I'll address that here: "robots" - people determined to just blaze through the queue clicking as fast as they can - are a fairly small part of the problem, and relatively easy to detect. The bigger problem is that once you give folks a "meh" vote, it's easy to avoid making a decision... And so a lot of folks who are otherwise-conscientious avoid making decisions, because why stick your neck out? I'm not talking about some pesky group of "others" - I'm talking about US.
    – Shog9
    Jun 20, 2014 at 17:15
16

Even for the trilogy sites, I still don't see what negative effect the queue has on the site.

What's bad about it, apart from the fact that trying to comment on an audit fails?

Whilst 50% of the reviews are NAN's...:

  • 4,000 reviews are improving the quality of the site through edits
  • 10,000 reviews are either flags or VtC's on bad content
  • 15,000 of them are leaving a signal to the user about their first-post; be it an upvote, downvote or comment.

Given the recent talks about improving the quality of the site, removing one of the tools which audits quality seems an awful idea.

The review queue is almost always empty. People are happy to review through the queue. I really don't see the reason to remove it.


If you want a specific goals for the first post queue, I'd say:

  1. To ensure new users get feedback about their new question.

    My gripe with the First Post queue is that it allows people to flag/ downvote/ VtC questions without giving feedback to the OP that there was anything wrong with the question.

  2. To target posts which are highly likely to be sub-par

    My other gripe with the first post queue is that it only works for a users first post. Expand the queue to:

    1. A users first n posts
    2. A users next n posts after a post of theirs was deleted
    3. A users next n posts after a question of theirs was closed
    4. A users next n posts after a post of theirs received a negative (or < y) score.

The beneficial outputs for me is that:

  1. New users get more feedback (and hopefully a more friendly experience) with their first actions on a site. People who purposefully go to a review queue are (hopefully) there to help and assist people. People who go to a question organically are there to answer them or get answers. They're less likely to give feedback to a user, short of casting a anonymous close/ downvote.

  2. We get to remove crap content quicker

5
  • 1
    50% effective was the best we could do in LQ for a while too - which also meant we couldn't really use the output of that queue for much. Getting it up over 90% opened up a lot of opportunities, and it'd be nice to have those for new user posts too - right now, those are a huge Achilles heel in the quality filtering system, as we have very little signal to work with when they appear.
    – Shog9
    Jun 20, 2014 at 17:29
  • @Shog9 your comparison against LQ queue looks slippery to me here (unless I missed something about what you denote with 50% vs 90%). LQ has only two actionable (non-skip) outcomes, OK and delete. But in FP, there are more outcomes, and per my understanding, NAN in theory adds in these, not makes that 50% miss you seem to assume it is. Even in case if you dislike NAN and feel it as some sort of "miss" anyway, comparison against LG doesn't feel right here
    – gnat
    Jun 27, 2014 at 11:04
  • NAN accomplishes nothing, @gnat, apart from removing the item from the queue. Crucially, it isn't used for anything anywhere else in the system, so it's essentially the same as clicking "skip" with the added "bonus" of depriving anyone with interest in the task from seeing it via review. We added it purely because folks couldn't be bothered to click "skip" when they didn't know what the hell they were doing; it was a hack. Now... We have a few options here to fix this, but they all boil down to some form of "use NAN for signal somewhere". I'll have a more specific suggestion here shortly.
    – Shog9
    Jun 27, 2014 at 15:39
  • @Shog9 oh... I see now thanks! I didn't expect it to be that way, need some time to chew it up. The way you describe, it makes a problem indeed (myself, I use NAN as I thought it works "in theory" - that is, I expect that my NAN indicates that post passed thorough verification and has no issues worth actioning)
    – gnat
    Jun 27, 2014 at 15:41
  • 1
    We must be careful not to expand the scope of this too far, lest we end up reviewing every new post from a good many users. At some point, the system has enough past signal to decide to just ignore new posts from a user, or even send them straight into LQ review. That said, I think it is worth adding some knobs that could be tweaked for this purpose: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:56
15

First Posts review queue should be about the User, not the Post.

At first glance, that may seem contrary to what SE stands for, being about content, but not the user. But the very nature of the First Posts queue is one that targets a User. It's not looking for "potential low quality posts", it is looking for "posts by new users".

I know SE doesn't want to be a social media site, however don't lose sight of the fact these sites are built and run by a community of volunteers, and to keep the site healthy we need to keep recruiting new members to replace older members that move on.

The First Posts queue is the perfect place to do that.

So I think we should expand this queue and make it clear that it is a queue about the User, and not the post.

Some ideas I had for that

  • Show the new user the result of their review. If it was "Post looks fine", then show them that. If it was "Something is wrong", show them that too along with links to appropriate resources on how to ask/answer.

  • Textual feedback should be required here. I don't care if its canned responses, or something as simple as "Post looks fine", but we need some kind of textual feedback to give the user that explains what the result of their review was.

  • Reviewers should not feel the need to welcome new users using this queue. It should be right there in the review text for the new user

    Welcome to Something.SE! Because you are a new user, your post has been reviewed by other community members, and the following feedback was left for you:

  • It should be clear to regular SE users that this is not a queue you go to when you want to evaluate posts. It is a queue you go to when you want to help out new users who are starting to contribute to the site for the first time. Reviewing posts is a part of that duty, however it is not the primary purpose.

  • Change the text on the queue to something that more accurately describes what you want users to do in the queue, like "Help guide these new users".

    Right now it says "Review first posts from new users", so its no wonder that many people think it is just a queue of posts that have a higher chance of being "bad", and don't understand that the original purpose of the queue is to welcome and help new users (I didn't know that until I read the meta post you linked in your question).

This post makes me think of a question I asked a short while ago on Meta.SO, where I asked for suggestions on how we could promote new user guidance and/or mentorship in how to use the SE sites, in order to maintain a healthy user base.

I think updating the First Posts queue to make it focused on giving the new user feedback would be a great step in that direction.

14

Having been poking at the 10k view of the first posts queue for a bit, there are two goals... to give the good posts some attention and to lynch provide prompt feedback for poor posts.

Yes, there's about a 1 in 5 chance that the post you're looking at will be utter crap, or at very least something you'll want to down-vote. But it is just as important to provide some sort of feedback to the new authors whose posts aren't terrible - whether that's an up-vote, or a comment, or even a small edit that puts a bit more of a shine on their already-useful contribution.

(from New Review Queue: First Posts)

The thing is that these actions are a tug of war to an extent - pulling the review in different directions with a big hazy gray area in the middle.

One idea would be to break this into two queues with clearly defined aspects and triggers for inserting into the queue or removal from the queue.

There's the new low quality queue. Posts that need a firm push that is triggered either by a close vote or a down vote (or more down votes than up votes?). For this, the available actions are only corrective - similar to the existing low quality queue. You can flag, close, comment and edit. If the post gets these organically, the post is removed (don't overwhelm a new user with half a dozen edits or comments from people trying to complete a review).

As I write this, this sounds quite a bit like the existing low quality queue... but that doesn't see too much action from new posts. But here's the kicker... you need to do two actions to complete the review. Its a "you are on the path for writing bad questions, you need some corrective action and we will show you how to go down that path.

Then there's the new user encouragement queue. As mentioned in What happens to new users?, if we encourage users with some interaction they are more likely to come back. This queue is only for posts with 0 or 1 votes and no comments. The possible actions are: vote, comment, edit, flag/close (because its still possible it needs that interaction). If a question organically receives this interaction outside of the queue, the review is removed. As with my idea for the new low quality queue, this would require two actions to complete a review.

If the queue is about guiding a user along the path to being a good SE user, the two queues represent a minor correction and a major correction to what is needed in a post.

Furthermore, the multiple necessary actions for completing a review would serve as a "this queue takes work, its not a simple 'no action needed' or 'up vote' and do the next review."

1
13

On sites of a certain size (too big to read every post without the risk of missing one or two, but not so big as to have the problems you mentioned), the first post queue is invaluable for finding promising new participants. Some communities have developed organic welcoming committees who patrol that queue. So given the choice between eliminating or fixing First Posts, I'd like to see it be a site optional queue.

But if we really want to fix the queue, we need to figure out why people misuse it. The cynical side of me says that people who actively are annoyed by new users use the queue to punish them. More likely most reviewers just don't care. Likely they visit the queue in order to get a badge or see review tasks get smaller. If that's the case, we could try disconnecting the FP queue from the numbers that might be driving no action reviewers.

I was going to suggest some incentives for being helpful, but every scheme I could think of amounts to giving out lucite plaques. Helpful people like being helpful for it's own sake.

2
  • 4
    Maybe an 'accurate reviewer' yearly badge? (reviewed more than 200 reviews in the queue and had a long 1 month outlook of > 90% accurate (accurate being 'review matched the state of the post after 1 month')). This could also be examined for driving the stick too (your reviews were accurate less than 25% of the time... you're going to take a month off from this queue).
    – user213963
    Jun 17, 2014 at 21:44
  • My personal take is that the vast majority of reviewers find it difficult to hold strong opinions on most of what they see; forcing them to express an opinion ends with them choosing whatever they see as doing the least harm. I've refined the feedback here into an explicit feature request: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:54
8

One useful thing First Posts accomplishes, in smaller tags, is it gets really really bad things closed or removed very quickly. Due to the CV queue overload, it's hard to get questions closed in tags with very few close-eligible active followers; so often questions just never close.

It also introduces some viewpoints from higher rep users not following these smaller tags, which can be a double edged sword (sometimes you have things closed/edited/commented on that shouldn't really be because the higher rep users don't really know the tag's specifics); but overall it's helpful getting additional points of view to tags that might only have one or two higher rep users commonly helping herd the sheep.

So, I would try to keep it in some fashion - perhaps keep it as is, but remove the audits. I don't feel like audits are needed for FP; the point is for people to look at questions, which they could do anyway just by looking at questions on the main page. Remove the badges, if that's the problem with the queue. I think enough people care about the site that FP would be read no matter whether a badge exists or not.

1
  • 2
    This is an excellent point.
    – Shog9
    Jun 20, 2014 at 17:31
6

What value do you see in having a First Posts review?

  1. For communities about programming it is often code formatting. Because some new members often seem to don't know about preview of posts, which makes their posts almost unreadable. Sometimes nobody even don't know how OP supposed his post to look like.

  2. Sometimes new members don't care about punctuation, capital letters, because they think that it is no rules as in Facebook or Twitter (however sometimes I meet very good-looking and formatted questions from 1 rep users). So their posts look like:

hey guys something is wrong with my code i dont know i am new please help this is my code:

{some bad formatted code}

whats wrong please help


What outputs do you see as valuable from such a queue?

Less bad formatted questions and unclear questions from new users (I understand that it is low-qua queue for some of such posts, but anyway...).

6

I've always thought that First Posts (and possibly Late Answers) should be reviewed by more than one person each. Somewhere between three and five people should be looking at each of them to provide feedback. This provides more of an opportunity for the user to actually get some feedback rather than No Action Needed.

Another option is to remove the badges from this review queue. But that opens up the question of whether anyone will review posts without the prospect of gaining a shiny gold badge in three or four years.

3
  • I recall that about a year ago, this queue was set up for more than one reviewer. It was even worse than now, giving lots of totally unjustified, indiscriminate "welcome upvotes" to the posts
    – gnat
    Jun 18, 2014 at 14:59
  • 1
    @gnat Without NAN, then yes that would happen. Jun 18, 2014 at 15:55
  • Thanks for the feedback - see feature request here: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:52
5

I would consider it a valuable output if review presented a reasonably accurate estimate / prediction of how a post is going to be received in a future.

To keep it practically scoped, I would limit "future" to one month after review completion (maybe 2-3 months if recently proposed longer term auto-deletion gets implemented).

In this case, usefulness of review could be analyzed by checking post status (positive, negative, neutral, deleted, edited) a month after review and comparing it against review outcome.

  • Somewhat more ambitious goal of review could be to estimate how a poster is going to be received, analysis for this could then involve "aggregate" of first, 2nd, 3rd etc posts and comparing it against review outcome. For the sake of simplicity I'll keep that out of my answer.

Results of retrospective analysis can be further used to adjust reviewers behavior if it deviates too much from observed data. System can plant more "known bad" audits into the queue if reviewers tend to be too optimistic and more "known good" ones for the opposite outcome.


Note how similar approach can be used to analyze and adjust behavior of individual reviewers.

System can check and analyze later state of the posts they reviewed and present them an individually tuned desired mix of audits. It can even report particularly gross individual deviations to moderators for further manual analysis.

Difference from "aggregate" case is that for "individual" analysis and corrections to be timely, a shorter (and thus less accurate) retrospective time frame is probably needed, a week or two instead of a month.


Speaking of review audits, their primary goal is said to help hone moderation skills and per my observations of FP queue, audits there are too infrequent to accomplish that goal. You simply don't show folks enough "known good" / "known bad" examples to learn from.

  • Note by the way that more "known bad" audits would help decrease the problem you seem to worry about, one about folks who "don't do anything at all", simply because such audits fail at No Action Needed.

Teaching reviewers to appropriately use Skip action would also help getting better quality, more relevant reviews.

1
4

I'm speaking as a Stack Overflow moderator, and this is based on my experience there, but I see value in splitting answers out of the First Posts queue and rolling reviews of them into the Late Answers queue in a new form. We review answers and questions differently, and lumping both into the First Posts queue doesn't work too well on SO.

The primary purpose for reviewing both Late Answers and answers within the First Posts queue is to find spam and non-answers. Very rarely do we see spam in questions, and that's usually caught because questions appear on the front page of the site. First Posts review of answers can catch spam or non-answers that appear on newer questions (where Late Answers doesn't trigger).

Reviews of questions are different, in that you're typically looking either for inappropriate questions to close, or for new posters who could benefit from a little guidance. The latter is very important on smaller sites (as others have stated), but doesn't scale to the Trilogy. People leaving answers generally don't need such guidance, and they are more likely to be positively received.

I know that when looking through a suspect reviewer's recent history, I skip question reviews in First Posts and only focus on if they approved bad answers. Question reviews aren't all that meaningful on Stack Overflow, but the answer reviews in First Posts are.

I'd be all for a Late and New User Answers review queue, which would be very useful on Stack Overflow, with a separate Questions By New Users review queue. That queue could be enabled on the smaller sites, and potentially disabled on SO, if we thought it didn't bring any benefits.

4
  • 5
    I'm still unconvinced that questions and answers should be split — yes, they do lead to different types of advice, but variety is intrinsic in the first posts queue. I'd rather not see yet another entry in the list of queues; if they're to be split, do it with an optional filter, like close reason filtering is done now. Jun 18, 2014 at 21:21
  • @Gilles - I'm not necessarily suggesting a new review queue, just that the answers from the First Posts queue be moved into Late Answers for an umbrella answer review queue. That would leave a First Questions queue and I think better separate mindsets for reviewing content in the two queues. Jun 18, 2014 at 21:24
  • I completely agree with your points, but came to a slightly different conclusion in my answer. Since filtering works so well, why not just combine the two queues entirely and handle question/answer through filters too? Neither queue ever gets very full, so they could benefit from being combined. Questions and answers are different, but the purpose of the review still substantially overlaps
    – mhlester
    Jun 19, 2014 at 5:04
  • 1
    Thanks for the feedback - see feature request here: meta.stackexchange.com/questions/236537/…
    – Shog9
    Jul 15, 2014 at 2:52
3

Speaking from experience with reviews on GIS.SE, I'd say the process is flawed:

The review menu item is always live, even when there is nothing for me to review, which is most of the time. Thus I'm reluctant to waste my time looking.

The few times there is something there to review, I'm unsure what I'm supposed to do or how to do it -- especially since I already have powers to comment, vote or edit outside of this "review" process (and I do exercise those powers).

If first posts are withheld until they pass "review" (is that the case; I don't know?) then the process isn't working because I see many low quality posts and non-upvoted good posts outside of "review".

[Forgive me if this bit is off topic.] The edit review process is also flawed because I'd sometimes like to see the full Q&A with comments before I judge a suggested edit but I cannot do that inside the "review".

6
  • 2
    Something I wrote for P.SE as guidance for how to complete a first post review: meta.programmers.stackexchange.com/a/6681/40980
    – user213963
    Jun 17, 2014 at 22:08
  • @MichaelT: Shouldn't that be offered as an answer to the OP?
    – Martin F
    Jun 17, 2014 at 22:46
  • 1
    I'm fairly sure that Shog is aware of that post (even if he doesn't comment on it, he does look at per site metas, and that one's a featured post on P.SE). I believe he's much more interested in the issues of "people just aren't doing useful reviews - is the queue useful?" We can give them all the guidance on we can write on how to do a good review, but if its not being used well - what can we do about it then?
    – user213963
    Jun 17, 2014 at 23:18
  • A system that relies on lengthy explanations hidden in some Q&As is obviously a waste of effort.
    – Martin F
    Jun 17, 2014 at 23:59
  • 1
    The documentation of the collective opinion of the people active in it is very useful for understanding where it is now and where it has been. For example, questions of topicality asked on meta can be used to demonstrate the evolution of the site's community over the years. While meta may not be known to everyone, it should be known to people who are active in reviewing - be it close votes or first posts.
    – user213963
    Jun 18, 2014 at 0:02
  • Re: "The edit review process is also flawed because I'd sometimes like to see the full Q&A with comments before I judge a suggested edit"... What's stopping you from opening the question in a new tab to review the full context? This is something I do constantly during my reviewing process.
    – Elder Geek
    Feb 24, 2015 at 15:27
2

I did not read through every single answer already given here, so i apologize if i'm duplicating something already said.

I think that First Posts shouldn't be limited to just a user's first post. Until they show that they know how to use the site, they will be sent back to the queue. Once the user has posted something with a positive score, we'll leave them out of this queue. But if they post a question that gets -1, their next question will also go into review. And if that one is negative, their next post also, until they get something positively scored.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .