What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

As it stands, the "First Posts" review queue is empty on Stack Overflow. As I seldom review that queue, it isn't empty because I have no posts to review; it's empty because, well it's empty.

However, in reviewing some of the 800 flags in the 10k Flag Queue, I came across so, so many posts by first users, which are blatantly low quality/ NAA, most of which had no comments at all, and those that did, I'm doubtful whether they were added during the review of the post in the "First Posts" queue (if it ever reached there).

Note that the guidance for the FP queue reads (emphasis mine):

Be sure to leave a comment if you can help the user out, upvote the question if you can't find any problems with it, or click Skip if you are not sure and want to go to the next item.

It seems a waste of everyone's time if all of the following posts had to cycle multiple reviews, be seen by 10k'ers in the 10k queue, before been seen and finally handled by diamonds (and the whole while, the OP has no idea they've done anything wrong, due to the lack of feedback):

... and there's more, many more, but you get the gist.

How can the "First Post" queue be changed to better address this?

  • Should the addition of a comment be enforced if a "flag" was made?
  • An extension of the above; a comment for anything other-than an up vote (yes, even a downvote)?
  • Should the ability to "Delete" / "Recommend Deletion" exist like the LP queue
  • Do we need more pairs of eyes per-post to ensure the correct action is (eventually?) taken?
  • Would the current system work, except the robo-reviewers are making this queue so ineffective?
  • ...?
share|improve this question
11  
I can confirm that all of these posts went through at least one review (either First Posts or Late Answers). Most of them went through multiple reviews. Update: Only four of those were marked "No Action Needed" by a reviewer. –  Bill the Lizard Mar 7 '13 at 12:28
    
You say that these posts were all flagged. Doesn't that mean that the review process is working? Posts can't be deleted directly from the First Posts review queue, you can only flag them and wait for diamond mods to delete. –  interjay Mar 7 '13 at 12:48
1  
@BilltheLizard: Will your mod powers let you see if any of the reviews added the helpful comments to the post (as opposed to the comments being added by helpful drive-by-ers). –  Matt Mar 7 '13 at 15:28
2  
Only one of the helpful comments was left by a person who completed a review (that one from the Late Answers queue). All the rest were by helpful pasers-by (mostly Andrew Barber). –  Bill the Lizard Mar 7 '13 at 16:15
8  
It's that damn Andrew Barber! He's so hot right now! –  LittleBobbyTables Mar 7 '13 at 16:31
add comment

10 Answers 10

Here's a breakdown of FP reviews in the last 30 days:

Total Reviews              64007
No Action Needed           37447
Edited (or suggested edit)  4458  
Upvoted                     5262  
Downvoted                   4025  
Flagged                     8464  
Commented                   2642  
Flagged + Commented          277      
Voted to close              1453   
Voted on a comment          2363       
Voted to delete                7      

Also, roughly 2000 posts that were passed with "no action needed" were later flagged, and about 1800 posts that had some action other than NAN or flag were later flagged. That's about 5% and 10% respectively.

So yeah, two potential problems here: folks aren't flagging everything that needs it, and they aren't leaving comments when they do.

I like Aditya's suggestion for multiple reviews - that's not a panacea, but it does help to have more eyes on stuff.

It would be nice to add canned comments (as with the LQ queue) for "not an answer" flags. I kinda think this would be useful even outside of review.

For completeness, here are the numbers for the similar Late Answers queue:

Total Reviews              19927
No Action Needed           10563
Edited (or suggested edit)  1030                 
Upvoted                      789       
Downvoted                   1097       
Flagged                     6072        
Commented                    944       
Flagged + Commented          271           
Voted on a comment           192      
Voted to delete               10                    
share|improve this answer
    
Would you be totally against adding comment prompts on downvotes as well? A downvote without a comment for a new user gives as much (i.e. as little) feedback as a silent flag. Obviously this is fair game normally on SE, but on the FP queue which is there to help new users out, we should be prompted to help them as much as possible IMO. –  Matt Mar 7 '13 at 20:12
7  
I think that would be incredibly noisy, @Matt. Not to mention impossible to nail down a usefully small set of canned comments. –  Shog9 Mar 7 '13 at 20:15
    
Humm, I agree on the difficulty of coming up with a list of useful subsets. A middle ground might be the prompt similar to the ones new users get when downvoting ("consider adding a comment if you think this post could be improved") or similar (might still be too noisy). Also, what are the next steps for seriously considering canned comments/ multiple reviews? Is a feature-request necessary, or is this question enough? –  Matt Mar 7 '13 at 20:23
1  
Is there any way to come back on those users who moved for NAN and then the post was later flagged or edited? –  jcolebrand Mar 8 '13 at 22:40
1  
I tend not to leave a comment when I flag as NaA because the poster is likely not to be notified of that comment (he only gets the notification if he visits the site before the answer is deleted). If this was fixed, I would leave helpful comments (like I do where I'm a mod). –  Gilles Mar 8 '13 at 22:44
2  
@Matt here's the problem with prompts on downvotes; across my 5 top SE accounts, I've cast 1769 downvotes. That's a hell of a lot of prompts. It always suggests "leave a comment" sometimes, I think that's plenty (or too much, since I've been aware of leaving comments for >1 year). –  Ben Brocka Mar 8 '13 at 22:53
1  
@BenBroka: I was specifically talking about downvotes on the FP queue, not prompts on downvotes across the site. The FP queue is all about educating new users; and I would put up with a little bit of more prompting on the FP queue, if it ensured a better first-time user experience for these new users. –  Matt Mar 8 '13 at 23:48
    
@Gilles I've had at least one person get snarkey after I clicked flag and it left up with the default "not an answer" message. "I DO NOT appreciate *new* users telling me how to use the site..." for about 3-4 messages. –  hayd Jun 12 '13 at 22:55
    
@Shog9 is the query accessible to anyone? I mean the breakdown of the FP review queue? I would love to check that out. –  Braiam Aug 27 '13 at 20:19
1  
I'm getting increasingly frustrated with the number of people who flag and downvote without commenting on AU. How the hell do they expect people to improve their posts if they don't explain the problem?! I understand that voting should be anonymous but allowing downvotes without comments is toxic. There are technical solutions available (anonymous "vote reasons", like "close reasons") so can't we have one? –  Oli Aug 27 '13 at 20:38
    
@Oli Why should I be forced to comment when I downvote Bob Joe's giant block of GIVE ME TEH CODEZ that has grammar so bad I can hardly read it? He obviously didn't put any work into the question, he isn't going to do anything even if he sees my comment, which he often won't. –  ɥʇǝS Jun 17 at 23:56
add comment

First Posts and Late Answers Review queue is almost always empty. If luck favors me then I find the posts available for review once in 3-4 days. I find this specially true at Stack Overflow (not at Ask Ubuntu where I am much more active).

Reason I believe is known to all - BADGES.

Some of the suggestions to combat it would be:

  • Time based restriction

    Don't let people hit the relevant buttons - I'm Done and No Action Needed for a couple of seconds since their last review (something like 15 seconds or so).

    To clarify, there shouldn't be any time-bound restriction for pressing Skip. Moreover, if the post is Flagged or a comment is posted, then this time-bound restriction should be lifted for pressing I'm Done.

  • Multiple Reviews

    Just like Suggested Edits Review Queue - multiple eyes on the same post.

I am heavily in favor of Time based restriction. It won't disturb the workflow of people who actually read the content. At the same time, make Robo-Reviewers do something while they cannot press the button to their liking. See this example of one of the Robo-Reviewers - reviewing 20 First Posts within exactly 250 seconds!!

signs of robo-reviewers

This definitely signals that the reviewer isn't reading the content at all.

share|improve this answer
11  
regarding "Don't let people hit the relevant buttons... for a couple of seconds", there is (severely under-implemented) related feature request: Drop delay for “Skip” and increase it for “action” review buttons (status-completed for only dropping delay for Skip, gimme a break) –  gnat Mar 7 '13 at 18:55
5  
The second suggestion is brilliant! –  Adnan Mar 8 '13 at 16:22
5  
That image is ridiculous. –  hayd Jun 12 '13 at 22:47
1  
+50 for the image! –  hayd Jun 15 '13 at 21:20
2  
I really like the time based restriction. Sometimes it shows 20 items in the review queue - by the time I review 1 there are no more items left... and i am not mad clicking NAN –  user221081 Jul 3 '13 at 9:50
    
@gnat: Just noticed that the question you linked is tagged as status-completed... It is status-completed as regards to dropping the delay for Skip button, but doesn't do anything about increasing the delay for "action" buttons... What do you think we should do? (And to add to the humor, just noticed that you had written it in brackets that it is status-completed - I would leave this comment as a humor - let me certainly take a break)... –  Aditya Jan 14 at 13:10
    
@Aditya one thing that comes to mind to address such a half-arsed completion, would be to open a dedicated feature request "increase delay for “action” review buttons"... –  gnat Jan 14 at 13:35
add comment

I've tried to do a bit of SO reviewing, and I've noticed that the FP queue is almost always empty because there are hundreds of badge-hungry robo-reviewers sitting on it (keep refreshing /review, you'll see up to ten posts in the queue at a time, but they disappear by the time you click on the queue). So the ineffectiveness could be thought as a part of the larger robo-reviewer problem.

However, I fail to understand what you mean by "deleted in the first posts review queue". There's no option to delete there (unlike the LQ queue which prioritizes the post for 10kers when it gets a delete vote). The only thing FP-reviewers can do is downvote, flag, edit, and comment. If it's getting handled in the flag queue, doesn't that mean that the FP queue is working?

share|improve this answer
    
Humm, I was mistakenly thinking there was an option to "Delete" (or "Recommend Deletion" on the FP queue). It sort of makes me think the whole FP queue is rather toothless in this circumstance, and we shouldn't need a post to go though the FP queue (multiple times), through the 10k review queue, and to reach a moderator before it is deleted. –  Matt Mar 7 '13 at 14:25
1  
Note also that the FP queue hints that comments should be left to help the user, and many of the posts I linked to had no comments at all (it's impossible for me to prove that the existing comments were during the review in the FP queue). Maybe the FP should require either a comment, or a up-vote (or a skip of course), to give new users the feedback they need, rather than an invisible flag and eventual deletion. –  Matt Mar 7 '13 at 14:26
    
@Matt: The "recommend deletion" is in the LQ queue. If you're a 10k, it VTDs the post. If you're not, it prioritises the post for 10k users when they visit their LQ queues. Requiring a comment on FP review would be really annoying, and I've come across tons of cases where there is no need at all to comment (again, the flaggable ones are the minority, mostly). "Require an upvote" -- naah, we already know what happens if you let them upvote too much. –  Manishearth Mar 7 '13 at 14:42
    
I was thinking more about if you flag a post, it should prompt you to add a comment like "Recommend Deletion" does in the LP queue. Similarly for down votes; as well as FP being there to cut-out the crap, it should also be there to educate new users at the earliest possible opportunity, and at the moment, it's just not doing that. –  Matt Mar 7 '13 at 14:46
    
(also, I'm sorry for changing my original post such that it makes your second paragraph less relevant, I wanted to keep the essence of my question, without focusing on my incorrect assumption about there being the ability to delete). –  Matt Mar 7 '13 at 14:47
1  
This is a large part of the problem, as a ordinary reviewer you just can't access the queue when it's convenient for you, so I just end up not bothering. When there used to be a queue, I could easily use up all my reviews in one sitting e.g. as an actual break from work/answering questions. –  hayd Jun 12 '13 at 22:46
add comment

The problem is not that the queue is ineffective. The problem is that a lot of reviewers don't look at the posts being reviewed.

I do a lot of review on this queue and try to leave comment on answer that are question and stuff like that. And when I just don't know what to do with a post I skip it, and comeback later to see what has been done. And most of the time I see that reviewers been robo-reviewing with "no action needed" when an action was clearly needed. I think the idea of @CharlesB is interesting. Making the FP review queue a bit like the edit queue would probably help. Maybe making the post reviewed more than twice, or needing to have three times the same review.

Also, adding some text at the top of the page to remind user to always let comment on post they flag, because the post are made by new users who don't know StackOverflow and that we need to help them learning about it.

Also adding a small list of comment magic tag like [faq] or [about] to make the reviewers learn that these tags exist for comments.

I think that queue should stay because there are a lost of post that I find and flag with the help of that queue. Though it can be improved.


Edit: I've just seen that there is already something about leaving a comment but you have to click the more link. Maybe putting it to a place where it'd be always visible would be better.

Edit2: Also I think your idea of adding obligatory comments on flag/downvotes is a good idea. The thing is that if people do not flag/downvote bad posts and just do "no action needed", well then the problem is not solved at all. Maybe there could be some action taken is someone does "no action needed" too much on posts that gets deleted afterwards.

share|improve this answer
1  
I'm sure it used to be more than one reviewer, there are loads of poor questions from new users with precisely 2 upvotes... –  hayd Jun 12 '13 at 22:41
add comment

As I said in this answer to - Propose a privilege to sit in the 500 reputation mark

Perhaps it would be worth while to:

Make access to the first posts review queue a 500 rep privilege.
or as suggested in a number of comments, even higher at 1k or 2k.

Hopefully this will prevent the blind from leading the blind.

As a possible side benefit it may slow down the badge hunters till they get a better idea of what the review queues are really for and what's expected when doing a review.

While I wholeheartedly agree that having multiple reviews on each post would help a lot, I think it would help even more to have slightly more experienced reviewers.

share|improve this answer
    
Why not do both? Have two review queues. One for those with a lot of reputation and one for those with a little reputation. Or a more continuous set. Up to 10,000 reputation (or whatever), see all posts that haven't been reviewed by someone with your reputation or better. This would keep low reputation reviewers from hiding posts from higher reputation reviewers. Note that increasing the reputation needed to process a review does not really solve this. I can still see and comment on those posts, just not find them via the queue. –  Brythan Jul 27 '13 at 17:22
    
@Brythan Make a feature request and see how it does, I doubt that people will go for it though –  apaul34208 Jul 27 '13 at 17:25
add comment

Remove the badges

Reviewing is not a game, and reviewing requires serious effort and the intention to improve the quality of the site. We want people who care about the quality of the site to review, rather than people who just do review for fun to get the badges.

The gold badge should be removed - the goal is high enough so that some people will just use the safest option available to bypass all audits and click through the reviews. The silver badge might be kept, but probably can be toned down to make it easier to obtain (so that robo-reviewer would stop causing damage at silver badge, and it makes a good enough incentive for users new to the review queue to keep working on it).

The top ranking of the day and overall ranking should be moved to somewhere less visible, like how reputation and edit is currently implemented. Reviewing work is not something that can be raced for without causing collateral damage. Your reputation is only increased when your work is recognized as good by other people, but the number of reviews doesn't have to.

share|improve this answer
1  
Instead of removing the badges, perhaps it would make more sense to change the badges. For example, badges for helpful feedback (review comments that have been voted up, as well as the current helpful flags and accepted edits) rather than for hitting "No Action Needed" a bunch of times. –  Brythan Jul 27 '13 at 17:17
    
@Brythan: Well, I suggest removing gold badge since removes the motivation to robo-review. If you have a better scheme to motivate people to review while preventing robo-review from working, then by all means post it as an answer. –  nhahtdh Jul 28 '13 at 16:02
add comment

I share your views on this queue being inefficient. A post is reviewed only once, and a lot of the "no action needed" triggers are wrong. There was an action to take.

A way to fix it would be to have posts reviewed by more people; the queue is almost always empty, so there's room for this. We could leave posts in the queue even after they have been reviewed, say stay in queue until three people reviewed it (wether actions have taken or not).

share|improve this answer
add comment

There are solutions here to prevent robo-reviewers and to create an atmosphere of fairer processing. My solution is intended to complement those and address the primary issue here: how to add more items to the First Posts queue in order to prevent it from being so empty (emphasis based on the questioner's first paragraph).

Implemented in addition to other solutions here, which are focused on the consumers, my solution is to add more First Posts in general by Broadening the Qualifications of the First Post.

Flag any post (question or answer) as a First Post if the user hits any of the following ordered qualifications (note: italicized bold values were selected arbitrarily):

  1. had at least 1 post sitting in the review queue within the last 24 hours
  2. registered less than 1 month ago (optional: on all SE sites)
  3. has less than 50 reputation
    1. must not be single-sourced reputation (I registered my e-mail address; +50!)
    2. reputation preferably determined using only post-based reputation
    3. optionally may be total post-based reputation across all SE sites
  4. has less than 5 posts total
  5. whatever other rules qualify a First Post not already trumped by prior listed rules
share|improve this answer
1  
This would just increase the work available to be done for robo-reviewers? It doesn't address queue being ineffective at all (It'd justbe a bigger ineffective queue!)... –  hayd Jun 12 '13 at 22:39
    
Everyone can only review 20 First Posts a day. Even if robo-reviewers hit their daily maximums, by having a likely greatly increased number of items available for review, that leaves items for the rest of us normal reviewers since the robo-reviewers have been maxxed-out. Additionally, this solution is to be implemented IN ADDITION to another solution, which addresses robo-reviewers. Additionally, the question at hand is basically how to make the queue more full, as even the first paragraph emphasizes that it is empty. –  JoshDM Jun 14 '13 at 14:48
add comment

I think that current system would benefit from all the solutions suggested in the question, but I think its obvious that the primary cause of the problem is robo-reviewers.

Those that would do a proper review are crowded out by those that game the system to get another colored dot next to their name, keeping the queue empty and ineffective.

To better address the robo-review problem I propose an automated flag set up to trip whenever someone uses the same review response X number of times in a row or when they use an identifiable pattern.

As in:

  • Up-voted in review X times in a row: Flag
  • Down-voted in review X times in a row: Flag
  • No Action Needed in review X times in a row: Flag
  • And so on...

Or:

  • Up-vote, down-vote, up-vote, down-vote, up-vote, down-vote: Flag
  • Up-vote, down-vote, down-vote, up-vote, down-vote, down-vote, up-vote: Flag
  • And so on...

Of course we wouldn't want to auto-ban people who tripped a flag in this way, there is an outside chance that someone really did run into 5 or 10 posts of the same quality in a row, or that their responses fell into a pattern for a short run, but as its unlikely, it would be an easy way to spot troublesome reviewers and bring them to a moderator's attention.

If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing?

Just for a little poetic justice, when people get caught gaming for a badge, they should not only be kicked out of the review queue, their precious badge progress should be reset to 0.

share|improve this answer
    
If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing? The most an automated process can do is alert for robo-reviewing. A human is still needed to make the final judgment. Vote fraud is detectable due to the timeframe between the votes and how the voters are related by vote (in a vote ring). –  nhahtdh Jun 14 '13 at 5:59
1  
@nhahtdh That's why I pointed out that it would be a way to bring problematic reviewers to a moderators attention in the previous paragraph. –  apaul34208 Jun 14 '13 at 16:26
add comment

One problem with reviewing is that all reviews are treated the same with no grading of reviews. Contrast this with edits, where there is a two stage process. People can suggest edits but can't edit on their own until they have more reputation. Perhaps reviews should also have a multiple step process.

Someone who has a thousand helpful reviews (a flag marked helpful; an upvoted comment; an accepted edit; etc.) and no misses (where a miss is defined as clicking No Action Needed on a post that was later addressed) would have their review processed immediately. Someone who had three misses in ten reviews would get ignored; the post would remain in the queue as if that person had hit skip. Someone who had two misses and a hundred other reviews would get reviewed again; if two more people with similar histories concur, the review would come out of the queue; if there is a disagreement, it could get flagged for moderator attention (or put in a higher level queue).

The point of all this is to increase the chances that harmful posts will get flagged and weak posts will get feedback. It will also increase the number of posts in the queue (since many posts will get multiple reviews). For high quality reviewers, things would stay the same. Newer and lower quality reviewers would get more help.

All numbers are examples -- someone should analyze the data to see what numbers make sense.

It's possible that an even simpler rule would work. For example, instead of stages of good reviews, simply add the total helpful reviews of all agreeing reviewers and subtract something for misses. This would allow consensuses of less experienced reviewers to match a single more experience reviewer.

Note that both systems rely on misses having some impact. These systems only give misses an impact on the review system. An alternative would be to give misses an impact on reputation (say -20 for not flagging or skipping). That would also make it harder for people to review carelessly. I think that this is less effective though. Someone can get high reputation by asking and/or answering questions well in a single way. A reviewer needs to respect multiple ways of posting. They're different skills and I believe that they are better treated as such.

share|improve this answer
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .