The "First Posts" review queue is ineffective and toothless [duplicate]

Question

As it stands, the "First Posts" review queue is empty on Stack Overflow. As I seldom review that queue, it isn't empty because I have no posts to review; it's empty because, well it's empty.

However, in reviewing some of the 800 flags in the 10k Flag Queue, I came across so, so many posts by first users, which are blatantly low quality/ NAA, most of which had no comments at all, and those that did, I'm doubtful whether they were added during the review of the post in the "First Posts" queue (if it ever reached there).

Note that the guidance for the FP queue reads (emphasis mine):

Be sure to leave a comment if you can help the user out, upvote the question if you can't find any problems with it, or click Skip if you are not sure and want to go to the next item.

It seems a waste of everyone's time if all of the following posts had to cycle multiple reviews, be seen by 10k'ers in the 10k queue, before been seen and finally handled by diamonds (and the whole while, the OP has no idea they've done anything wrong, due to the lack of feedback):

• Java sever code to Push Notification to particular device in Android
• How can I remove the printing protection from password-protected PDF files?
• How to play an opening animation when iOS app launches
• Why jsfiddle is not rendering at all in IE8 and previous versions?
• Stagefright architecture
• How do I create an iPhone app with background and rotating images?
• How to embed a UITableView in a UIScrollview
• Hadoop Streaming Job failed error in python
• Generate unique patterns of numbers (e.g. 1221 considered same pattern as 2112)
• Uploading a file in PHP
• What is a StackOverflowError?
• jqGrid: Combining sortableRows and gridDnD (Duplicate Rows)

... and there's more, many more, but you get the gist.

How can the "First Post" queue be changed to better address this?

• Should the addition of a comment be enforced if a "flag" was made?
• An extension of the above; a comment for anything other-than an up vote (yes, even a downvote)?
• Should the ability to "Delete" / "Recommend Deletion" exist like the LP queue
• Do we need more pairs of eyes per-post to ensure the correct action is (eventually?) taken?
• Would the current system work, except the robo-reviewers are making this queue so ineffective?
• ...?

Answer 1

Here's a breakdown of FP reviews in the last 30 days:

Total Reviews              64007
No Action Needed           37447
Edited (or suggested edit)  4458  
Upvoted                     5262  
Downvoted                   4025  
Flagged                     8464  
Commented                   2642  
Flagged + Commented          277      
Voted to close              1453   
Voted on a comment          2363       
Voted to delete                7      

Also, roughly 2000 posts that were passed with "no action needed" were later flagged, and about 1800 posts that had some action other than NAN or flag were later flagged. That's about 5% and 10% respectively.

So yeah, two potential problems here: folks aren't flagging everything that needs it, and they aren't leaving comments when they do.

I like Aditya's suggestion for multiple reviews - that's not a panacea, but it does help to have more eyes on stuff.

It would be nice to add canned comments (as with the LQ queue) for "not an answer" flags. I kinda think this would be useful even outside of review.

For completeness, here are the numbers for the similar Late Answers queue:

Total Reviews              19927
No Action Needed           10563
Edited (or suggested edit)  1030                 
Upvoted                      789       
Downvoted                   1097       
Flagged                     6072        
Commented                    944       
Flagged + Commented          271           
Voted on a comment           192      
Voted to delete               10                    

Answer 2

First Posts and Late Answers Review queue is almost always empty. If luck favors me then I find the posts available for review once in 3-4 days. I find this specially true at Stack Overflow (not at Ask Ubuntu where I am much more active).

Reason I believe is known to all - BADGES.

Some of the suggestions to combat it would be:

• Time based restriction

  Don't let people hit the relevant buttons - I'm Done and No Action Needed for a couple of seconds since their last review (something like 15 seconds or so).

  To clarify, there shouldn't be any time-bound restriction for pressing Skip. Moreover, if the post is Flagged or a comment is posted, then this time-bound restriction should be lifted for pressing I'm Done.

• Multiple Reviews

  Just like Suggested Edits Review Queue - multiple eyes on the same post.

I am heavily in favor of Time based restriction. It won't disturb the workflow of people who actually read the content. At the same time, make Robo-Reviewers do something while they cannot press the button to their liking. See this example of one of the Robo-Reviewers - reviewing 20 First Posts within exactly 250 seconds!!

This definitely signals that the reviewer isn't reading the content at all.

Answer 3

I've tried to do a bit of SO reviewing, and I've noticed that the FP queue is almost always empty because there are hundreds of badge-hungry robo-reviewers sitting on it (keep refreshing /review, you'll see up to ten posts in the queue at a time, but they disappear by the time you click on the queue). So the ineffectiveness could be thought as a part of the larger robo-reviewer problem.

However, I fail to understand what you mean by "deleted in the first posts review queue". There's no option to delete there (unlike the LQ queue which prioritizes the post for 10kers when it gets a delete vote). The only thing FP-reviewers can do is downvote, flag, edit, and comment. If it's getting handled in the flag queue, doesn't that mean that the FP queue is working?

Answer 4

The problem is not that the queue is ineffective. The problem is that a lot of reviewers don't look at the posts being reviewed.

I do a lot of review on this queue and try to leave comment on answer that are question and stuff like that. And when I just don't know what to do with a post I skip it, and comeback later to see what has been done. And most of the time I see that reviewers been robo-reviewing with "no action needed" when an action was clearly needed. I think the idea of @CharlesB is interesting. Making the FP review queue a bit like the edit queue would probably help. Maybe making the post reviewed more than twice, or needing to have three times the same review.

Also, adding some text at the top of the page to remind user to always let comment on post they flag, because the post are made by new users who don't know StackOverflow and that we need to help them learning about it.

Also adding a small list of comment magic tag like [faq] or [about] to make the reviewers learn that these tags exist for comments.

I think that queue should stay because there are a lost of post that I find and flag with the help of that queue. Though it can be improved.

---

Edit: I've just seen that there is already something about leaving a comment but you have to click the more link. Maybe putting it to a place where it'd be always visible would be better.

Edit2: Also I think your idea of adding obligatory comments on flag/downvotes is a good idea. The thing is that if people do not flag/downvote bad posts and just do "no action needed", well then the problem is not solved at all. Maybe there could be some action taken is someone does "no action needed" too much on posts that gets deleted afterwards.

Answer 5

As I said in this answer to - Propose a privilege to sit in the 500 reputation mark

Perhaps it would be worth while to:

Make access to the first posts review queue a 500 rep privilege.
or as suggested in a number of comments, even higher at 1k or 2k.

Hopefully this will prevent the blind from leading the blind.

As a possible side benefit it may slow down the badge hunters till they get a better idea of what the review queues are really for and what's expected when doing a review.

While I wholeheartedly agree that having multiple reviews on each post would help a lot, I think it would help even more to have slightly more experienced reviewers.

Answer 6

Remove the badges

Reviewing is not a game, and reviewing requires serious effort and the intention to improve the quality of the site. We want people who care about the quality of the site to review, rather than people who just do review for fun to get the badges.

The gold badge should be removed - the goal is high enough so that some people will just use the safest option available to bypass all audits and click through the reviews. The silver badge might be kept, but probably can be toned down to make it easier to obtain (so that robo-reviewer would stop causing damage at silver badge, and it makes a good enough incentive for users new to the review queue to keep working on it).

The top ranking of the day and overall ranking should be moved to somewhere less visible, like how reputation and edit is currently implemented. Reviewing work is not something that can be raced for without causing collateral damage. Your reputation is only increased when your work is recognized as good by other people, but the number of reviews doesn't have to.

Answer 7

I share your views on this queue being inefficient. A post is reviewed only once, and a lot of the "no action needed" triggers are wrong. There was an action to take.

A way to fix it would be to have posts reviewed by more people; the queue is almost always empty, so there's room for this. We could leave posts in the queue even after they have been reviewed, say stay in queue until three people reviewed it (wether actions have taken or not).

Answer 8

I think that current system would benefit from all the solutions suggested in the question, but I think its obvious that the primary cause of the problem is robo-reviewers.

Those that would do a proper review are crowded out by those that game the system to get another colored dot next to their name, keeping the queue empty and ineffective.

To better address the robo-review problem I propose an automated flag set up to trip whenever someone uses the same review response X number of times in a row or when they use an identifiable pattern.

As in:

• Up-voted in review X times in a row: Flag
• Down-voted in review X times in a row: Flag
• No Action Needed in review X times in a row: Flag
• And so on...

Or:

• Up-vote, down-vote, up-vote, down-vote, up-vote, down-vote: Flag
• Up-vote, down-vote, down-vote, up-vote, down-vote, down-vote, up-vote: Flag
• And so on...

Of course we wouldn't want to auto-ban people who tripped a flag in this way, there is an outside chance that someone really did run into 5 or 10 posts of the same quality in a row, or that their responses fell into a pattern for a short run, but as its unlikely, it would be an easy way to spot troublesome reviewers and bring them to a moderator's attention.

If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing?

Just for a little poetic justice, when people get caught gaming for a badge, they should not only be kicked out of the review queue, their precious badge progress should be reset to 0.

Answer 9

There are solutions here to prevent robo-reviewers and to create an atmosphere of fairer processing. My solution is intended to complement those and address the primary issue here: how to add more items to the First Posts queue in order to prevent it from being so empty (emphasis based on the questioner's first paragraph).

Implemented in addition to other solutions here, which are focused on the consumers, my solution is to add more First Posts in general by Broadening the Qualifications of the First Post.

Flag any post (question or answer) as a First Post if the user hits any of the following ordered qualifications (note: italicized bold values were selected arbitrarily):

1. had at least 1 post sitting in the review queue within the last 24 hours
2. registered less than 1 month ago (optional: on all SE sites)
3. has less than 50 reputation
   1. must not be single-sourced reputation (I registered my e-mail address; +50!)
   2. reputation preferably determined using only post-based reputation
   3. optionally may be total post-based reputation across all SE sites
4. has less than 5 posts total
5. whatever other rules qualify a First Post not already trumped by prior listed rules

Answer 10

One problem with reviewing is that all reviews are treated the same with no grading of reviews. Contrast this with edits, where there is a two stage process. People can suggest edits but can't edit on their own until they have more reputation. Perhaps reviews should also have a multiple step process.

Someone who has a thousand helpful reviews (a flag marked helpful; an upvoted comment; an accepted edit; etc.) and no misses (where a miss is defined as clicking No Action Needed on a post that was later addressed) would have their review processed immediately. Someone who had three misses in ten reviews would get ignored; the post would remain in the queue as if that person had hit skip. Someone who had two misses and a hundred other reviews would get reviewed again; if two more people with similar histories concur, the review would come out of the queue; if there is a disagreement, it could get flagged for moderator attention (or put in a higher level queue).

The point of all this is to increase the chances that harmful posts will get flagged and weak posts will get feedback. It will also increase the number of posts in the queue (since many posts will get multiple reviews). For high quality reviewers, things would stay the same. Newer and lower quality reviewers would get more help.

All numbers are examples -- someone should analyze the data to see what numbers make sense.

It's possible that an even simpler rule would work. For example, instead of stages of good reviews, simply add the total helpful reviews of all agreeing reviewers and subtract something for misses. This would allow consensuses of less experienced reviewers to match a single more experience reviewer.

Note that both systems rely on misses having some impact. These systems only give misses an impact on the review system. An alternative would be to give misses an impact on reputation (say -20 for not flagging or skipping). That would also make it harder for people to review carelessly. I think that this is less effective though. Someone can get high reputation by asking and/or answering questions well in a single way. A reviewer needs to respect multiple ways of posting. They're different skills and I believe that they are better treated as such.