First Post / Late Answer Robo-Review Solutions

Question

I would like to suggest a couple changes in the First Post/Late Answer review process.

1. Rework the Reviewer and Steward badges to something more like this:

Completed at least 250/1,000 constructive* review tasks. (*No Action Needed Not Included) This badge is awarded once per review type.

or as Animuson suggested:

Why not go back to the requirements the Reviewer badge used to be? Reviewed x posts, with at least y posts actioned. – animuson♦

As mentioned in other posts on this problem, robo-reviewers tend to just hit the no action needed button to try and fly through as quickly as possible.

My understanding of badges, is that they are used to encourage behaviors that benefit the community.

When does "No Action Needed" really benefit the community?
_{Note: I'm not counting not creating additional problems as a benefit...}

• If a post is good up-vote it.
• If a post is bad: down-vote, edit, leave a comment, and or flag.
• If you're not sure: click Skip (Not no action needed)

To better illustrate my point a gratuitous freehand circle-

I know there are cases when there doesn't seem to be any problem with a post, but you don't feel it warrants an up-vote, hence the No Action Needed button, but
should we be handing out badges to people who just hit the button 250 or 1,000 times?

---

**2.** Set up an automated flag to trip whenever someone uses the same review response X number of times in a row.

As in:

• Up-voted in review X times in a row: Flag
• Down-voted in review X times in a row: Flag
• No Action Needed in review X times in a row: Flag
• And so on...

Of course we wouldn't want to auto-ban people who tripped a flag in this way, there is an outside chance that someone really did run into 5 or 10 posts of the same quality in a row, but as its unlikely, it would be an easy way to spot troublesome review patterns and bring them to a moderator's attention.

If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing?

Sorry, if you've seen me bring this up already, here, and here.
Just thought it was time to try for a feature request.

Answer 1

No.

(Although I see where you're coming from.)

The "No Action Needed" button was introduced in the first place partly because the behavior you want to encourage was itself overdone to a troublesome extent: "The current review system encourages fake reviews; some people upvote everything rather than actually fixing problems"

If someone really wants that badge and is just gaming the queue, I would much, much, much rather that he or she do so by clicking "No Action Needed" than by upvoting crappy content, downvoting good content, or making stupid edits. Let the badge jockeys have their patch of pixels (or get caught by the audits, the other thing that was created in response to this problem). This button is a defense against them causing real damage to the content, which is the most important part of these sites.

Aside from that, sometimes there really, truly is no action needed, and if you reach that conclusion after a real review of the post, that should definitely count as a review. "No action" does benefit the community: there are still eyballs on the posts to catch the times when action is called for, and sometimes the right thing to do is nothing.

Answer 2

Focusing on your "flagging" idea - those criteria would not make for very useful flags, and it would probably just fill the queue with the equivalent of VLQ flags. Someone clicking the same option multiple times in a row in that queue does not necessarily mean that they are a bad reviewer, and we'd be getting way too many false positives.

Instead, I'd like to see an automatic flag for users who consistently use an action opposite of the rest of the community. In the instance of the First Posts and Late Answers queues, this is not possible because a single No Aciton Needed vote will end review for that item, but hopefully it will require more reviewers at some point.

If that were to be implemented, we'd then have something to compare their activity against, rather than just blindly assuming they aren't reviewing correctly. The flag could also be expanded to cover all the queues. For example:

• In the First Posts and Late Answers queue, someone completing the item as Reviewed (having taken some action) would counter one or two users who had selected No Action Needed.

• In the Close Votes queue, a person who chose to vote to close when three other people left it open, or one or two people opting to leave it open when three or four chose to close. The opposite could be applied to the Reopen Votes queue.

• In the Low Quality Posts queue, a person saying that it Looks Good when two or more other users had already recommended deletion of the post. Also, keep track of users who said something looks good but then later got deleted by a moderator.

• In the Suggested Edits queue, a person who rejects an edit that then gets approved, or a person who approves an edit that then gets rejected.

---

Basically, keep track of whenever a user takes the opposite action of the majority of the community. If a user consistently takes the opposite action of many other users, then generate a flag so that moderators can review their history and make sure they're using the review queues properly.

This could potentially catch some of those users who are consistently the only person to approve or reject many edits in a row, or who consistently say a low quality post looks good when it had three or four Recommend Deletion votes on it already.

This would obviously be a fairly complicated algorithm similar and would probably remain secretive similar to the post ban algorithm, but I think being notified of users who are always choosing the seemingly wrong actions would be useful in the long-run.