I would like to suggest a couple changes in the First Post/Late Answer review process.

1. Rework the Reviewer and Steward badges to something more like this:

Completed at least 250/1,000 constructive* review tasks. (*No Action Needed Not Included) This badge is awarded once per review type.

or as Animuson suggested:

Why not go back to the requirements the Reviewer badge used to be? Reviewed x posts, with at least y posts actioned. – animuson♦

As mentioned in other posts on this problem, robo-reviewers tend to just hit the no action needed button to try and fly through as quickly as possible.

My understanding of badges, is that they are used to encourage behaviors that benefit the community.

When does "No Action Needed" really benefit the community?
Note: I'm not counting not creating additional problems as a benefit...

  • If a post is good up-vote it.
  • If a post is bad: down-vote, edit, leave a comment, and or flag.
  • If you're not sure: click Skip (Not no action needed)

To better illustrate my point a gratuitous freehand circle- enter image description here

I know there are cases when there doesn't seem to be any problem with a post, but you don't feel it warrants an up-vote, hence the No Action Needed button, but
should we be handing out badges to people who just hit the button 250 or 1,000 times?

**2.** Set up an automated flag to trip whenever someone uses the same review response X number of times in a row.

As in:

  • Up-voted in review X times in a row: Flag
  • Down-voted in review X times in a row: Flag
  • No Action Needed in review X times in a row: Flag
  • And so on...

Of course we wouldn't want to auto-ban people who tripped a flag in this way, there is an outside chance that someone really did run into 5 or 10 posts of the same quality in a row, but as its unlikely, it would be an easy way to spot troublesome review patterns and bring them to a moderator's attention.

If we can catch things like vote fraud with an automated process, why not try something similar with robo-reviewing?

Sorry, if you've seen me bring this up already, here, and here.
Just thought it was time to try for a feature request.

  • 4
    Why not go back to the requirements the Reviewer badge used to be? Reviewed x posts, with at least y posts actioned.
    – animuson StaffMod
    Jul 21 '13 at 4:24
  • @animuson that sounds like a good idea, I wasn't aware that had been the case.
    – apaul
    Jul 21 '13 at 4:27
  • @animuson why was it changed?
    – apaul
    Jul 21 '13 at 4:28
  • They changed it when they implemented the new review queue because the stats were tracked differently. Previously the stats were all combined from all the queues. Presumably the "actioned" requirement was dropped because when the new review queues were originally created, there wasn't a No Action Needed button. One had to take action or skip it altogether.
    – animuson StaffMod
    Jul 21 '13 at 4:30
  • @animuson any thoughts on the 2nd point? I'd like to here a moderator's view
    – apaul
    Jul 21 '13 at 4:33
  • I think simply upvoting and simply downvoting are also variants of robo-reviewing pattern so I don't think idea #1 is that effective. Idea #2 seems workable actually. Also there was a feature request which proposed multiple reviewers for each reviewed post but I don't know what happened to that. Jul 21 '13 at 4:38
  • @doubleDown I think the multiple review idea was posted on: meta.stackexchange.com/q/170586/217863. I'm having second thoughts on point 1 as well, I think I may steal animuson's idea
    – apaul
    Jul 21 '13 at 4:42
  • 2
    This would reinforce that acting on a post is considered good, irrespective of actual need. While that may lead to more improvements being made to posts, a large portion will be trivial unneeded "improvements". These will end up ballooning the Suggested Edits queue.
    – asheeshr
    Jul 21 '13 at 4:47
  • @AsheeshR I've seen cases of the kind of "improvements" you're talking about. I reject them in the edit review regularly.
    – apaul
    Jul 21 '13 at 4:50
  • Exactly, so solving one problem will just cause another. And there are less people active on the Suggested Edits queue than First Posts/Late Answers.
    – asheeshr
    Jul 21 '13 at 4:57
  • @AsheeshR I think measures like these need to be a small part of a larger reworking of our review system, I agree that the trivial "improvements" are a problem, hopefully too many "looks good" actions in the Suggested Edit Review could also be covered by an automated flag
    – apaul
    Jul 21 '13 at 4:58
  • 1
    "robo-reviewers tend to just hit..." -- solution for this would be to increase delay for “action” review buttons. Feature request on it says status completed but it's a lie; it's only half done
    – gnat
    Jul 21 '13 at 8:29


(Although I see where you're coming from.)

The "No Action Needed" button was introduced in the first place partly because the behavior you want to encourage was itself overdone to a troublesome extent: "The current review system encourages fake reviews; some people upvote everything rather than actually fixing problems"

If someone really wants that badge and is just gaming the queue, I would much, much, much rather that he or she do so by clicking "No Action Needed" than by upvoting crappy content, downvoting good content, or making stupid edits. Let the badge jockeys have their patch of pixels (or get caught by the audits, the other thing that was created in response to this problem). This button is a defense against them causing real damage to the content, which is the most important part of these sites.

Aside from that, sometimes there really, truly is no action needed, and if you reach that conclusion after a real review of the post, that should definitely count as a review. "No action" does benefit the community: there are still eyballs on the posts to catch the times when action is called for, and sometimes the right thing to do is nothing.

  • I thought about that a bit, that's why I suggested adding an auto flag system to prevent robo-reviewers from being able to just switch tactics and push a different button.
    – apaul
    Jul 21 '13 at 5:29
  • That seems like a separate feature, and I don't object to it as strongly, but it sounds like more trouble than it's worth to me right now.
    – jscs
    Jul 21 '13 at 5:30
  • Would it be more appropriate to separate the two ideas and post them as separate requests?
    – apaul
    Jul 21 '13 at 5:32
  • It seems that way to me, but it's your choice.
    – jscs
    Jul 21 '13 at 5:33
  • If the problem is robo-reviewing for badges, isn't the solution making the badges obsolete? Jul 21 '13 at 6:35

Focusing on your "flagging" idea - those criteria would not make for very useful flags, and it would probably just fill the queue with the equivalent of VLQ flags. Someone clicking the same option multiple times in a row in that queue does not necessarily mean that they are a bad reviewer, and we'd be getting way too many false positives.

Instead, I'd like to see an automatic flag for users who consistently use an action opposite of the rest of the community. In the instance of the First Posts and Late Answers queues, this is not possible because a single No Aciton Needed vote will end review for that item, but hopefully it will require more reviewers at some point.

If that were to be implemented, we'd then have something to compare their activity against, rather than just blindly assuming they aren't reviewing correctly. The flag could also be expanded to cover all the queues. For example:

  • In the First Posts and Late Answers queue, someone completing the item as Reviewed (having taken some action) would counter one or two users who had selected No Action Needed.

  • In the Close Votes queue, a person who chose to vote to close when three other people left it open, or one or two people opting to leave it open when three or four chose to close. The opposite could be applied to the Reopen Votes queue.

  • In the Low Quality Posts queue, a person saying that it Looks Good when two or more other users had already recommended deletion of the post. Also, keep track of users who said something looks good but then later got deleted by a moderator.

  • In the Suggested Edits queue, a person who rejects an edit that then gets approved, or a person who approves an edit that then gets rejected.

Basically, keep track of whenever a user takes the opposite action of the majority of the community. If a user consistently takes the opposite action of many other users, then generate a flag so that moderators can review their history and make sure they're using the review queues properly.

This could potentially catch some of those users who are consistently the only person to approve or reject many edits in a row, or who consistently say a low quality post looks good when it had three or four Recommend Deletion votes on it already.

This would obviously be a fairly complicated algorithm similar and would probably remain secretive similar to the post ban algorithm, but I think being notified of users who are always choosing the seemingly wrong actions would be useful in the long-run.

  • Great points, perhaps this would make for a much better feature request. If an algorithm could be reasonably designed to sniff out consistent opposite actions it would be much more effective than looking for repetition or patterns.
    – apaul
    Jul 21 '13 at 20:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .