9

Say we have this review:

This is an obviously bad review here, as the post should have at least been met with a flag/comment, as this is clearly NAA.

My question boils down to the following:

  • Should this be something that should be flagged?
  • To ♦ moderators: If I do report this, what are the repercussions from this? Would this be worthy, on the first mistake? Is this something that would you want to see and track from the first mistake, or something that should be reported/tracked after n or more mistakes? If so, what is n?
Improve this question
0

2 Answers 2

Reset to default
7

For unambiguously bad reviews that happened recently, as a moderator I see value in having these brought to my attention. For "unambiguously bad", I mean No Action Needed reviews on the following answer types:

  • Clear spam
  • Follow-on questions left as answers
  • "Me too" comments that have no additional information
  • Complete gibberish

We'd want these to be recent reviews (within the last few days), because often someone who was reviewing that poorly will be caught by the system eventually. It doesn't help us much to be made aware of a bad review someone left a couple of months ago, if they were review-banned in the interim. Only if a non-answer or spam was allowed to survive would a flag on an old bad review be useful.

The point of a flag like this would be to catch terrible or abusive reviewers much earlier and prevent damage from being done. Prioritizing recent reviews works toward that.

Knowing about single obviously terrible review can be useful, and if you had multiple instances of recent bad reviews (2 or more), that's definitely worth a flag. We regularly field flags about people leaving multiple bad recent reviews, and that helps us catch those who have evaded audits or who otherwise fell through the cracks. We also get flagged about individual terrible reviews (approving spam, etc.) and many of those have brought larger problems to our attention.

Improve this answer
8
  • My plan was to use this only for recent reviews. Can you provide me a format that you would prefer for the flags? Something like "[ SpotDetector ] Confirmed NAA was reported No Action Needed in First Post queue."
    – Blue
    Oct 29, 2018 at 18:25
  • @FrankerZ - For formatting, a direct link to the review or reviews is the most helpful. If there are multiple bad reviews, put them all in the same flag and call out each bad review ("Multiple recent bad reviews: [link1], [link2]", etc.).
    – Brad Larson
    Oct 29, 2018 at 20:06
  • The first post queue is a link (In markdown) that goes straight to the queue.
    – Blue
    Oct 29, 2018 at 20:08
  • It's unlikely you would ever find this with an old post anyways, no? I assume the most apparent way to see this would be the review queue, and AFAIK things don't stay in that for months.
    – JMac
    Oct 29, 2018 at 20:24
  • @BradLarson I just ran a script, and fetched the last 3 days worth of bad reviews. 26 reports in total, so ~ 8ish a day. Can you have a look through those, and tell me your thoughts, if these should be auto flagged? If some of these don't look actionable, maybe this indicates a question was also incorrectly deleted, so perhaps there is positive merit there? Where should I go from here?
    – Blue
    Oct 29, 2018 at 22:11
  • @FrankerZ - I see at least six of those that were debatable, and I wouldn't have acted on a reviewer based on those. People disagree on link-based answers, and several of those had content aside from the link, so I'd rule those out. They tended to be grouped around the low query scores, so maybe adjusting the scoring to not highlight those and focus on the follow-on questions and "me too" non-answers would reduce the number of those in the results.
    – Brad Larson
    Oct 30, 2018 at 15:27
  • 1
    @BradLarson I've created a bot in the Workshop Room that posts when it detects a possible bad review. I need feedback, especially from the diamond moderators, if they feel like they would take action. All the info on how to provide feedback can be found here. If you could help me over the next few days, and just give me an idea of what you would find actionable, it would greatly help me come up with a filter that would be acceptable for auto-reporting.
    – Blue
    Oct 31, 2018 at 3:35
  • 1
    After a nice long chat with Jon Clements, I've actually pivoted a bit (It seemed that it wasn't very helpful for one off reviews, or putting in the manpower to come up with a tp/fp system for people to dig in and review manually). Now, you can simply just ask for a report. The script just checks for any deletions, and you can filter/dim down search results. (ie. @SpotDetector 3d 5 will show you all users who have reviewed as 'No Action Needed' on 5 or more deleted posts within the last 3 days.)
    – Blue
    Nov 3, 2018 at 14:27
2

The "No Action Needed" button primarily exists to stop robo-reviewers from upvoting the answer instead. Thankfully, that didn't happen, and there are users (like you) who watch the review queue for bad "No Action Needed" reviews.

Should this be something that should be flagged?

I wouldn't flag based off a one-off instance like this. At first glance, the answer appears like an answer. While it is expected of reviewers to carefully look at the posts, users do make mistakes sometimes (though I would have spotted that it's not an answer after just three seconds). In this case, I'd check the user's reviews to check for a pattern of robo-reviewing, and only if such a pattern exists report it.

If I do report this moderators: What are the repercussions from this? Would this be worthy, on the first mistake? Is this something that would you want to see and track from the first mistake, or something that should be reported/tracked after n or more mistakes? If so, what is n?

If I were a moderator looking into such a flag, and looking at the user's reviews, I wouldn't ban them just for a single mistake (misclicks can happen, it's possible that the user noticed and flagged the answer later). I'd only want to track them if they show a pattern of incorrect reviews. I'd ban them if they met criteria similar to automated bans for failing audits (3-5 or more incorrect reviews at a time, with increased scrutiny shortly after coming off the ban). If they were under that pattern, but still show a propensity for incorrect reviews, I'd warn them and encourage them to pay more attention when reviewing.

Improve this answer
8
  • When you fail an audit, you get a shaky message that shows "STOP! Look and Listen", to catch robo-reviewers, but also as an attempt to educate the user on correctly reviewing posts (Starts with a warning, and then leads to review bans). Is ignoring this review the best course of action? I'm curious if putting in a system to help educate users who have made mistakes, would be worthwhile.
    – Blue
    Oct 27, 2018 at 15:39
  • 1
    In this case, the review ban system is also the warning system. It's not uncommon for a user to get a 1- or 2-day review ban just to get their attention on a bad review. Moderators generally don't use the mod message system for reviews.
    – animuson StaffMod
    Oct 27, 2018 at 16:45
  • @animuson I was referring to the chat system, not the mod-message system.
    – Sonic the Anonymous Hedgehog
    Oct 27, 2018 at 17:31
  • @FrankerZ Most sites don't have review audits. Only a few large ones do. Audits only represent a tiny proportion of reviews on the sites with audits enabled, and a single failed audit really ends up meaning some amount of bad reviews until the system audited them. My answer scales according to that, and I'd also trust that sometimes users can make accidental mistakes. In the case of FP and LA, the review doesn't have very much of an outcome other than merely dismissing the task, and it's possible for the user to later flag it.
    – Sonic the Anonymous Hedgehog
    Oct 27, 2018 at 17:35
  • 3
    @Sonic That's... still not a great way of warning someone. Setting up a private chat room just to warn someone in private about review behavior is a pretty big waste of time and effort versus just giving them a review ban. We've always encouraged moderators to just give review bans if they want to bring something to the attention of a reviewer.
    – animuson StaffMod
    Oct 27, 2018 at 19:16
  • 1
    @animuson So here's my next question. Natty currently runs in the SOBotics channel on SO. It examines NATO posts, and experienced users rate questions as TP: They're NAA, or FP: It looks fine. I wanted to piggy back on that data, pull all verified posts that were confirmed NAA, examine the related First Post/NATO review for that post, and then auto flag to a moderator if the review completed as 'No Action Needed'. What are your thoughts?
    – Blue
    Oct 28, 2018 at 12:54
  • @FrankerZ If the user upvoted the post in the queue, it will show up as "Reviewed", so robo-reviewers who upvote instead of clicking "No Action Needed" will not be caught. Also, there's always the possibility that they later flagged the post after clicking that button.
    – Sonic the Anonymous Hedgehog
    Oct 28, 2018 at 19:52
  • @sonic Unfortunately I can only work with what I'm shown. I had put in a feature request to show more information (Like Voted/Flagged/Edited/Commented), but that question was downvoted/roomba'd. I think this would be the best with what I can work with.
    – Blue
    Oct 28, 2018 at 22:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .