68

Problem

User submits a suggested edit which should be rejected (at least 3 reasons pop-up straight away...)

You happen to come across this and you want to do the right thing, which is reject the suggested edit. So you hit Reject button and select your reason. Oops...

enter image description here

So then you refresh the page and click the more link to see the reviewers and stats

enter image description here

Right, I am not saying this is not entirely possible to come across 11/35/14 suggested edits which really improved the post... but how likely is this to happen?

UPDATE

aimed at @Lightness Races in Orbit's suggestion

This is a better example I suppose

which should be clear based on the first 4 steps in the revision history

Let Me Flag

Currently, there isn't any way to actually flag these guys for moderators attention. I know I can go to their profiles/ activity tabs/ comments and leave a comment explaining how to act in the suggested edits queue... I can write a custom message under one of their comments and explain the problem to the mod...well, I am not going to do that..

These are painful ways of achieving something that should be exposed right there at the review stage. There needs to be another/simpler way to tell our mods and other already establish and trusted users that there are some potential robo-reviewers which need handling (possibly a ban1)

enter image description here

The mechanics

A suggested review flag [SRV] flag's meta could be as follows (obviously just an idea)

  • Reviewer
  • Link to the review
  • Optional: reason for flagging
  • Date/time
  • Flagger

Each user would have a counter against his profile stored in the back end. How many times his action has been flagged, and how many times mods(10K+ possibly as well) have approved such a flag (robo-alert). This number would not be exposed to users, so nobody but the diamond mods know the raw numbers/count. If the system detects, let's say, 10 approved flags the reviewer gets an automatic ban(to be discussed).

I think we could add those types of flags to the mods tools and let those 10K+ help handle those types of flags.

Any ideas are welcome as it's quite difficult for one person to invent and think throughout the entire system.

Consequences

Pros

  • slowly disposing of robo-reviewers

  • in a long run: improve the quality of suggested edits being approved

  • users awareness -> "my actions are somehow analyzed further"

  • privileges page could educate users a bit more of What do I approve/reject and consequences for being a robo-reviewer.

Cons

  • Increased load of moderation required

  • Implementation of such system (developer time, etc)

1 - I am not the one to judge when a ban is necessary. It would up to the mods when someone actually deserves a ban. It could also be dependent on the amount of approved flags a reviewer has received

Improve this question
10
  • 20
    Good idea but, IMO, user must check related user's review history to see if he had other suspicious reviews. Some users just approves the edits that they are sure and skip the rest. That will also cause a high accept and very low decline rate.
    – Mp0int
    Mar 3, 2014 at 13:03
  • 13
    Otherwise, that will just become a tool to dump tons of suspects to mods and only a few of them would be real robo-revewers.
    – Mp0int
    Mar 3, 2014 at 13:13
  • 9
    That edit is an improvement IMO. It removes the need to scroll horizontally.
    – Martin Smith
    Mar 3, 2014 at 13:30
  • 2
    @MartinSmith and it fixes other minor issues as well. But that's not the point. Point is - when someone sees a probable robo reviewer, there is no way to flag him for future evaluation.
    – Mołot
    Mar 3, 2014 at 13:42
  • 1
    @LowerClassOverflowian So? Flags described in this request would not trigger any automatic action, so no harm done to reviewer if he acted properly. And "skip" by all means is a valid default in all review queues, every other action should be taken only if reviewer is really sure what he's doing. If someone feels unsure about rejecting if there is even a remote chance edit was actually OK? Yes, he should skip anything he don't feel like approving. In the course of 10 to 20 edits it is pretty probable, especially when reviewer is new to the task.
    – Mołot
    Mar 3, 2014 at 13:53
  • Oh yes, I'd wish not only flagging reviews, but also audits. But single wrong review shouldn't flag a moderator attention, until it's really bad, for example approving spam or vandalism.
    – Danubian Sailor
    Mar 3, 2014 at 14:14
  • 1
    If we're that concerned about user's who skip being accused of robo-reviewing then start counting skips. But I bet if we could see skip counts the trend would be that more skips correlates with a higher reject ratio.
    – OGHaza
    Mar 3, 2014 at 14:53
  • @Mołot - I disagree with your comment to Martin. That very much is the point. You're suggesting that you might have flagged those users for robo-reviewing, even though the edit appears to be valid. The last thing we need is a queue mostly filled with unhelpful flags.
    – JDB
    Mar 3, 2014 at 16:29
  • @JDB not me, but the one who posted it, if you care to notice that difference. One that genuinely believed that these reviews might be robo. And if you want to avoid flags that may be used incorrectly, you should ask to remove flagging system altogether... Other kinds of flags are not always marked helpful now, are they?
    – Mołot
    Mar 3, 2014 at 19:57
  • New reviewers may be afraid to reject. Given the perceived approval rate, I conserve my votes for rejection. Only by eliminating skip could a true metric be obtained; however, there are subjects and certain edits I'm not comfortable to approve or reject to which skip functionality is imperative. I concur with this flag request, but better examples could be cited.
    – Jason Sturges
    Mar 5, 2014 at 19:18

3 Answers 3

Reset to default
29
+50

First, we should look at ways that the system could catch problematic reviewers earlier. The suggestion here would have stopped at least one of the reviewers from your example (now manually banned) before they got to this post, and would have made the other two prove that they were paying attention.

Beyond that, moderators currently have "leaderboards" of potentially problematic users of other sorts, and we have access to overall statistics for reviews in the various queues. We use these statistics to pick out terrible reviewers that otherwise weren't stopped by the audits. I go here from time to time to clean things up.

What I could see would be another kind of user list that we could browse to at any time, which would provide a more filtered view of potentially troublesome reviewers. I made a suggestion here for one way of identifying these reviewers, where if they chose "No Action Needed" or did anything but flag a post that was later flagged as spam / offensive / not an answer and deleted, this would be marked against them. We could have a sorted list of reviewers who accumulated these kinds of bad reviews, with a one-click ability to look at their review history and impose a manual review ban if needed.

Your proposed "bad review" marking system could feed into that. Users could tag individual bad reviews, but rather than those triggering an urgent moderator flag, they would appear as a mark against this reviewer in this sorted list of reviewers. These could be separate from my suggested system-applied marks for bad spam reviews (which really should be more urgent than bad suggested edit ones), but would also feed into the sorting. At a glance, we could tell when a user was racking up a bunch of these from the community (like we can see if a post is being flagged by a bunch of people), take a closer look at their review history, and ban if needed.

This wouldn't enter into the flag queue, and would be a separate area we could go to and check regularly. Like I said, we have similar views for other kinds of bad behavior, and if done right this could allow us to act on the worst reviewers fairly easily.

Improve this answer
2
  • 9
    +1 thanks for making us aware of moderators currently have "leaderboards" of potentially problematic users. If you, as a diamond mod, see potential in my idea - you see how this could fit with the already implemented ones then that is great IMO. I just threw this out there but obviously I am seeking input from community, specially mods so my original feature request can be improved or maybe even implemented at some point. Justification is needed and thank you for your answer:)
    – user221081
    Mar 3, 2014 at 15:47
  • are there any plans to implement such system then?
    – user221081
    Mar 7, 2014 at 10:40
14

I really like this idea and I really want to upvote... except that I'm pretty sure a flag like this would be heavily overused.

See, there are a lot of different opinions on what constitutes a proper edit. Sure, some are obviously bad, but many are in a sort of gray zone. By "gray zone" I mean that some people think it's perfectly OK and others think that it's completely unacceptable. It's best to have those fights here on Meta and build up some consensus. Flagging "bad" reviews is just going to pile more work on the mods (whatever class of mod you may choose to meta-review), and really... what course of action are they supposed to take? Rollback the edit and put it back into the review queue? Or are mods suddenly supposed to make final decisions on all edits? Putting mods in charge of determining what constitutes an acceptable review seems counter to the idea of community moderation.

Besides, we already have 2 "flag" capabilities, one of which you have just demonstrated... if you see a really bad edit that was approved:

  1. Flag the post for moderator attention and explain the situation.
  2. Post to meta.

The second option will likely garner moderator attention and also invite some helpful insight into whether the reviews were really that bad.

Improve this answer
5
  • if you see a really bad edit that was approved, post it on Meta. It will be taken care of. the edit or the reviewers?
    – user221081
    Mar 3, 2014 at 14:20
  • @mehow - if a mod sees your Meta post (which is likely), then that mod can take independent action on the post/users.
    – JDB
    Mar 3, 2014 at 14:26
  • 5
    As for what action to take, if a user is regularly making poor reviews, they can be review banned. That is what I see as the primary purpose of such a queue, not reversing the individual actions (which whoever is doing the flagging can already do on their own).
    – Servy
    Mar 3, 2014 at 15:46
  • @Servy - In which case, flagging the post for mod attention (and explaining the issue) should be sufficient... no need for a new flag. Sticking a flag on the review queue would encourage people to flag on reviews like the one above... without doing any investigation on their own. "Hey... I completely disagree with this edit! They must be robo-reviewing!" would probably be the norm.
    – JDB
    Mar 3, 2014 at 16:24
  • 2
    Option #3: if unsure if it's really that bad, post to the tavern.
    – John Dvorak
    Mar 3, 2014 at 16:54
3

There was nothing wrong with that edit whatsoever, which highlights the main problem with your proposal: this is all subjective. You are asking for a system that allows you to flag up reviewers as being "wrong", but this requires that you know best at all times. Not really what SO is about.

@Mołot said, in comments on this question:

@MartinSmith and it fixes other minor issues as well. But that's not the point. Point is - when someone sees a probable robo reviewer, there is no way to flag him for future evaluation.

But that's entirely the point: you've found a perfectly reasonable edit, decided that you don't like it, and come on to meta asking for a way to flag edits. Do you see the problem here? Your first example is an example of how your proposed system would be broken, spamming moderators with nonsense flags because, in this case, you are a worse reviewer than the guy you're complaining about. The potential for non-malicious abuse is extraordinary.

Improve this answer
9
  • Agreed, there was nothing wrong with this edit. Probably a bad example cited only due to all three reviewers having rejected zero reviews. There are far better examples that should have been posted.
    – Jason Sturges
    Mar 5, 2014 at 18:59
  • 6
    Perhaps they skip all reviews until they find one worthy of being Approved.
    – hayd
    Mar 5, 2014 at 19:03
  • 1
    this is not about this specific edit... whatever... have your GO...
    – user221081
    Mar 5, 2014 at 21:39
  • @mehow: Then present an actual example of the problem you're describing, because right now it's a dataless rant.
    – Lightness Races in Orbit
    Mar 5, 2014 at 23:54
  • @LightnessRacesinOrbit done. See the UPDATE section in my OP
    – user221081
    Mar 7, 2014 at 10:44
  • @mehow: It's better, but still marginal. To me there is no indication of "robo-reviewing", just stupid people not bothering to check the meaning of a tag before approving it. There is a difference between the two, because at first glance the edit would have been fine.
    – Lightness Races in Orbit
    Mar 7, 2014 at 13:54
  • well, i disagree again. "the stupid people not bothering to check the meaning" should not be allowed to review then...
    – user221081
    Mar 7, 2014 at 15:02
  • @mehow: Unfortunately life doesn't and can't work that way. Who gets to choose what is correct and what is not?
    – Lightness Races in Orbit
    Mar 7, 2014 at 15:46
  • (Note that I said "marginal"; I don't entirely disagree!)
    – Lightness Races in Orbit
    Mar 7, 2014 at 17:05

You must log in to answer this question.