21

First and foremoest: this topic has nothing to do with the existing audit system. It detects people blindly voting, it does that well, nobody disputes that. This topic is about detecting, warning and correcting moderators that continuously and structurally cast votes that disagree with final consensus on the review in question.

Been active for about a month now, got my 3k reputation so I can see all the review queues, and been busy at work taking my responsibility to do the community part of moderation. While at this, I've been stumped a few times already at where I saw ridiculous edits being approved, and good questions being closed without real good reason, for weird reasons just because the post had been flagged, and the flag blindly followed.

I decided to dive into this further when I just noticed the handling of this particular suggested edit. Notably, the moderation system actually worked fine in this case — it's a prime example of a bad edit, adding more information in a link that without a doubt should have been a comment, and it got rejected for exactly that reason. More notably however, 2 out of 5 reviewers actually approved this edit that is so obviously out of line. And most notably, one of the 2 approvers "has approved 337 edit suggestions and rejected 8 edit suggestions". And that's where I suspect we're having a problem that isn't fixing itself.

I've dived into this before, and noticed that most people that actually do a proper job reviewing have a reject rate of between 25% and 50%, like myself. The specific person I'm talking about has rejected 8 out of 345 suggested edits, or about 2.5%. Now that should not only raise some red flags automatically in the system methinks, it should've done that 250 edits ago - even 8 out of 100 would've been an exceptionally low ratio.

Looking up relevant discussion history here on meta found me this topic on forcing some kind of balance between approvals and rejections, and this one on robo-approvers killing review quality. The first has as accepted answer that the 'gotchas' in the queues should be able to fix this, for which I've seen too many examples already to be sure they don't (apparently they read just well enough to catch the gotchas, and approve nearly everything else), and the second boils down to a different problem that was fixed, but doesn't fix this.

I consider this to be a real problem — it's a serious defect in the review system if people can keep this up, and multiply because they aren't being corrected, since at some point there will be enough robo-approvers to become statistically relevant, and they will accidentally start hitting enough posts together to allow a lot of bad moderation decisions to slip through. The sample post I linked demonstrates this — it boiled down to two that didn't actually read the edit the properly, vs. three that did. That's very close to being the other way around.

So, I definitely think the system should analyze reviewing statistics, and keep track of when people:

  • Have an approve/reject ratio that is way outside the site-wide normalized range
  • Make a statistically relevant number of decisions that do not agree with the eventual outcome of the review

In both cases, if a flag is raised, the user should be put on review alert (with notification of this) where moderators (and/or for example 10k+ rep users) should review (some of) their decisions, until it is either decided that the user will not learn and have his reviewing privileges revoked or downscaled, or he has improved his moderation quality sufficiently to remove the flag.

Let it be said that I'm all for differing opinions, that's what makes a democratic moderation system work, but if someone disagrees with the rest of the system 90+% of the time one has to wonder whether he should be part of the system and be allowed to continue 'sabotaging' it.

Improve this question
20
  • 19
    I'm sorry - I did not read to the end...
    – juergen d
    Commented May 12, 2013 at 18:52
  • 3
    meta.stackexchange.com/questions/157121/…
    – Oded StaffMod
    Commented May 12, 2013 at 18:53
  • I don't know what is automatically detected or not, but if you see a particularly problematic case, flag for moderator attention.
    – Bart
    Commented May 12, 2013 at 18:53
  • 2
    It may also be the case that certain people are more likely to take any action on an edit if it is to approve. They may have reviewed 500 proposed edits that they neither approved nor rejected. I think it's dangerous to rely on ratio metrics unless the full gamut of activity is available.
    – Aaron Bertrand Staff
    Commented May 12, 2013 at 18:55
  • @Oded as said in my post, the gotchas appear to be 'so obviously wrong' that the robo-approvers manage to evade them properly, and just approve everything that looks 'kinda right'. Otherwise the user in question wouldn't have been able to get to casting 345 votes.
    – Niels Keurentjes
    Commented May 12, 2013 at 18:55
  • @AaronBertrand that's why I don't propose an automated sanction, just a peer review if someone appears to be 'operating his own criteria structurally instead of the generally accepted ones'. Also, the self-analysis could obviously factor in skipped reviews, I assume those are also being counted.
    – Niels Keurentjes
    Commented May 12, 2013 at 18:59
  • That's the thing, I'm not sure where a peer could determine how many reviews were skipped, or even if the system currently keeps track of that. Similarly it is possible for a user to up-vote a lot of stuff they find useful, but be extremely cautious about down-voting. Does this mean they are failing as a community contributor? The ratio isn't necessarily what's important, IMHO, but rather the quality of the actions themselves.
    – Aaron Bertrand Staff
    Commented May 12, 2013 at 19:02
  • Well and that's why the ratio should just be a guideline for initiating a review of his 'conflicting decisions', and if the user is then 'cleared' because the quality is actually good - all is fine and he won't be flagged again anywhere soon.
    – Niels Keurentjes
    Commented May 12, 2013 at 19:10
  • @Oded The thing is that you can avoid all audits in the suggested edit queue by refreshing the page ! meta.stackexchange.com/questions/175911
    – Hugo Dozois
    Commented May 12, 2013 at 20:03
  • also, rejecting when 3 others approve, or approving when 3 others reject, is suspicious
    – Doorknob
    Commented May 12, 2013 at 20:26
  • 4
    @Doorknob if it happens a statistically relevant number of times - disagreement belongs in the democratic system, but structural continuous disagreement means you don't understand the policies or the intentions.
    – Niels Keurentjes
    Commented May 12, 2013 at 20:28
  • I agree some statistical measures of outliers could be applied here, e.g. looking at quantiles. For the record, we (moderators) do have advanced statistics about what the reviewers do, and we can manually check their behavior—if something makes us check, that is. But this part of the system should definitely be improved.
    – slhck
    Commented May 12, 2013 at 21:30
  • 3
    Funny that a topic about 'reviewers not reading all that well before casting their votes' has already garnered 4 close votes because it would be a duplicate of a completely unrelated topic. This has nothing to do with the audit system that is in place to detect people voting without reading, this is about people structurally casting votes that do not agree with final consensus on the decision, or are far too lenient or strict in general.
    – Niels Keurentjes
    Commented May 12, 2013 at 21:46
  • 2
    @Doorknob That is a totally incorrect assumption to make. I've frequently rejected something only to have 3 robo-reviewers approve it. Fortunately I monitor for this sort of thing and I can go back and make the manual edit (or sometimes rollback) myself. You could ask what makes me think my review was right and the others were wrong? My answer is that I am more likely to be right than three fresh 2K users who are gunning for badges.
    – slugster
    Commented May 12, 2013 at 23:43
  • 2
    Am I the only one to sense the irony in this post being peer-closed and re-opened?...
    – nickhar
    Commented May 12, 2013 at 23:50

3 Answers 3

Reset to default
5

The only automatic system against bad reviewing is the review audits. Failing to pass a number of audits will result in an automatic short time suspension from reviewing.

Other than that, moderators recently got access to reviewing statistics for all queues and the ability to manually suspend users from reviewing:

  • Moderator newsletter, March 2013

    Review Queue – Advanced Statistics

    We have made a more in-depth set of review statistics available to moderators. Each queue now includes the number of times a reviewer has picked each available action, and what percent of their total actions it represents. This should provide you with a much better overview of how your community is using review, as well as a way to detect those who seem to be reviewing in bad faith and may need to be temporarily blocked (see below).

    ...

    Suspending Users from Reviewing

    We have implemented a system for blocking users from using the review queues for 1 to 30 days.

    ...

    This page also includes a list of user who are currently banned, either by a moderator or due to failing audits. This functionality is available regardless of whether your site has audits enabled.

  • Moderator newsletter, April 2013

    New Audit Stats Available with Advanced Review Stats

    We have added some new statistics for your studying pleasure to the Review Audit Stats we introduced in last month’s issue.

    ...

    The following new statistics are available on this page in each queue:

    • AvgDuration(s) notes the average time that the user spends on each task in that queue in seconds;
    • AuditTotal notes the total number of audit tasks that the user has run into in that particular queue;
    • AuditPass% notes what percentage of audit tasks that the user has successfully completed compared to all tasks run into within that particular queue.

On the smaller sites, these tools haven't been around long enough for meaningful stats to accumulate. On the original trilogy, however, I imagine the moderators already have a very good idea of who is repeatedly abusing the review system and I wouldn't be surprised if a few review suspensions have already been triggered.

Improve this answer
2
  • 1
    Thanks @Yannis, interesting information, especially this: "Each queue now includes the number of times a reviewer has picked each available action, and what percent of their total actions it represents." If those stats can be calculated, it should also be possible to present the moderators with 'top lists' based on them, and their ratios. Might I suggest also adding the 'controversial decisions' stats for individual users, listing their recent review votes that didn't match final decision? I would even be very interested in seeing that on my own profile for myself as reference.
    – Niels Keurentjes
    Commented May 13, 2013 at 0:52
  • Same for me - useful to know and understand even if it's just provided for mods to assess.
    – nickhar
    Commented May 13, 2013 at 0:57
4

I'm broadly in agreement here and largely because I've been active over the past few months and done a bit of (ok, a fair amount of) queue-busting...

The only passage in your post I'm pondering over is:

I've dived into this before, and noticed that most people that actually do a proper job reviewing have a reject rate of between 25% and 50%, like myself.

That's quite a bold ratio and one that can't possibly take wider factors into account. Whilst it's possible (and reasonable) to look at the ratio of approvals/rejections, I don't think it's a ratio thats solid, why?:

  • Good/bad edits aren't created in equal numbers or over time.
  • Not all users regularly visit the queue, when they do visit they may approve 5 edits and reject none and leave.
  • Some users don't vote with equal weight and use the skip button when they feel like it.
  • It concerns human behaviour (in all it's unique/subjective glory!)

I do however, support the idea that bad calls are worth looking at and potentially considered (system-wise) - where a user is on the failed side of a:

  • Rejection when 3 others approve
  • Approval when 3 others reject

Theoretically, this could be used in assessing a users worthiness to review edits (much like the audit reviews used in the close-vote and other queues); something that would help mods and the wider community prevent the flow of poor approvals.

EDIT:

It's clear from the resultant comments that this is a subjective issue because different users use the queues in sometimes subtly different ways.... Something born out of individual technique. What do I mean? - As a personal example, I:

  • Visit the queues daily (for my sins)
  • I don't filter tags, so effectively drink from the fire-hose.
  • frequently skip those I can't answer or don't have the technical merit to assess
  • Correct/edit (Approve/Reject) obvious blunders in all tags
  • Approve/reject edits forcefully in the tags I'm experienced with

As a result, any solution is going to have to be cunningly designed to thwart the robo-reviewers and badge-burglars that frequent the queues and that are likely to do so in the future as well as allow enough flexibility to contribute in their own way.

In the meantime, and that's been pointed out in the comments, those with enough rep do have the manual ability to:

  • Edit questions/answers manually
  • Roll back shoddy/robo edits

So not all is lost whilst the system and solution evolves (through community views and the SE team).

Improve this answer
16
  • The reject ratio mentioned is a rough indication from what I've observed on a small subset of reviews 'backchecked' to see whether my own votes agreed with eventual consensus, I have no idea whether the 25-50% bandwidth really makes sense, it might also be 15-60% if you look at 99% of moderators. It's the other 1% that reject 2% or 95% that warrant looking at.
    – Niels Keurentjes
    Commented May 12, 2013 at 21:39
  • @Niels It could be an influencing factor, but I like the general idea! :)
    – nickhar
    Commented May 12, 2013 at 21:40
  • @Niels Honestly, I think I've hit skip far more than any other button in any queue... I don't know if those clicks are tracked!?
    – nickhar
    Commented May 12, 2013 at 21:46
  • me neither, and if they aren't it means the reject ratio is probably indeed a bad candidate for initiating automated peer reviews. Still means the other part of my suggestion (peer reviewing people that frequently cast votes that do not comply with eventual consensus) is possible.
    – Niels Keurentjes
    Commented May 12, 2013 at 21:48
  • Just something I've noticed on multiple occasions - sometimes the exception is the right call, while the three that you're deeming were "right" were just dog-piling.
    – Aaron Bertrand Staff
    Commented May 12, 2013 at 22:01
  • @AaronBertrand Indeed. Seen it a few times too.
    – nickhar
    Commented May 12, 2013 at 22:06
  • 2
    I want to nitpick a couple of your points: 1. approve/reject rates can be averaged over time. 2. People who avoid making hard or negative reviews by using the Skip button are bad reviewers - negative reviews need to be made, if they don't/can't make the call they should get out of the review queue. Sure it's okay to use Skip if you cannot gauge the technical accuracy of an edit (I use it myself) but if someone is using it too often then they shouldn't be reviewing. People should not be reviewing only the low hanging fruit!
    – slugster
    Commented May 12, 2013 at 22:26
  • 2
    @slugster Nitpicking is good! Yes, agree that they can be averaged over time. What I mean by skip is 'not within my field of experience' rather than avoiding decisions, but I still like to see them looking for complete clangers which need to be pulled up - but I didn't make this clear enough; I'm remarkably happy negatively reviewing edits/posts and so should others. I completely agree on the low-hanging fruit too.
    – nickhar
    Commented May 12, 2013 at 22:34
  • Come to think of it, I'm pretty sure that 'skips' must also be tracked, since otherwise the system would eventually keep confronting you with the same questions to vote on. So obviously it will have to keep track of the ones you skipped as well.
    – Niels Keurentjes
    Commented May 12, 2013 at 22:56
  • @Niels I think the system already tracks Skip so that you don't get presented with the same review again, but I agree with you that they need to be aggregated as part of the stats. Personally I only use Skip for duplicates or wiki edits where I don't have the technical expertise to make a decision, something like that should be tracked and used as part of the decision process for whether people should be review banned or not.
    – slugster
    Commented May 12, 2013 at 23:23
  • 1
    I tried constructing some queries on the data subsite but it would appear all possibly sensitive data such as reviewing behaviour is not published, pity but understandable. I'm very curious if a developer could produce some real world statistics on common accept/skip/reject ratios, and whether it's possible to easily query for users whose reviewing decisions 'frequently disagree' with the final result of a review.
    – Niels Keurentjes
    Commented May 12, 2013 at 23:33
  • @slugster What is wrong with reviewing the "low hanging fruit", as you call it?
    – user200500
    Commented May 13, 2013 at 0:01
  • @Asad while those edits do have to be reviewed like any other, if you want the shiny badges you have to do the work, don't just pick off the easy stuff. Not everyone has the technical expertise to make judgement calls on some questions, but when that lack of expertise or motivation stretches to most questions then you have to ask why the person is even in the review queue. We could argue this all day, but i dislike lazy and robo reviewers - if people are only targeting the easy questions then they should get out and let genuine reviewers do it.
    – slugster
    Commented May 13, 2013 at 0:08
  • @Asad And before you ask "but then how do people learn?", you learn by doing, not by avoiding. It comes down to people's motivations for being in the queue - is it just to get a badge, or is it to make a difference and help with the quality of the site?
    – slugster
    Commented May 13, 2013 at 0:10
  • 1
    In the grand scheme of things, it doesn't matter if a correct suggested edit was approved by a "genuine reviewer" or one of these lazy layabouts that keep snapping up your easy reviews. As with answering "low hanging fruit" questions, the motivations of the user do not matter, as long as there is a benefit to the site.
    – user200500
    Commented May 13, 2013 at 0:13
3

I don't think the approve/reject ratio has any meaning in this case.

Most of my reviews are approved (though there are far less of them than in the example you provided). That's not due to the fact that I blindly approve everything. I read each suggested edit I review carefully.

When I see an edit that looks valid, I approve it. When I see an edit I'm not sure of, I usually skip it and let other people decide. I only reject edits on rare occasions in which I'm sure the suggested edit is bad.

Improve this answer
7
  • I just looked at your history and your last page already shows 2 rejections. When I say a 'normal' ratio appears to be within 25 to 50 percent, I'm inviting the site admins to dive into the database and correct the number - I'm just guessing from a limited set of samples. My point is about a 2% approval rate being obviously 'suspicious' at the very least, especially if a significant number of those approvals are 'close calls'. You won't run into the latter if skip potentially conflicting votes anyway.
    – Niels Keurentjes
    Commented May 12, 2013 at 19:45
  • If you arrive at those numbers, and only a small number of your decisions were 'controversial', the system shouldn't flag you. If you get to those numbers by placing 100 votes that weren't the eventual result, your voting behaviour should be inspected. I should've been shorter about making that point I suppose, rather ironic that @Gnat votes to close this question as a duplicate of an unrelated issue :P
    – Niels Keurentjes
    Commented May 12, 2013 at 19:57
  • 4
    Sorry, but I've down voted you based on the statement I only reject edits on rare occasions. As per my other comment, reviewers should not be concentrating on the low hanging fruit, IMVHO this makes you not much better than a robo reviewer. Bad edits need to be rejected. I understand you won't be able to make a decision on every bad edit you see, but avoiding negative decisions and taking mainly positive ones isn't a totally helpful way to review no matter how good it makes you feel about what you're doing.
    – slugster
    Commented May 12, 2013 at 22:30
  • 2
    @slugster I don't know what you're talking about. It makes you the exact opposite of a robo-reviewer, who will never consider the Skip button at all. Perhaps it isn't as helpful as making informed decisions on both good and bad edits, but withholding judgement when in doubt is infinitely superior to just blindly approving or rejecting everything. I don't see how you could compare those two behaviors at all.
    – user200500
    Commented May 12, 2013 at 23:54
  • 1
    @Asad My comment is not about using the Skip, it's about using it to avoid any negative review actions. We don't want only good edits approved, we also want bad edits rejected, these are equally or possibly even more important. Robo-reviewers are bad, but those that only choose the easy reviews are almost as bad, no matter how much they think they are helping. Hitting Approve and Skip is easy - people need to learn how to hit Reject as well.
    – slugster
    Commented May 13, 2013 at 0:00
  • 2
    @slugster Sure, learning to use Reject is a good thing. But to equate skipping a review with an actively harmful practice is just wrong-headed. If someone skips a review, it's still there in the queue for those that care to make the "tough decisions". In another place you claimed that if a person can't make tough decisions, they should get out of the review queue. Why? What benefit is there to disqualifying users from providing what small measure of help they can?
    – user200500
    Commented May 13, 2013 at 0:03
  • @Asad you're taking a very liberal and inclusive approach, which is okay - but I'm arguing something slightly different to that. There is a reasonable amount of behavior detection built into the system, and I'm arguing that those who hit the Skip button excessively and almost never hit Reject - IOW the outliers from normal review behavior - should be tracked along with suspected robo reviewers. Sure they may help a little, but do they fit in with the desired community behavior over all? They wouldn't necessarily be banned or anything, but at least they will be identified.
    – slugster
    Commented May 13, 2013 at 0:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .