3

I'm posting this just as feedback and discussion point.
Perhaps it could be used to tweak the audit system in some way.
(Plus it's amusing)

The audit (possible spam)

audit-image

The problem, & 2 questions/ideas

Ignoring Jon's fame, the following could be true for plenty of other audits:

The user has lots of reps, many many badges, etc.

(Q 1)
What about audits only using posts from users below a certain rep? Such as less than 1k.

This would perhaps stop the reviewer from seeing lots of rep/badges and just assuming it must be ok, and instead actual take time to review properly.

I know it could be said "it worked, you did your job", but a "quick" glance would lead to "Looks OK" and pass the audit.

Audits are supposed to check people are reviewing adequately enough, but perhaps quick glances are not always enough, so doesn't really confirm the user is "doing enough" in general or on other reviews if they "always" do quick glances - i.e. they were lucky on this occasion and could have been real spam.

Users doing quick/robo reviews are more likely to hit "looks ok" as it's no work.
Whereas "Edit" and "Recommend Deletion" require work and choices. "Skip" gets them no review score.

(Q 2)
So, would it be possible/useful to only have audits which are not ok? - so always need an edit or recommend deletion?

Improve this question
2
  • 4
    The flipside is that an actually LQ post by someone like Jon Skeet makes a really good audit.
    – jscs
    Commented Jun 5, 2015 at 4:10
  • @JoshCaswell Yes, that answer should have been a comment. Not sure what the audit result was as the link was to a different audit. But is it a good audit when considering robo reviewers or those not bothering to look much will likely see "Jon Skeet" and/or "700k rep" and as a result just click "Looks ok" without actually reviewing. Whereas a post from a 500 rep user is much less likely to be assumed as ok.
    – James
    Commented Jun 5, 2015 at 14:14

1 Answer 1

Reset to default
7

What about audits only using posts from users below a certain rep? Such as less than 1k.

Because users with 2K reputation never post anything bad?

I ran the numbers on this recently; reputation is a pretty good indicator of post quality, but it's hardly a 100% guarantee. Heck, I've even heard rumors of users with several thousand reputation points posting blatant spam, while non-answers, link-only answers, and such - while considerably less common among folks with experience - are far from unknown.

The idea is to check that folks are paying attention, not just blindly clicking based on simple heuristics. We can't always achieve that; there are plenty of "tells" for experienced reviewers. But "high rep users can do no wrong" would be a particularly bad heuristic to train folks on.

So, would it be possible/useful to only have audits which are not ok? - so always need an edit or recommend deletion?

Sure. That's how it works in the Suggested Edits queue, for instance. There may have been one or two complaints about folks trained to just blindly reject everything... but even short of that, there's the more subtle problem of gently encouraging folks to look for problems without looking for mitigating factors.

...y'know, I could keep rambling on all night. But examples are better. So here are a bunch of examples from the past few days of "known-good" audits based on posts from authors with > 2000 reputation.

FWIW, 35 of Jon's answers have been used as audits that reviewers have failed to identify as ok at one point or another. In some cases, multiple reviewers have failed the same audit.

Improve this answer
6
  • "and as shocking as it might be to fail a Skeet audit" you may mean BalusC here, or there's a Skeet answer only visible for 10kers?
    – Braiam
    Commented Jun 5, 2015 at 4:07
  • 2
    The audit in the OP here is from Skeet. BalusC doesn't have quite the same lofty stats, but... They're up there.
    – Shog9
    Commented Jun 5, 2015 at 4:08
  • 1
    Interesting info, thanks. "Because users with 2K reputation never post anything bad?" It was more that robo reviewers might think high rep users can do no wrong. So robo review sees high rep user and just clicks "Looks OK" and do nothing else, which is not really paying attention. But I do understand there's no perfect algorithm. Was just ideas :)
    – James
    Commented Jun 5, 2015 at 12:49
  • in another (good) answer of yours you stated "In general, you shouldn't worry about review suspensions" Perhaps on certain sites with large user base, it would be better if users worried about suspensions? If the suspensions were handed out more readily to stop poor reviewing, would it be that detrimental? Are there enough reviewers to take up slack from more people being suspended? (note: I'm not saying I have some amazing idea you lot haven't thought of/discussed etc)
    – James
    Commented Jun 5, 2015 at 13:18
  • 1
    No. You should either worry about understanding the posts you're reviewing and making useful decisions... Or just not review. Worrying about suspensions puts your focus on the wrong thing; you don't motivate people to care through fear.
    – Shog9
    Commented Jun 5, 2015 at 14:58
  • "you don't motivate people to care through fear" I'm just suggesting fear could be used to instil awareness and consideration towards bad actions. We have motivation through reward, so can use fear of a negative outcome to prevent poor reviewing. Heck, I'm not suggesting we chain 'em up and thrash 'em with a cat o' nine tails, just "suspend from reviewing". It's hardly a serious fear...
    – James
    Commented Jun 6, 2015 at 4:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .