29

To begin with, I thought it was a great idea. Testing if people really cared. But now I have more than 7k, proven I not intend to destroy the community.

I edit post after post, and every time - "congratulations, this was only a test", and at that time, I am flat and almost "done" in my willingness to edit more posts, look for more Close Votes and so on.

It tires me out that I have been very serious, only to know that it just was a test. EVERY TIME.

This experience totally kill my lust and will to continue say take a look at Close Votes.

I understand and appreciate the system, but if a person had passed "the test" lets say ten times, could you make it so this person not have to be tested each and every time afterwards? 10 "passed" tests for each area should be enough.

  • 7
  • 2
  • 4
    I agree to this.. When i first got the review feature, i used to check in every now and then.. The test part was good as it helped me get into the right way.. But then now after around 100+, i still get these not just on odd occasions and stopped review part honestly.. I just check in review every few days or so now compared to minutes.. Something else could be done regarding the frequency atleast i guess.. – Roy M J Jan 8 '14 at 9:49
  • 3
    "How do you decide if a reviewer is a legit reviewer and not a robo approver who has reviewed a lot and gotten lucky" A : by passing the test 10 times in a row. A "robo"cant be so lucky. – davidkonrad Jan 8 '14 at 10:00
  • @psubsee2003, I really cant imagine "robos" are the issue here. – davidkonrad Jan 8 '14 at 10:01
  • @psubsee2003 Yes, but why can I edit other peoples questions, and answers - without review, but not edit Close Votes without a constantly review, a test?? – davidkonrad Jan 8 '14 at 10:08
  • @S.L.Barth, it is not even remotely related. – davidkonrad Jan 8 '14 at 10:09
  • @psubsee2003, you do a pass by a factor of 1000. To keep uptodate, if you think people will slack and be less strict once they get their pass. And I will say, I have seen a lot of good questions being closed, so the system as it is now is not perfect either. – davidkonrad Jan 8 '14 at 10:12
  • 1
    @davidkonrad I guess we disagree there. The post I linked to suggests to increase the number of audits depending on the level of suspicion. You suggest to decrease the number of audits depending on the level of trust. To me that seems to be the same idea, fleshed out in opposite directions. YMMV. – S.L. Barth Jan 8 '14 at 10:17
  • @psubsee2003, well..? I have the privileges I want. 7k on SO. I just want to get rid of the annoying test everytime I try to give back to the community. – davidkonrad Jan 8 '14 at 10:17
  • @psubsee2003, I am not suggesting "stopping them all together" - where do you see this? Read my post. – davidkonrad Jan 8 '14 at 10:24
  • @S.L.Barth - "You suggest to decrease the number of audits", yes the number of passed tests, or tests needed to be passed after you have passed lets say 10 tests. – davidkonrad Jan 8 '14 at 10:26
  • the number can be 20 if not 10 .. the total number of pass can be between 10 to 25 .. a bot cant be tht lucky anyhow .. but once i pass those number .. system should not test me again ... – Dhaval Jan 8 '14 at 10:28
  • @davidkonrad I did (and couple of times) and I misinterpreted what you were asking. If you only want to stop the message itself, then that's ok. I apologize and I'm going to clean up my comment chain here since it isn't relavant. – psubsee2003 Jan 8 '14 at 10:29
  • This comment section is a mess.. OT: I review a lot and the audits really don't bother me, they aren't that frequent and they're easy to spot (hence wasting very little of my time). In CV and SE queues I try to skip them so as to not use up my review quota. – OGHaza Jan 8 '14 at 16:50
20

I've got over 20k on the site. I was an elected community moderator for two years, and now I work here with developer level access.

I still get audits, just like everyone else.

Any time you have a repetitive task such as this, I think it's good to have these to catch you if you doze off. As you noted, almost all of the time, you see 'congratulations'.

I'm not declining this, but if we lift these after a certain point, there's really nothing sticking in the back of people's heads saying pay close attention (I know that many people would anyway, and just stop reviewing if it gets monotonous).

I'm just drawn back to the impetus for implementing these and remember the horror, and perhaps I'm thinking too hard. Still, I'd rather go for reducing them, if anything, and not getting rid of them entirely after folks have proven that they're good reviewers.

Even stellar reviewers fail these once in a while (some rather obvious cases, too). I'm going to look into it more.

  • 2
    I've got over 20k on the site. I was an elected community moderator for two years, and now I work here with developer level access. : What's the next step ? – ʞunɥdɐpɐɥd Jan 8 '14 at 14:33
  • 1
    @insertcleverxmasname Create an alter ego, wash rinse and repeat. In ten years I'll be getting three different paychecks and nobody will be the wiser. Ssssh, don't tell anyone. – Tim Post Jan 8 '14 at 14:38
  • If you fail every audit it´s there any consequence? – axierjhtjz Jan 8 '14 at 14:38
  • @axierjhtjz Yes, the system will give you a little time off from reviewing. – Tim Post Jan 8 '14 at 14:38
  • That´s nice because I´ve seen lots of new questions those days back that really didn't deserve being approved. If there wasn't any kind of 'punishment' like action for failing those audits, in my opinion, that wasn't right. Thanks for the clarification @TimPost – axierjhtjz Jan 8 '14 at 14:41
  • 1
    Certainly - reduce the rate. Maybe award rep for passing an audit... – Magoo Jan 9 '14 at 3:29
  • @Magoo Awarding rep for passing an audit might actually be a good idea. You at least get something for the time you spent looking at the audit, and there's additional incentive to get into review and pass audits. Going to bring it up and see what others think. – Tim Post Jan 9 '14 at 7:40
  • OK. As a user I do not have the full overview of the need of audits, like people earned the stellar bagde also happend to fail. The surprises a little. Can only speak for myself. As speaking there is a hump on 104k close votes to be reviewed. I think that would be less, if regular reviewers not should go through an audit before actually review "real" close votes. – davidkonrad Jan 9 '14 at 11:15
  • I think reputation points for passing an audit would be a "terrible" idea. Then people is (maybe) somehow motivated to pass audits for the reps alone, and stop reviewing once they have earned the points. I still think it could be done "smarter", like a privilege after 10 passed audits in a row or something like that. But not so sure anymore reading after your answer. Reducing them is probably the best solution. – davidkonrad Jan 9 '14 at 11:22
  • You get the accepted answer. Not only can I understand your considerations, it seems that you on SO have changed the politics Today, I have edited 40 close votes, 20 Suggested Edits, 20 First Posts and 20 Low Quality Posts without being tested! Nice! I have also, due to your answer, reconsidered the concept of being tested once in a while. It is Ok sometimes, but everyday and everytime is a plague. You seem to have changed that. – davidkonrad Jan 24 '14 at 22:03
4

As I see it, audits have two primary roles.

The first of these is to educate new reviewers on how to properly review. If the reviewer fails the audit, they get a message explaining what to differently next time, and if they pass they get a message encouraging them that they are doing the right thing. This is good.

The second role audits play is to catch out sloppy reviewing, whether by a new or a seasoned reviewer. In this case, education isn't so important - what's important is letting the user know when they've made a mistake, and keeping track of users that are consistently making mistakes (e.g. robo-reviewers).

Note that the "audit passed" messages are only really useful in the first case - once a user has been educated on how to review, we only really care if they still keep getting it wrong. For a reviewer who has learned how to review properly and is doing a good job, the audit passed messages only serve to slow down reviewing by adding extra stuff they need to wade through.

I don't propose reducing the number of audits given to "educated" reviewers[1] - they're still as capable of making honest mistakes and/or losing attention as a new reviewer is. Instead, I propose that educated reviewers shouldn't be given audit passed messages - they should just proceed to the next review task, the same as if the audit were actually an ordinary review. They should definitely still get failed audit messages, however.

As for what constitutes an "educated" reviewer, I think five consecutive passed audits should be sufficient. An educated reviewer who fails an audit should be reset to being a "new" reviewer, at which point they would need five consecutive passed audits to again be considered an educated reviewer. This should be on a per-queue basis - being educated in one review type should not imply being educated in a different queue.

EDIT: re-reading at the question, I'm not really sure if this is basically what the OP is proposing or not. This seems to be the solution the OP has hinted at in the question title, but the final paragraph of the question suggests he's actually interested in reducing or eliminating audits for educated users (which I'd somewhat oppose). Either way, I'll leave this here in the hope that it proves useful somehow.


[1]: Possibly some reduction is in order, but that's not the point of the proposal. Even if it were appropriate, I'd suggest that dropping the audit rate by more than 50% or so would greatly increase the risk of honest mistakes slipping through between audits.

  • Yes, skipping the message could reduce the "nag"-factor. And it is also my impression that the audits slows down the overall review-process. But after reading comments and answers I am not so sure there is something to do about it which not implies some disadvantages or riscs. I think a good bullet proof review system is more important than easy access at any cost. – davidkonrad Jan 9 '14 at 11:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .