Do I ever get rid of "congratulations, this was only a test" message?

Question

To begin with, I thought it was a great idea. Testing if people really cared. But now I have more than 7k, proven I don't intend to destroy the community.

I edit post after post, and every time - "congratulations, this was only a test", and at that time, I am flat and almost "done" in my willingness to edit more posts, look for more Close Votes and so on.

It tires me out that I have been very serious, only to know that it was just a test. EVERY TIME.

This experience totally kills my lust and will to continue or say take a look at Close Votes.

I understand and appreciate the system, but if a person had passed "the test" lets say ten times, could you make it so this person does not have to be tested each and every time afterwards? 10 "passed" tests for each area should be enough.

Answer 1

I've got over 20k on the site. I was an elected community moderator for two years, and now I work here with developer level access.

I still get audits, just like everyone else.

Any time you have a repetitive task such as this, I think it's good to have these to catch you if you doze off. As you noted, almost all of the time, you see 'congratulations'.

I'm not declining this, but if we lift these after a certain point, there's really nothing sticking in the back of people's heads saying pay close attention (I know that many people would anyway, and just stop reviewing if it gets monotonous).

I'm just drawn back to the impetus for implementing these and remember the horror, and perhaps I'm thinking too hard. Still, I'd rather go for reducing them, if anything, and not getting rid of them entirely after folks have proven that they're good reviewers.

Even stellar reviewers fail these once in a while (some rather obvious cases, too). I'm going to look into it more.

Answer 2

As I see it, audits have two primary roles.

The first of these is to educate new reviewers on how to properly review. If the reviewer fails the audit, they get a message explaining what to differently next time, and if they pass they get a message encouraging them that they are doing the right thing. This is good.

The second role audits play is to catch out sloppy reviewing, whether by a new or a seasoned reviewer. In this case, education isn't so important - what's important is letting the user know when they've made a mistake, and keeping track of users that are consistently making mistakes (e.g. robo-reviewers).

Note that the "audit passed" messages are only really useful in the first case - once a user has been educated on how to review, we only really care if they still keep getting it wrong. For a reviewer who has learned how to review properly and is doing a good job, the audit passed messages only serve to slow down reviewing by adding extra stuff they need to wade through.

I don't propose reducing the number of audits given to "educated" reviewers^[1] - they're still as capable of making honest mistakes and/or losing attention as a new reviewer is. Instead, I propose that educated reviewers shouldn't be given audit passed messages - they should just proceed to the next review task, the same as if the audit were actually an ordinary review. They should definitely still get failed audit messages, however.

As for what constitutes an "educated" reviewer, I think five consecutive passed audits should be sufficient. An educated reviewer who fails an audit should be reset to being a "new" reviewer, at which point they would need five consecutive passed audits to again be considered an educated reviewer. This should be on a per-queue basis - being educated in one review type should not imply being educated in a different queue.

EDIT: re-reading at the question, I'm not really sure if this is basically what the OP is proposing or not. This seems to be the solution the OP has hinted at in the question title, but the final paragraph of the question suggests he's actually interested in reducing or eliminating audits for educated users (which I'd somewhat oppose). Either way, I'll leave this here in the hope that it proves useful somehow.

---

_{[1]: Possibly some reduction is in order, but that's not the point of the proposal. Even if it were appropriate, I'd suggest that dropping the audit rate by more than 50% or so would greatly increase the risk of honest mistakes slipping through between audits.}