I recently received this rather amusing message while reviewing a user's first post:

Congratulations!

This was only a test, designed to make sure you were paying attention. This post has already been removed, but thanks for taking time to leave feedback for the author.

  • What is the purpose of this test?
  • How does the audit work?
  • What happens if I fail?
  • Why was the system expecting me to negatively review a good post, or positively review a bad post?
  • A review is obviously an audit. Should I report it?
  • Do review audits count toward the Reviewer badge?
  • Do robo-reviewers dream of electric sheep?

Return to FAQ index

  • 13
    Is this being done on all sites, including betas? – Neil Fein Mar 4 '13 at 4:01
  • 13
    @NeilFein currently they are active on Stack Overflow, Super User, Programmers and Server Fault. – Rory Mar 29 '13 at 12:34
  • 10
    @TimPost, can you confirm whether this is a network wide now, or limited to certain sites. – mpdonadio Jun 13 '14 at 19:01
  • 4
    UPDATE: AFAIK, this is only on sites where a lot of robo-reviewing has taken place and has been reported. Code review graduated, so I can't remember a beta currently having review audits. – M.A.R. ಠ_ಠ Sep 3 '15 at 12:07
up vote 244 down vote accepted

What is the purpose of this test?

The test (known as a review audit) is designed to help new reviewers hone their moderation skills, while nudging more experienced users who don't seem to be paying close attention to what they're reviewing. Some people were in the habit of up-voting everything they reviewed without regard to quality; extremely low-quality posts with a high score are very problematic in a system where the best information naturally rises to the top, so "audit" tasks are mixed in to make sure folks are paying attention.

This is currently active only on Stack Overflow, Server Fault, Super User, Software Engineering, Ask Ubuntu, and Mathematics.

How does the audit work?

  • First Posts, Late Answers and Low-Quality Posts - The test presents a reviewer with a post that was previously deleted and known to be of extremely poor quality (flagged as spam, "not an answer", or "very low quality"), or known to be of high quality (many up-votes, no down-votes, etc).

  • Close Votes and Reopen Votes - The test presents a reviewer with a question which the system has determined should be open or closed based on various criteria (highly-voted with no close votes, quickly closed with no answers, etc).

  • Suggested Edits - The test presents a reviewer with a randomly generated edit utilizing the Markov chain algorithm to randomly insert various words that may appear to be useful at first glance but actually make no sense and are meant to vandalize the post.

If a user passes a test, they are informed that they passed and thanked for their time and feedback. If a user fails a review audit, they are presented with a message that they didn't pass and encouraged to pay more attention in the future.

The result is then recorded for analysis. Only diamond (♦) moderators or Stack Exchange employees can see the data; it is not accessible publicly even to high reputation users.

What happens if I fail?

For most, nothing. You'll be given some guidance on why the post was inappropriate and what you could have done instead, and then allowed to continue reviewing.

STOP

Repeated failures, especially in short succession can lead to your review privileges being suspended for a week or longer, depending on the frequency of the failures. Stack Exchange is continuing to analyze the data to determine the best action to take in these cases.

Why was the system expecting me to negatively review a good post, or positively review a bad post?

Review audits are chosen automatically. The system isn't perfect, meaning that every so often a post slips through normal community detection, causing the system to generate a bad, invalid, or questionable audit.

These can be split into two cases:

The system expected me to positively review a bad post

  • The most common case is when a low-quality, off-topic, or otherwise close-worthy post gets upvoted. Sometimes visitors may upvote posts without paying attention to the site rules.
  • You identify a post as being an exact duplicate of an older, highly-voted post. To prevent audits from being obvious, the vote count is displayed as zero and the user card is either anonymized or shown as having been posted by a new user. This may make you think you are looking at an exact duplicate.

The system expected me to negatively review a good post

  • Other users may have cast opinionated downvotes, which means they disagree with the post rather than think it is a bad post. You may have had a different opinion, or attempted to review it objectively.
  • The post may have been marked as spam as part of a spamming attack where only a few users knew the details. In this case, moderators can clear the spam flags, to prevent them from being used as audits.

Don't worry! The fact that you identified a bad audit means that you are paying attention, and are not the type of reviewer that the audits are intended to catch.

If you encounter a bad audit, post it on the per-site meta tagged to draw attention to it, containing a link to either the audit task or the post, and why you disagree with the audit. Other users can cast votes to counteract the automatic decisions that led to it being chosen as the wrong type of audit (e.g. downvote and close an off-topic question that got picked as a known-good audit). If you get banned as a result of failing a bad audit, the ban can be lifted or shortened by ♦ moderators at their discretion.

A review is obviously an audit. Should I report it?

There are many ways to identify audits (e.g. click to see the post itself, the user, the vote count, etc.). Almost all of these ways involve having to further investigate the post. By this time, you've already carefully reviewed the post, which is the intent of the queues.

As stated earlier, the main purpose of audits is to catch robo-reviewers, who mindlessly review without even looking at the posts, and to help educate new reviewers. Easy-to-identify audits help legitimate reviewers pass and help better hone new reviewers' skills.

On the other hand, if you spot a user who appears to be gaming the system (e.g. they are making bad reviews but somehow passing every audit, they are using a user script to identify review audits, etc.) flag one of their posts for moderator attention and explain the situation. Moderators will take a further look and maybe issue a manual ban.

Do review audits count towards review badges?

Passed audits count as a successful review towards the review badges. A failed audit does not count toward the badges, but does not decrement your progress either; it simply adds none.

Do robo-reviewers dream of electric sheep?

Robo-reviewers don't sleep. They wait.

Important Note:

The information contained in this answer is subject to change as the review system is further refined. Changes that impact the mechanics of the test, or consequences of failing it repeatedly will be reflected in this answer as they're made, but not necessarily immediately after they're made.

  • 176
    I'd like a "I still disagree, your test if flawed" button for "failed" tests. – CodesInChaos Nov 30 '12 at 15:33
  • 20
    @CodesInChaos: keep in mind, these test posts have already been deleted - so if you disagree strongly, you should probably flag them for moderator attention / bring them up on Meta / etc. – Shog9 Nov 30 '12 at 17:33
  • 37
    I got the "failed" message when closing some questions. It presented a real one and I clicked on the "Close" button just to see how many other people had considered it a "closable" question, but it immediately said I failed the test despite not yet submitting a close vote... I quite often open that close dialog up just to see what other people are thinking (in the event I am overlooking why the question is being voted to be closed). – dreamlax Apr 24 '13 at 17:32
  • 26
    @dreamlax You should not review based on what others are thinking, because that only amplifies existing reviews. The review systems works because multiple people form an independent opinion. If there is (semi-)consensus that probably means the consensus is true and actions can be based on that consensus. However, if that "consensus" is because everyone agrees with the ones before him only the first independant opinion determines the consensus and thus it becomes worthless. – dtech Apr 27 '13 at 12:15
  • 24
    @dtech: I see your point and fair enough. For me though, I don't blindly follow what everyone else is thinking, I only check to see if perhaps my mind has had a severe lapse of judgement. – dreamlax Apr 28 '13 at 3:02
  • 17
    Can users ever graduate out of having to take these tests? I've done hundreds of reviews, and have passed every test thrown at me. How long before the system figures out that I can recognize garbage? Maybe when I earn the Steward badge? – Barmar Jul 3 '13 at 19:14
  • 15
    I suppose. It just feels a little condescending after a while. – Barmar Jul 3 '13 at 19:22
  • 36
    I just failed my first audit by clicking "no action needed", on an answer that had already been deleted, which I knew by clicking through to the full question. Because my understanding of reviewing was to recommend future action (or to take action myself), I concluded that the problem had already been resolved, so no further action was needed. Have I misunderstood the purpose of reviewing, such that I shouldn't be looking at the live post to see the current state of affairs in its larger context? Is my misunderstanding of a different flavour? Advice/clarification would be appreciated. – nmjk Jul 8 '13 at 19:52
  • 10
    Is it theoretically possible for the random algorithm to generate helpful edit suggestions? And, if that happens to me, can that be taken as conclusive proof that this is the matrix, and I am the Chosen One? – root Aug 15 '13 at 16:47
  • 11
    I was thinking about posting a new question, but hopefully a comment here will suffice. My problem with this is that I sometimes want to view some close reasons, which I can do before closing, except when these tests occur. I wanted to check if something was a duplicate by opening the close dialog, but was then "busted". – keyser Aug 16 '13 at 14:11
  • 4
    @Keyser Don't worry about occasionally hitting them as long as you're actually paying attention. Nothing bad will happen if you do that once in a while, it's sort of expected. – Tim Post Aug 16 '13 at 14:27
  • 16
    @dtech Your point, that reviews should not be based on what others think, is correct. However, consider this case: I notice a question has had close votes, but see nothing wrong with it. Before I comment "this post is fine", I'd like to see why it was close-voted in the first place. For instance, it's considered a duplicate, and maybe it is? So I take a look at the linked post to double-check. It is only then that I can decide whether the post truly was a duplicate, or if people may have misjudged the post; maybe they only look similar, but actually ask 2 different things? – Nolonar Jan 29 '14 at 23:28
  • 23
    I would very much like an "I respectfully disagree" button added to the failure dialog. I suppose it may be a character flaw, but if someone solicits my opinion (as SO does by asking me to review things), and then tells me I'm wrong, I expect a chance to argue my position. It just seems polite. – MrTheWalrus Apr 3 '14 at 23:25
  • 10
    I suspect that one real consequence of these audits is to encourage active members not to review. I decided to be a good citizen and do my share and clear off the review list whenever presented. But when you get one or two of these stupid and condescending responses a day, I decided whats the f***ing point and gave up reviewing. How many others have done likewise? You don't get anything for acting as a reviewer other than be insulted by the system for your contribution. – TerryE Jul 12 '14 at 14:29
  • 17
    I don't get these at all; if the point isn't to go along with the herd, then what's the point of the test? If 5 people said a question is poor and should be closed, but I don't think it should be, then there's a disconnect. Either you are encouraging groupthink or you encourage people to make their own decisions, but the fact there's some hidden algorithm that determines if you reviewed "correctly" defeats the purpose of thinking independently. – Wayne Molina Jul 18 '14 at 19:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .