I believe I have had slew of difficult audit tests recently on StackOverflow. I failed two a couple of days ago and I remember thinking after both of them that I disagree with the fact that I failed. I remember one of the reviews started with something along the lines of "I know I don't have enough rep to comment on this but..." and then actually went on to provide what I thought seemed like a valid answer to the question and provided a few lines of code if I recall correctly, which is why I decided, 'well I suppose it looks ok'. Not the best answer in the world, but it was in fact an answer that answered this specific question.
Because of these failed audits, I had a 2 day suspension from doing any reviews. Today was the day after, and I failed just one review audit and then I get a 7 day review suspension. The one I failed was this: https://stackoverflow.com/review/late-answers/2738008 (if nobody else can see this link, I'll paste some of it here).
Perhaps I'm wrong, but my review process for late answers usually goes something like this:
- Look at the current rating. Obviously this isn't a definite indicator, but it's easy to look at and something I see automatically. If I see a -4, most of the time I'm going to be flagging it for something.
- Look at the format of the answer (not reading it, just looking at it). This just helps me get a general "feel" for things. Does it actually look like an answer would normally look? It's not something that takes me any more than 1-2 seconds to determine, but usually good answers will contain a block (or a few blocks) of code with a bit of text around it explaining it. Of course, not all fall into this nice neat little box, but on StackOverflow, most do, so anything out of the ordinary raises my eyebrow.
- Read the answer. If it obviously looks like it's not the answer, someone with low rep posting a comment or question as an answer, is just a one-liner with no code, spam, etc. I will flag it. Most times, if it shouldn't be there, it's blatantly obvious.
- If it's just a few lines of code with no text, I try to look at whether it really answers the question. Usually (if I deem it appropriate) I will add a comment telling the person to try to add some text to their answer since code-only answers are typically not very good.
- Move on to the question. I figure, it's probably grayed out for a reason, right? We're not auditing the question, but the answer. So usually I skim through it, enough to make sure I basically know what it's asking without reading it in meticulous detail, but always paying close attention mostly to the title and looking for specific key words in the question to quickly understand what the person is talking about. Often you can tell what people are asking just from the title anyway.
- Go back to the answer and see if it contains the key words from the question.
In the review audit above, going though the list, it looks like a valid answer. It was ranked -1, so I knew I likely wasn't going to find a "quality" answer, but a bad answer is still an answer nonetheless. It contains a block of code. It says "it works perfectly" at the end, which shows it's not someone saying they are having that problem too, but an actual solution to something. Then moving on to the question, skimming through it, I can see that it's asking something about variable-length arrays, and going back to the answer it mentions variable-length arrays and has things like float m[i][j], and in programming, using square brackets generally represents arrays. So without going through everything with a fine-tooth comb, I clicked on "It looks good" because it appears to be at least an attempt at answering the question.
But, of course, I was wrong. Even though the answer is talking about the same topic as the question, it basically answers the question of "how to do it" rather than "how does the compiler treat it when you do it". After reading it thoroughly, I can understand why it was flagged initially.
Would anyone else agree with me that a 9 day total suspension is a bit much for (I believe) 3 failed review audits?
I have some suggestions, for the review tasks.
Tutorials:
If I remember the first few times I did review tasks I wasn't quite sure what to do under every circumstance. Provide a little bit of info on how to do a review task (even though it's laid out pretty understandably right now), and also the basics of what to do in specific situations, taking the user through when you should (or shouldn't) use the various "flag" options specific to that Stack Exchange site. A link to the tutorial should remain beside each of the review task types so that the user can re-take them. The tutorial should be mandatory before users are allowed to perform review tasks of that type.
Choosing which questions will show up as audits
My main thought here is to try to somehow rank review audits in order of their level of difficulty and with as little moderator intervention as possible. I would assume that the more people that fail a review audit would be directly proportional to its difficulty. If there are audits that a large percentage of reviewers are failing, have a moderator take a look at them and write some sort of short description specific to that review task and possibly provide a link of what should have been done and why. In the example above (my failed review audit) perhaps something along the lines of:
Even though the answer is talking about the same topic as the question, and provides some code, it doesn't explicitly answer the question asked. Be careful! See [this link] about [something relevant]
The more difficult audits should be presented to reviewers, but not counted towards your audit fail score if you get it wrong. The point is, that these are supposed to be tricky, and therefore teach the reviewer what to do in specific situations while not penalizing them as they learn.
For the audits that are deemed to be simple (or the ones that are specifically designed to weed out bots), these would be the bulk of the audits that show up and should therefore count for more when the user fails the audit. StackOverflow tells me that audits are "designed to see if you were paying attention". If I'm not paying attention, I'll fail a simple audit just as much as a difficult one.
Micro-Tutorials:
A few times, while doing review tasks, I really didn't know whether it would be better to accept or reject it for specific reasons and so quite often I would just skip it without ever learning (unless I specifically look it up). In addition to the regular tutorial showing the basics, I suggest randomly (say every 10-15 reviews, similar to the audits) show some sort of a FYI page reminding the reviewer of some sort of a policy that they may not be aware of / may have forgotten or some more difficult situations that could arise. Perhaps make it optionally interactive by having something like "what would you have done in this situation and why?" with a multiple choice list, and show them the answer after it. Some "tricky" circumstances may be:
- What is the policy on homework
- Reminding the users to check the tags when approving/rejecting edits and what to do if an edit deletes an important tag
- What to do with code-only answers
- When, if ever, a "bad" answer should be flagged for deletion
- When a question should be moved to another SE site and which one
- Other pitfalls, etc.
Just today, I was presented with this question in a review task from a new user. It is basically someone who had a homework assignment and didn't know the first thing about how to get started and was asking for help. I flagged it for deletion because it doesn't show any effort in looking for the answer, but I noticed that at least one person reviewing it accepted the question and it also had a +1 vote. So being all paranoid after my 2 day suspension, I asked there if I did the right thing in flagging it for deletion and another user commented that they would have given the OP a break. Now, I see the question is [On hold], so other people with greater powers than I have decided that it isn't a good question. So here, we have at least 3 users that approved of this question while they shouldn't have. Examples like this could be used to teach people as they go along in addition to the audits.
