Abstract
Matching user to tags they already know something about in review seems like a good way to increase accuracy of reviews. It also has the potential to increase user engagement with review in general.
What causes people to fail review audits?
For the purpose of this analysis, I'm focusing on suggested edit audits:
The test presents a reviewer with a randomly generated edit utilizing the Markov chain algorithm to randomly insert various words that may appear to be useful at first glance but actually make no sense and are meant to vandalize the post.
I primarily picked that category because it closely resembles Documentation review. It has the added benefit of being a simple test of reviewers' ability to understand what they are reading. Other review audits tend to be tests of how well the reviewer understands (and conforms) to Stack Overflow culture.
Suggested edit audits present the reviewer with a bit of nonsense that (statistically speaking) appears to be sensible at first glance. So we might expect that reviewers that pass the audit will do so by spending a bit more time thinking about what they are reading. (Readers of Thinking, Fast and Slow will recognize this as the process of engaging "System 2" thinking.) In fact, this appears to be a correct assumption:
AuditPassed reviews duration_s reputation account_age tag_answers tag_score
----------- ------- ---------- ---------- ----------- ----------- ---------
null 8870130 22.2 16780 810 71 203
True 251856 31.4 14156 773 26 73
False 17353 25.8 9545 770 24 59
On average, reviews take 22.2 seconds. Audits tend to slow down reviewers a bit as they have to parse nonsense. But reviewers who fail the audit take less time (25.8s) than those who pass (31.4s). I've included a few other factors that might make a difference (and were easy to collect). Reputation is a bit misleading, since I'm looking at reputation now for a user who might have failed a review years ago. Account age doesn't have that problem because I calculated it by subtracting the date of the review from the date the user created their account. People who have been around longer do a better job avoiding audits, but, somewhat surprisingly, don't seem particularly more likely to pass them.
Does having experience in a tag correlate with more careful reviewing?
I also included tag answers and score in the above table. I calculated this by looking at all the tags applied to the post under review. (If the post was an answer, I looked at the question's tags.) There does seem to be a correlation between passing audits and having more experience in a tag, which is encouraging. But like reputation, I'm not able to determine what tags a person was active in at the time of review. There's also a strong survivorship bias since people who fail a lot of review audits are also getting suspended from reviewing.
More than that, I really want to know if people with no experience in a tag have a harder time passing audits. Perhaps more to the point, do people with experience in a tag take more care than those who do not? To figure that out, I flipped the data around and compared users who had at least one answer in the tag they were reviewing to those with no experience answering in any of the tags.
Tag answer? reviews passed_audits failed_audits failure % duration_s reputation
----------- ------- ------------- ------------- --------- ---------- ----------
True 5446437 129791 8475 6.13 23.7 20616
False 3693015 122068 8878 6.78 20.7 10909
On average, people who know enough about a tag to answer a question, spend 3 seconds longer in reviewing that tag. As a result, they are slightly more successful passing audits. They are also dramatically more successful avoiding audits in the first place. At first, this surprised me. But then I looked at the code and discovered it feeds more audits to people who seem like they aren't paying enough attention to review. Assuming the heuristic is accurate (and I think it is), that means people are less engaged with reviews of tags they don't have knowledge of even if they do end up passing audits.
Conclusion
It seems likely that people with at least some interest in a tag will be more careful and less likely to approve bad edits. Presumably, they also enjoy the task more and tend to come back to review more often. Personally, I quickly lose interest in review when I don't use a tag filter. However, not everyone knows about the filter mechanism or wants to fiddle with them. So it would be helpful to filter reviews based on what we know about the user's tag preferences by default. (When a user runs out of reviews, they could optionally revert to no filter.)
I could see further refinements such as holding reviews for people who have tag experience. But I'd want to validate the basic premise first.
