36

Going through /review is a bit finicky, for new users and older ones. That's evident when you see statistics of the retention rate of reviewers; most give up after just a couple of reviews. There have been several strategies that were somewhat useful for attracting reviewers but they have not stuck, due the nature of the solution, trying to patch something temporarily.

What I propose is different.

One of the most common problems that have users with review, is that they have no idea of how to deal with the post presented, because, for the reviewer, they are babbling some obscure Klingon that is only spoken by a small sect of monks that lives in a remote mountain of Kronos. That's not right.

As alternative, the system offers us filters, to decide which topics the reviewer would like to focus on; the problem is that it is poorly advertised and its usage is low by new reviewers (and sometimes old ones). (The statistics points that many users doesn't use them outside of the close queue) Also, it has the inconvenience that it is static, and needs manual intervention to modify it.

I propose that instead the system should use what it knows about us (I'm open to using other data points), to give us relevant posts to review, keeping us around our comfort zone from the beginning. Obviously, the manual filter should override this preference.

  • 1
    'Tis surprising that there are up-votes but no comments or answers as responses. On the whole, it seems like a good idea — people should be offered material to review that aligns with their demonstrated interests automatically. On those occasions when I review (a lot in times past, rather seldom now), I use the filters. But it would be sensible for new reviewers to be shown (mostly) material that they're probably familiar with. It leaves a problem for less-frequented tags (getting exposure); hence the 'mostly' — some reviews may need to be shown anyway. Maybe the SKIP button should be large! – Jonathan Leffler Dec 9 '16 at 19:12
  • @JonathanLeffler well, there's always the author for less frequent tags – Braiam Dec 9 '16 at 19:32
  • How did you find out this Tag Future Report thingie? – Oriol Jan 29 '17 at 10:57
  • @Oriol in a meta post, somewhere talking about the interesting tab/front page. – Braiam Jan 29 '17 at 12:00
16

Abstract

Matching user to tags they already know something about in review seems like a good way to increase accuracy of reviews. It also has the potential to increase user engagement with review in general.

What causes people to fail review audits?

For the purpose of this analysis, I'm focusing on suggested edit audits:

The test presents a reviewer with a randomly generated edit utilizing the Markov chain algorithm to randomly insert various words that may appear to be useful at first glance but actually make no sense and are meant to vandalize the post.

I primarily picked that category because it closely resembles Documentation review. It has the added benefit of being a simple test of reviewers' ability to understand what they are reading. Other review audits tend to be tests of how well the reviewer understands (and conforms) to Stack Overflow culture.

Suggested edit audits present the reviewer with a bit of nonsense that (statistically speaking) appears to be sensible at first glance. So we might expect that reviewers that pass the audit will do so by spending a bit more time thinking about what they are reading. (Readers of Thinking, Fast and Slow will recognize this as the process of engaging "System 2" thinking.) In fact, this appears to be a correct assumption:

AuditPassed reviews duration_s reputation account_age tag_answers tag_score 
----------- ------- ---------- ---------- ----------- ----------- --------- 
null        8870130       22.2      16780         810          71       203
True         251856       31.4      14156         773          26        73
False         17353       25.8       9545         770          24        59

On average, reviews take 22.2 seconds. Audits tend to slow down reviewers a bit as they have to parse nonsense. But reviewers who fail the audit take less time (25.8s) than those who pass (31.4s). I've included a few other factors that might make a difference (and were easy to collect). Reputation is a bit misleading, since I'm looking at reputation now for a user who might have failed a review years ago. Account age doesn't have that problem because I calculated it by subtracting the date of the review from the date the user created their account. People who have been around longer do a better job avoiding audits, but, somewhat surprisingly, don't seem particularly more likely to pass them.

Does having experience in a tag correlate with more careful reviewing?

I also included tag answers and score in the above table. I calculated this by looking at all the tags applied to the post under review. (If the post was an answer, I looked at the question's tags.) There does seem to be a correlation between passing audits and having more experience in a tag, which is encouraging. But like reputation, I'm not able to determine what tags a person was active in at the time of review. There's also a strong survivorship bias since people who fail a lot of review audits are also getting suspended from reviewing.

More than that, I really want to know if people with no experience in a tag have a harder time passing audits. Perhaps more to the point, do people with experience in a tag take more care than those who do not? To figure that out, I flipped the data around and compared users who had at least one answer in the tag they were reviewing to those with no experience answering in any of the tags.

Tag answer? reviews passed_audits failed_audits failure % duration_s reputation
----------- ------- ------------- ------------- --------- ---------- ----------
True        5446437        129791          8475      6.13       23.7      20616
False       3693015        122068          8878      6.78       20.7      10909

On average, people who know enough about a tag to answer a question, spend 3 seconds longer in reviewing that tag. As a result, they are slightly more successful passing audits. They are also dramatically more successful avoiding audits in the first place. At first, this surprised me. But then I looked at the code and discovered it feeds more audits to people who seem like they aren't paying enough attention to review. Assuming the heuristic is accurate (and I think it is), that means people are less engaged with reviews of tags they don't have knowledge of even if they do end up passing audits.

Conclusion

It seems likely that people with at least some interest in a tag will be more careful and less likely to approve bad edits. Presumably, they also enjoy the task more and tend to come back to review more often. Personally, I quickly lose interest in review when I don't use a tag filter. However, not everyone knows about the filter mechanism or wants to fiddle with them. So it would be helpful to filter reviews based on what we know about the user's tag preferences by default. (When a user runs out of reviews, they could optionally revert to no filter.)

I could see further refinements such as holding reviews for people who have tag experience. But I'd want to validate the basic premise first.

  • There has been experiments in this line? – Braiam Apr 22 '18 at 23:31
  • Hey Jon have you heard something about this being in the pipeline or someone having any interest? – Braiam Dec 22 '18 at 19:41
  • Not at the moment. There’s a a chance we might do something like this next year. Thanks for reminding me about this. – Jon Ericson Dec 23 '18 at 8:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .