I've been dissatisfied with the quality of the audits in suggested edits for a while now; the auto-generated text is OK for checking if someone's even looking at the screen, but that's about it. And anyone who likes can use one of several tactics to (at the cost of a small additional effort) guarantee that they never fail, since the audits are always negative. But the existing audits do fill a useful niche in guarding against blatant spam and defacement, so there's no real need to remove them, merely augment them. But suggested edits can't be up- or down-voted, closed, or deleted like questions and answers can, so how can we get good automated selection of reviews, when there's no other feedback mechanism as there is for every other audit type?
That's where the "maximally unanimous" part comes in. I've written a SEDE query to approximate the desired behavior, but briefly, the idea is to pick reviews where a) all reviewers agreed and b) as many reviewers as possible contributed before c) a final (redundantly) binding review was made.
Specifically, if there are n identical reviews (3 on SO, 2 elsewhere), 0 conflicting reviews, and a ♦ mod (including Community ♦) cast a vote, that review can be selected for audits. If the post owner reviewed, however, it should not be selected, since they are not restricted to the same information an ordinary reviewer has available, or the same guidelines. (They can approve radical changes that ordinary reviewers should reject on sight, or choose to reject perfectly good changes because of stylistic decisions or just plain orneriness.) Likewise, if Community ♦ rejects because of edit conflict (as opposed to Reject and Edit/Improve Edit), the review should also not be selected for audits.
Projected audit quality
After hand-reviewing the results from the query on SO, I came up with the following breakdown:
- 74 — great (unambiguously correct w/o domain knowledge)
- 14 — N/A (post owner reviewed)
- 8 — fair (correct, assuming no contradicting domain knowledge)
- 2 — poor (wrong, assuming no contradicting domain knowledge)
- 2 — terrible (unambiguously wrong w/o domain knowledge)
That's a 4.9% error rate, which I consider marginally acceptable for good audit generation. Of note: both terrible audits made the same mistake of rejecting a tag edit for being too minor, then turning around and applying that same tag with other edits. That can be roundly condemned, and apparently that would improve audit accuracy by a large margin.
On SU, an approximate breakdown looks like this:
- 56 — great (unambiguously correct w/o domain knowledge)
- 10 — N/A (post owner reviewed)
- 2 — fair (correct, assuming no contradicting domain knowledge)
- 1 — poor (wrong, assuming no contradicting domain knowledge)
- 2 — terrible (unambiguously wrong w/o domain knowledge)
Error rate is also about 4.9%. Unfortunately, there appears to be a remarkable dearth of maximally unanimous rejections on SU; in the entire history of the site, there are exactly 20 that the query can find, and a few of those are owner-rejected. I hope there are somewhat more from ♦ mods, but I'm not holding my breath.
On SF, it looks like this:
- 39 — great (unambiguously correct w/o domain knowledge)
- 7 — fair (correct, assuming no contradicting domain knowledge)
- 3 — N/A (post owner reviewed)
- 1 — poor (wrong, assuming no contradicting domain knowledge)
- 1 — terrible (unambiguously wrong w/o domain knowledge)
Error rate is 4.2%, which is a little nicer, but nothing to write home about. SF doesn't even have enough maximally unanimous approvals, which is pretty disturbing. They might have to sit this one out. Both of the bad reviews were approvals of lousy tag wikis, either WP copy-paste or awfully similar — one of them by a user that was elected mod a few months later. There's no audit design that can help with that. (OK, that's not true. SE employees hand-picking audits would do the job. Maybe. But that's not very scalable, and the accuracy problems would be a real pain.)
Summary
SO, and quite possibly several other sites (depending on volume), can make use of rigorous selection to generate audits from real reviews without sacrificing accuracy unacceptably. The exact inclusion rates relative to the Markov-ish auto-audits would need to be worked out from a better sample than the query can give (including ♦ mod approvals/rejections and excluding post owners a little better). As long as there's some flow, however, it seems almost entirely beneficial to throw them in.
