Are the suggested edit audits too easy?

Question

Yes, this might be another post complaining about audits. But, in this case, I want the audits to be more difficult.

Specifically, I'm concerned that the suggested edits audits are just too darn easy to spot. They don't behave anything like actual edits, and so after a tiny bit of experience (and electric shocks from audit failures) I think they wouldn't slow down a roboreviewer at all.

Unlike the other queues, which use real data (and are legitimately, in my opinion, difficult), the suggested edits audits look like this:

In about 500ms I can tell that is a bogus edit (less time than it takes for the buttons to unfreeze). After just a few of these, I'm about 99% confident I could tell you which edits were audit edits and which ones were not, and I don't think I actually have to spend extra time on them. On the other queues, picking out the audits is very hard because they use real data, and so I wouldn't bother to do so.

Were I a roboreviewer, I could get on with my Accept-happy life, only pausing to reconsider when someone actually changes the text. (Seriously -- how many real edits have you see which change the text in such a drastic manner? 95% of edits are grammar, spelling, formatting or some combination of those).

Are the suggested edits audits too easy? Is the "Markov-chain random-edit-word-salad" too easily detectable? Or does it legitimately slow down and/or deter robo-reviewers? Should we (and can we) be using real data for the suggested audits queue too?

Answer 1

As Martijn notes, they're supposed to be easy - reviewing real suggested edits isn't really all that difficult.

And most reviewers do pass them - nearly 90% of edit review audits are passed.

But a small number of people do fail them. Repeatedly. And those are the folks we badly need to keep out of review, because they are simply not reading what's in front of them.

FWIW, I have seen folks who've repeatedly failed audits, exhibited all the signs of an uncaring reviewer, been blocked for a couple days... And then come back and started actually evaluating edits with a bit of care. In other words, this can actually teach folks to be better reviewers.

I'm not averse to harder, more interesting audits - in particular, "known good" edits would be educational - but the need there is less pressing.

Kate Gregory has some interesting ideas for how such audits might be implemented, btw.

Answer 2

They are easy and obvious on purpose.

The point is to catch out those reviewers who robo-review without bothering to look at the edit itself.

If we got the robo-reviewers to pause to actually look at the suggested audit, then the purpose of the audit has been served.

Answer 3

Yes. Too easy.

• a lot of added text (lots of green)?
• the text added in the first spot is out of place? .

Checkmate! Audit detected.

Press Reject.

---

I can think of some easy ways to make things more interesting

I: Few is more

Don't add the improper edits (e.g. nonsensical text) to so many spots. Sometimes, it's more effective (for auditing purposes) to limit the incorrect edits to say 1 or 2 spots.

II: Add a single line at the bottom/top of post

I think we should add different dummy line depending on type of post.

And the dummy line should be normal, coherent English.

Example for questions

• "Thanks in advance"
• "Please help"
• "Peace out"
• "XOXO"
• "Hi I am new to [pick one of the question's tags] ."
• "This doesn't work for me. Please help!"

Example for answers

• "I have the same question. Have you found an answer?"
• "Hope this helps"

III: Add tags to question title

Just take one of the tags used in the question, and prefix it

For example, let's say a question is tagged perl and grep)

Quantifier follows nothing in regex

change it into one of these monstrosities

[perl] Quantifier follows nothing in regex
[perl][grep] Quantifier follows nothing in regex
perl: Quantifier follows nothing in regex

in essence, just do everything that "Should I use tags in titles?" in Help Center's tagging section says you should avoid.

IV: Improve formatting... NOT!

Basically just add random bad formatting to the post e.g. inappropriate capitalization and/or backticks (though it's much more easier to abuse backticks in non-random manner)

A more complex variation would be to have a database of "keywords" on which the bad formatting would be applied (i.e. words we don't want the formatting to be applied to in reality) instead of just applying to random words.

Backticks/inline code spanning

I think in general

• language names
  • C#
  • PHP
  • HTML
• technology/tool/brand names
  • XNA
  • DirectX
  • Excel
  • Matlab
  • Photoshop
  • Google Map
  • WinForms
  • iOS
  • SQL Server
  • Firefox
  • Internet Explorer
• technical terms (in a general sense)
  • vertical scrollbar
  • background task
  • grid panel

would work well for this. Although we should be careful that the keywords we choose are not used as function or class names (2-word terms are pretty safe, I'd say)

Indentation

Check for code blocks

print "I am a banana!\n";
print "You know nuffin, Jon Skeet!\n";

then indent by 4 more spaces

    print "I am a banana!\n";
    print "You know nuffin, Jon Skeet!\n";

V: Replace link

Edit comment should say "Fixed broken link", but in actuality we are replacing a good, working link with a fake, non-working link.

For example,

• If link's domain is not Stack Overflow, change domain to, say Stack Overflow.
• If link's domain is Stack Overflow, change domain to, say Google.

---

Concluding statement

I think the message I want to convey is, we don't want reviewers to just look at the edit (which is what the current audit does); we want them to look at the edit properly. We can hopefully achieve the latter by making the audit less of a walk in the park.

Also, it'd be good if we can use audits not only as a robo-review deterrent, but also as a means to educate reviewers about good reviewing practices.

For a less programmatic approach, see Kate Gregory's proposal (improperly closed as duplicate IMO) to let reviewers choose past suggested edits as review audits.