What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Yes, this might be another post complaining about audits. But, in this case, I want the audits to be more difficult.

Specifically, I'm concerned that the suggested edits audits are just too darn easy to spot. They don't behave anything like actual edits, and so after a tiny bit of experience (and electric shocks from audit failures) I think they wouldn't slow down a roboreviewer at all.

Unlike the other queues, which use real data (and are legitimately, in my opinion, difficult), the suggested edits audits look like this:

enter image description here

In about 500ms I can tell that is a bogus edit (less time than it takes for the buttons to unfreeze). After just a few of these, I'm about 99% confident I could tell you which edits were audit edits and which ones were not, and I don't think I actually have to spend extra time on them. On the other queues, picking out the audits is very hard because they use real data, and so I wouldn't bother to do so.

Were I a roboreviewer, I could get on with my Accept-happy life, only pausing to reconsider when someone actually changes the text. (Seriously -- how many real edits have you see which change the text in such a drastic manner? 95% of edits are grammar, spelling, formatting or some combination of those).

Are the suggested edits audits too easy? Is the "Markov-chain random-edit-word-salad" too easily detectable? Or does it legitimately slow down and/or deter robo-reviewers? Should we (and can we) be using real data for the suggested audits queue too?

share|improve this question
6  
They are easy on purpose. –  Martijn Pieters Mar 10 '13 at 18:01
1  
"Were I a roboreviewer, I could get on with my Accept-happy life, only pausing to reconsider when.." ... well, then you're not really a roboreviewer, are you? –  Bart Mar 10 '13 at 18:02
2  
@Bart: In the 500ms that the buttons are paused, I can "review" the audit and save myself. The edits are so easy I don't have to spend more time than is necessary. –  nneonneo Mar 10 '13 at 18:06
    
What "real data" are you proposing we use? –  animuson Mar 10 '13 at 18:07
3  
Rejected edits. Given the number of edits reviewed, we must have some good rejected edits to use as audit fodder. –  nneonneo Mar 10 '13 at 18:08
    
+1 If you click "Approve" as soon as it's available, you don't recognize such edit as spam. You don't want to catch people having "bad judgement", you want to catch people having no judgement at all. –  yo' Mar 10 '13 at 18:12
    
@nneonneo I would love to see if such roboreviewers exist. The ones who only change their behavioral pattern on an audit. I simply don't have the data to confirm or deny it. –  Bart Mar 10 '13 at 18:13
1  
@Bart: Well, the audit consequences mean that the ones dumb enough to keep clicking Accept must be gone by now. The audits must help weed out some of the bad reviewers, but my thesis here is that many of them must remain by adapting (and not in the way we want them to adapt, i.e. by actually reading the edits). –  nneonneo Mar 10 '13 at 18:15
    
That partially ignores the fact of a growing user-base and an ever increasing amount of users with the privileges to review. –  Bart Mar 10 '13 at 18:16
3  
@tohecz: Actually, we want people with reasonable judgement manning the queues. After all, what use is it if the reviewer gets it wrong often? We may as well have a real roboreviewer. –  nneonneo Mar 10 '13 at 18:17

3 Answers 3

up vote 21 down vote accepted

As Martijn notes, they're supposed to be easy - reviewing real suggested edits isn't really all that difficult.

And most reviewers do pass them - nearly 90% of edit review audits are passed.

But a small number of people do fail them. Repeatedly. And those are the folks we badly need to keep out of review, because they are simply not reading what's in front of them.

FWIW, I have seen folks who've repeatedly failed audits, exhibited all the signs of an uncaring reviewer, been blocked for a couple days... And then come back and started actually evaluating edits with a bit of care. In other words, this can actually teach folks to be better reviewers.

I'm not averse to harder, more interesting audits - in particular, "known good" edits would be educational - but the need there is less pressing.

Kate Gregory has some interesting ideas for how such audits might be implemented, btw.

share|improve this answer
    
Just out of interest, are you aware of any reviewers whose pattern of acceptances only changes upon an audit? Does that happen? –  Bart Mar 10 '13 at 18:37
1  
I would like to see some more interesting audits appear in the queue. I'm actually surprised that over 10% of edit review audits fail -- I was expecting a smaller number because of the penalties of failing them. –  nneonneo Mar 10 '13 at 18:40
    
Well, the penalty is... You have to stop reviewing for a while. It's not really all that harsh. I tend to assume most of the folks making these mistakes are just... really careless. –  Shogging through the snow Mar 10 '13 at 18:41
1  
It's harsh enough to temporarily stop more failures from occurring. Hence I would expect it to be self-limiting. –  nneonneo Mar 10 '13 at 18:43
1  
@Bart: to some extent most folks' patterns change with audits - else they'd be failing! But yeah, there are some folks who almost never reject anything but audits. –  Shogging through the snow Mar 10 '13 at 18:57
1  
@Bart, I'm an example. Not with edits, but new posts/answers. Yes, I became somewhat careless, and got reprimanded. Hasn't happened again (OK, got a few audits "wrong" since, where I, even after looking it over again with much more care, disagreed with the reject decision). –  vonbrand Mar 10 '13 at 20:06
2  
@Shog9 We're only testing for correctly identifying vandalism, how about also testing for the others - too minor is reasonably easy - change a single letter or word (replace with synonym?), the others are admittedly more difficult. –  Dukeling Apr 5 '13 at 21:47
2  
I actually think "too minor" would be an extremely difficult audit, @Dukeling - folks tend to have very different ideas of what it actually means. Also, you couldn't really get away with just changing a single letter (or necessarily a single word) - there's a 6-char limit on suggested edits, so extremely tiny changes would be a dead giveaway. –  Shogging through the snow Apr 5 '13 at 21:55

They are easy and obvious on purpose.

The point is to catch out those reviewers who robo-review without bothering to look at the edit itself.

If we got the robo-reviewers to pause to actually look at the suggested audit, then the purpose of the audit has been served.

share|improve this answer
7  
I hear this argument all the time. But I don't know if I buy it. The edits are so easy that the short mandatory pause is enough to detect them. –  nneonneo Mar 10 '13 at 18:07
1  
@nneonneo: But you actually looked at the edit! Score! –  Martijn Pieters Mar 10 '13 at 18:07
    
Sure. And the point of that 500ms pause was to get people to look at the edit. –  nneonneo Mar 10 '13 at 18:08
    
@nneonneo: And it wasn't enough. People still didn't look at the edit. And perhaps you are fast reader with a good grasp of english, and thus can spot the fake quite easily. Many (most?) people actually have to look a little longer, after auto-approving the last 15 edits.. –  Martijn Pieters Mar 10 '13 at 18:09
4  
Really? My criteria for detecting these edits is not the English content, but the fact that it looks nothing like 95% of edits in the queue. It stands out on that basis alone. Simple human pattern matching suffices to weed it out. –  nneonneo Mar 10 '13 at 18:10
3  
Again, you are looking at the edits. For you the difference is obvious. I don't agree that only pattern matching will save the robo-reviewers; I just went through the suggested edits queue and found no evidence for an obvious pattern difference. –  Martijn Pieters Mar 10 '13 at 18:16
    
I do not agree. Audits in other queues seem to be designed to also teach more about proper audits. The suggested edits ones are so basic that they don't teach people, for example, that major changes to the code in the answer (to another valid code, not some nonsense) are not valid. Same for adding an answer in the question. And many other common invalid ways people edit posts. –  Szymon Mar 13 at 20:06

Yes. Too easy.

  • a lot of added text (lots of green)? check
  • the text added in the first spot is out of place? check.

Checkmate! Audit detected.

Press Reject.


I can think of some easy ways to make things more interesting

I: Few is more

Don't add the improper edits (e.g. nonsensical text) to so many spots. Sometimes, it's more effective (for auditing purposes) to limit the incorrect edits to say 1 or 2 spots.

II: Add a single line at the bottom/top of post

I think we should add different dummy line depending on type of post.

And the dummy line should be normal, coherent English.

Example for questions

  • "Thanks in advance"
  • "Please help"
  • "Peace out"
  • "XOXO"
  • "Hi I am new to [pick one of the question's tags] ."
  • "This doesn't work for me. Please help!"

Example for answers

  • "I have the same question. Have you found an answer?"
  • "Hope this helps"

III: Add tags to question title

Just take one of the tags used in the question, and prefix it

For example, let's say a question is tagged and )

Quantifier follows nothing in regex

change it into one of these monstrosities

[perl] Quantifier follows nothing in regex
[perl][grep] Quantifier follows nothing in regex
perl: Quantifier follows nothing in regex

in essence, just do everything that "Should I use tags in titles?" in Help Center's tagging section says you should avoid.

IV: Improve formatting... NOT!

Basically just add random bad formatting to the post e.g. inappropriate capitalization and/or backticks (though it's much more easier to abuse backticks in non-random manner)

A more complex variation would be to have a database of "keywords" on which the bad formatting would be applied (i.e. words we don't want the formatting to be applied to in reality) instead of just applying to random words.

Backticks/inline code spanning

I think in general

  • language names
    • C#
    • PHP
    • HTML
  • technology/tool/brand names
    • XNA
    • DirectX
    • Excel
    • Matlab
    • Photoshop
    • Google Map
    • WinForms
    • iOS
    • SQL Server
    • Firefox
    • Internet Explorer
  • technical terms (in a general sense)
    • vertical scrollbar
    • background task
    • grid panel

would work well for this. Although we should be careful that the keywords we choose are not used as function or class names (2-word terms are pretty safe, I'd say)

Indentation

Check for code blocks

print "I am a banana!\n";
print "You know nuffin, Jon Skeet!\n";

then indent by 4 more spaces

    print "I am a banana!\n";
    print "You know nuffin, Jon Skeet!\n";

V: Replace link

Edit comment should say "Fixed broken link", but in actuality we are replacing a good, working link with a fake, non-working link.

For example,

  • If link's domain is not Stack Overflow, change domain to, say Stack Overflow.
  • If link's domain is Stack Overflow, change domain to, say Google.

Concluding statement

I think the message I want to convey is, we don't want reviewers to just look at the edit (which is what the current audit does); we want them to look at the edit properly. We can hopefully achieve the latter by making the audit less of a walk in the park.

Also, it'd be good if we can use audits not only as a robo-review deterrent, but also as a means to educate reviewers about good reviewing practices.

For a less programmatic approach, see Kate Gregory's proposal (improperly closed as duplicate IMO) to let reviewers choose past suggested edits as review audits.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .