What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 129 Stack Exchange communities.

While the review audits are great and much needed, as far as I can tell they do not apply for the suggested edits queue. Robo reviewers can still approve crap and smile all the way to their badges.

Is there a reason behind this? Constructing dummy suggested edits which are obviously bad shouldn't be too hard, and those who will approve them will get the nice warning message.

share|improve this question
7  
shouldn't be too hard - yes indeed. "Just take a reasonably good post, make a suggestion to wrap it into senseless "Hi" and "Thanks", add an absurd edit comment like "improved formatting" and voila..." (suggested here) –  gnat Dec 20 '12 at 9:36
2  
Cheers @gnat let's hope it will get implemented soon! :) –  Shadow Wizard Dec 20 '12 at 9:38
3  
for extra points, make a spelling error in the edit comment: "formated code" for example - especially if there's no code in the post. Or "grammer impoving". Shouldn't matter, but often it's a tip about how careful the editor is being. –  Kate Gregory Dec 20 '12 at 12:43
5  
What should happen when the reviewer hits "improve"? –  CodesInChaos Dec 31 '12 at 14:29
2  
@CodesInChaos good point! In such case, the reviewer can tick or untick checkbox "this edit was helpful", effectively approving or rejecting the suggestion this way. If he ticked it for honeypot suggestion he's nailed. –  Shadow Wizard Dec 31 '12 at 14:47
1  
Where does the term honeypot come from? –  Cole Johnson Jan 28 '13 at 17:14
1  
@ColeJohnson: en.wikipedia.org/wiki/Honeypot_(computing) –  Spontifixus Jan 28 '13 at 17:16
    
@Cole and for the record, I'm not the first here on Meta to use it. :) –  Shadow Wizard Jan 28 '13 at 20:49
1  
I guess they added some... I got one the other day while I was half-asleep - or I just dreamt that... :/ –  Cole Johnson Jan 31 '13 at 17:36
1  
@Cole lol! Be careful in future reviews then, unless you want two days of forced break. :) –  Shadow Wizard Jan 31 '13 at 18:16
    
I'm always careful, but I guess I'll stop reviewing at night... –  Cole Johnson Jan 31 '13 at 18:39
    
@ShaWizDowArd It doesn't, it told me off the moment I hit "Improve" before I got to the "was this useful" box :) –  Deanna Feb 5 '13 at 11:05
    
@Deanna sorry for that! Might be a bug, consider reporting this as a separate issue. –  Shadow Wizard Feb 5 '13 at 11:32
    
@ShaWizDowArd Probably my fault. I should probably have rejected this then come back for the spelling mistake and line breaks :) –  Deanna Feb 5 '13 at 15:12

5 Answers 5

up vote 70 down vote accepted
+200

We've rolled out audit tasks for suggested edits in our more recent builds. What we're doing here is actually kind of fun. They're also sometimes hilarious.

sample fake spam edit review

Since we know suggested edits have really noisy history, the approach we use for other queues (selecting "known good" or "known bad" content to then fake numbers on) won't really work. Instead, we're actually creating new, bad, suggested edits*.

We're building a super simple model (basically a Porter Stemmer + Markov Chains**) of a few thousand posts per-site, and using that to create "looks OK at a glance, but deeply flawed" audit edits. Thus "involve boy code machines".

We'll probably keep tweaking the algorithm, but based on a day or so of data it looks like it's convincing enough to catch really egregious reviewers.

*Not in a technical sense, these audits never get into the DB as suggested edits; but in a display sense.

**We're not doing anything fun with Markov Chains, just a random walk through the model to generate text.

share|improve this answer
1  
The main problem I have with this is that it isn't a very common problem to have with a suggestion. Editing in complete nonsense happens, but not often. I would prefer to have audits that were performing invalid edits of the same category as more commonly rejected edits, such as editing in a comment or very minor edits. With these being so different not only do audits stand out more from non-audit reviews, but it's less likely to teach newer reviewers when something should be rejects. –  Servy Jan 29 '13 at 19:46
28  
@Servy Early numbers suggests double-digit percentages of reviewers don't reliably reject these nonsense edits, it's a real problem. –  Kevin Montrose Jan 29 '13 at 19:47
2  
As terrible as that is, I'm not surprised. I agree this will help address a very serious problem, it's just that it's only addressing people approving everything without reading (which is unfortunately a lot), and making no real attempt at addressing the fact that there are also lot of people trying to review that just don't have a good understanding of what should or shouldn't be approved, or rush and don't take the time to look beyond the obvious. This audit will also never be as effective as it will be today; malicious users will get used to it eventually. –  Servy Jan 29 '13 at 19:50
3  
Was this deployed net-wide, or SO only? –  Yannis Jan 29 '13 at 19:55
4  
@Yannis - network wide, due to difference in volume we only have meaningful numbers on SO currently. –  Kevin Montrose Jan 29 '13 at 20:02
4  
I've seen a couple of these so far, and some are sneakier than others. The first one, I was completely baffled when I saw an edit that contained cout V is a root nbuser resultset containable the the in the text with the comment Improved Formatting, Corrected Spelling. I'm looking forward to "Find the honeypots", although I still wish we could flag prior edits for honeypots. –  LittleBobbyTables Jan 29 '13 at 20:19
12  
Obligatory: fake CS paper generator from MIT. –  Lorem Ipsum Jan 29 '13 at 21:13
3  
@ShaWizDowArd For the suggested edit queue there is no automatic punishment yet, we want to watch these audits for a little bit before we start trusting them (though early signs are good). Same thing we did/are doing for the other queues. –  Kevin Montrose Jan 29 '13 at 22:11
6  
@George: FWIW, we're far more concerned about the handful of "militantly lazy" reviewers than folks engaged enough to recognize audits at this point. There are quite a few "tells" if you actually bother to read the page - but that's the whole idea. –  Shog9 Jan 30 '13 at 15:23
26  
Also, can we please add honeypot examples that change random keywords that aren't code into inline code blocks? These need to be rejected but so many people approve them!!! –  animuson Jan 30 '13 at 18:18
15  
Also edits that add a ton of bolding/italicising. –  ThiefMaster Jan 30 '13 at 18:31
2  
@KevinMontrose - when are users going to be suspended from the Review Edit queue? There is one user in particular I have sadly been keeping tabs on, who has a 400/10 accept/reject rate, and has missed their last 6 review audits. Isn't that enough to revoke a user's review priveleges? –  LittleBobbyTables Feb 8 '13 at 15:19
3  
@LittleBobbyTables - The audits started having teeth over the last week or so. The user you are referring to is now banned from all review queues for a significant duration. I believe they exposed a slight glitch in the process, which I hear is being addressed. –  Brad Larson Feb 12 '13 at 20:58
2  
Just saw my first honeypot edit. The funny thing was, I already rejected it because the word "iPhone" in the title was all lowercase before I even checked if the edit made sense :-). –  user2428118 Mar 12 '13 at 10:10
3  
@KevinMontrose Do we have animuson's suggestion of turning random keywords into inline code yet? –  Jonathan Hobbs Sep 16 '13 at 23:28

I think it should be possible to add a suggest as honeypot button to all the items in all the queues. Maybe only to those with a certain rep, or certain number of helpful flags. I have come across any number of egregiously bad edits, which I've linked to before.

If you wanted to construct some, may I suggest:

  • add a peripherally related tag while leaving terrible grammar and spelling untouched - tons of these every day but here's an example if anyone is skeptical http://stackoverflow.com/review/suggested-edits/1263995
  • add SOLVED to the title while making no other changes
  • add "Need some help with this" at the end of a post leaving Hi, Thanks, and general bad-question content untouched
  • correct jquery to jQuery throughout a post without formatting any of the misformatted code or fixing any of the spelling mistakes
  • add "Thanks in advance" to a post that didn't have it: http://stackoverflow.com/review/suggested-edits/1263994
  • fix a simple typo in the title while ignoring uncapitalized i, thanks in advance, regards, and general wiffle-waffle and repetition in the body http://stackoverflow.com/review/suggested-edits/1342576 (that one was fixed with Improve so you have to look at the edit history to compare what Dom did and what rds did.)

I have seen all of these in the wild, and seen "This edit was already approved, please visit the post" when I tried to reject them.

share|improve this answer
4  
Nice idea, but don't think it will happen this way. Hope to be wrong though! –  Shadow Wizard Dec 20 '12 at 12:10
1  
So the mods would review these suggested honeypots? –  Asad Dec 20 '12 at 12:58
    
@Asad sure, that's a good idea. I have no clue how the audit items get into the other queues, but whatever that mechanism is, I'd love to see it applied to suggested edits. And for all the queues, I think a good way to get honeypot items is for good reviewers to suggest them –  Kate Gregory Dec 20 '12 at 14:17
4  
Edits that DO fix some problems, but leave other, while NOT adding any new problems absolutely can't be honeypots. –  Oleg V. Volkov Jan 1 '13 at 18:25
    
@OlegV.Volkov those are supposed to be rejected as too minor. So fixing jQuery but missing 10 other things should be a fine honeypot. Ditto adding a (possibly even wrong) tag to a horrible post that needs major work. IMO anyway. –  Kate Gregory Jan 1 '13 at 18:59
3  
@KateGregory, "minor" is too subjective term for my personal liking. Better make honeypots from unquestionably bad edits. –  Oleg V. Volkov Jan 1 '13 at 19:07
1  
Well, why is this still not added? –  Cole Johnson Jan 28 '13 at 17:14
    
I'd +1 this more if I could, fantastic idea –  LittleBobbyTables Jan 28 '13 at 18:11

The honeypot edits implemented by the StackOverflow team are cool, but they have one weakness: as far as I can see, all honeypot edits presented to users are made by "new" user<number> accounts with 1 reputation (which is not that rare) and have never been reviewed before.

That way, somebody who writes an auto-approve bot will only have to wait until a suggested edit gets reviewed by others before approving it to make sure it is not a honeypot edit.

So, honeypot edits shouldn't just have fake "users" proposing them, but also fake users approving them in order to fool smarter bots.

TL;DR: Honeypot edits are great, but they could be made even better by including fake accepts.

share|improve this answer
2  
Nope, from what I've seen the users are real and with more than 1 rep. Can you show example? Edit: hmm.. it appears that all audits are now suggested by Community user which is indeed very bad idea, likely a bug. –  Shadow Wizard Apr 8 '13 at 12:15
1  
@Sha Wiz Dow Ard Take this one for example. (You can't see the user anymore.) After 32 minutes, I was the only one to Approve / Reject it. If it would be a real edit, it would have gotten at least one Approve / Reject within 5 minutes. - Regarding the Community user: you'll only see that after approving / rejecting it. –  user2428118 Apr 8 '13 at 12:20
    
I don't think anyone thinks a robo reviewer is actually a script. It's a person who just goes through clicking Approve the moment it's enabled, without thinking or reading the post at all, grinding for a badge. If they pause long enough to check the rep of the suggester, or whether anyone else has reviewed (which you can't tell easily), or go read the original question, or even -gasp- read the diff of the edit, they'll spot that it's a review. It's super easy to spot if you actually read anything on the screen at all. Yes people fail them. –  Kate Gregory Apr 8 '13 at 12:22
    
@Kate Gregory: Even for human "robo"-reviewers it would make it more challenging to include fake accepts, because people are more likely to just copy other's decisions than clicking 'accept' without looking. –  user2428118 Apr 8 '13 at 12:25

If the team doesn't want to craft their own honeypots we could have a meta question where people put in their suggestions for the suggested edit queue; it'd be amusing and helpful!

To pick a random (shorter) question off my front page (and make it slightly worse):


svn commit throws error

when i try to

commit js code in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed svn: Commit failed (details follow): svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user


The honeypot suggested edit could be:

svn commit throws error

when I try to

commit js code

in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed
svn: Commit failed (details follow):
svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user
share|improve this answer
1  
Sorry, didn't really understand what you mean here.. do you suggest something or did something happen to you that is related to my question? –  Shadow Wizard Dec 20 '12 at 9:11
1  
Sorry, I meant that if the team doesn't want to create their own honeypots we could have a question on meta where we craft some appalling examples for them to put in the queue. –  ben is uǝq backwards Dec 20 '12 at 9:12
3  
OK, it's clear now but I don't think it's a good honeypot; personally I'll skip it since I don't know if this is proper syntax or not. IMO should add some obvious spam link to the end or something like that. :) –  Shadow Wizard Dec 20 '12 at 9:26
    
@sha, you're a responsible reviewer though. You're trying to find the bad ones... –  ben is uǝq backwards Dec 20 '12 at 9:32
11  
This isn't obvious enough to be a honeypot. –  Asad Dec 20 '12 at 9:51
    

Can we not just take suggested edits that were either universally accepted, or universally rejected and use them as the honeypot bait?

To further ensure the good suggestions are good suggestions, we could only use the universally accepted answers in the queue if no further edits have been made to the post since the suggested edit (i.e. the suggested edit fixed everything that needed to be fixed).

Yes, it might not be as obvious as the honeypot questions we're using in other review queues, but people should still be choosing the right answer.

share|improve this answer
14  
Not automatically that's for sure. Crap get approved, that's a fact and the opposite is probably true as well. The honeypot suggestions should at least be chosen manually by moderators if not constructed by them. –  Shadow Wizard Dec 20 '12 at 12:58
    
@ShaWizDowArd: I was going to suggest that having a universal acceptance or rejection would surely be enough to prove that the suggestion was either good or bad, but turns out I was wrong e.g.. –  Matt Dec 20 '12 at 13:14
2  
Yeah, I was also naive a while ago.. welcome to the club! :-) –  Shadow Wizard Dec 20 '12 at 14:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .