98

While the review audits are great and much needed, as far as I can tell they do not apply for the suggested edits queue. Robo reviewers can still approve crap and smile all the way to their badges.

Is there a reason behind this? Constructing dummy suggested edits which are obviously bad shouldn't be too hard, and those who will approve them will get the nice warning message.

16
  • 11
    shouldn't be too hard - yes indeed. "Just take a reasonably good post, make a suggestion to wrap it into senseless "Hi" and "Thanks", add an absurd edit comment like "improved formatting" and voila..." (suggested here)
    – gnat
    Commented Dec 20, 2012 at 9:36
  • 2
    Cheers @gnat let's hope it will get implemented soon! :) Commented Dec 20, 2012 at 9:38
  • 3
    for extra points, make a spelling error in the edit comment: "formated code" for example - especially if there's no code in the post. Or "grammer impoving". Shouldn't matter, but often it's a tip about how careful the editor is being. Commented Dec 20, 2012 at 12:43
  • 7
    What should happen when the reviewer hits "improve"? Commented Dec 31, 2012 at 14:29
  • 2
    @CodesInChaos good point! In such case, the reviewer can tick or untick checkbox "this edit was helpful", effectively approving or rejecting the suggestion this way. If he ticked it for honeypot suggestion he's nailed. Commented Dec 31, 2012 at 14:47
  • 2
    Where does the term honeypot come from?
    – Cole Tobin
    Commented Jan 28, 2013 at 17:14
  • 1
    @ColeJohnson: en.wikipedia.org/wiki/Honeypot_(computing) Commented Jan 28, 2013 at 17:16
  • @Cole and for the record, I'm not the first here on Meta to use it. :) Commented Jan 28, 2013 at 20:49
  • 1
    I guess they added some... I got one the other day while I was half-asleep - or I just dreamt that... :/
    – Cole Tobin
    Commented Jan 31, 2013 at 17:36
  • 1
    @Cole lol! Be careful in future reviews then, unless you want two days of forced break. :) Commented Jan 31, 2013 at 18:16
  • I'm always careful, but I guess I'll stop reviewing at night...
    – Cole Tobin
    Commented Jan 31, 2013 at 18:39
  • @ShaWizDowArd It doesn't, it told me off the moment I hit "Improve" before I got to the "was this useful" box :)
    – Deanna
    Commented Feb 5, 2013 at 11:05
  • @Deanna sorry for that! Might be a bug, consider reporting this as a separate issue. Commented Feb 5, 2013 at 11:32
  • @ShaWizDowArd Probably my fault. I should probably have rejected this then come back for the spelling mistake and line breaks :)
    – Deanna
    Commented Feb 5, 2013 at 15:12
  • Just trying something Shadow, don't worry.
    – Ollie
    Commented Oct 18, 2020 at 0:39

5 Answers 5

88
+200

We've rolled out audit tasks for suggested edits in our more recent builds. What we're doing here is actually kind of fun. They're also sometimes hilarious.

sample fake spam edit review

Since we know suggested edits have really noisy history, the approach we use for other queues (selecting "known good" or "known bad" content to then fake numbers on) won't really work. Instead, we're actually creating new, bad, suggested edits*.

We're building a super simple model (basically a Porter Stemmer + Markov Chains**) of a few thousand posts per-site, and using that to create "looks OK at a glance, but deeply flawed" audit edits. Thus "involve boy code machines".

We'll probably keep tweaking the algorithm, but based on a day or so of data it looks like it's convincing enough to catch really egregious reviewers.

*Not in a technical sense, these audits never get into the DB as suggested edits; but in a display sense.

**We're not doing anything fun with Markov Chains, just a random walk through the model to generate text.

28
  • 35
    @Servy Early numbers suggests double-digit percentages of reviewers don't reliably reject these nonsense edits, it's a real problem. Commented Jan 29, 2013 at 19:47
  • 2
    As terrible as that is, I'm not surprised. I agree this will help address a very serious problem, it's just that it's only addressing people approving everything without reading (which is unfortunately a lot), and making no real attempt at addressing the fact that there are also lot of people trying to review that just don't have a good understanding of what should or shouldn't be approved, or rush and don't take the time to look beyond the obvious. This audit will also never be as effective as it will be today; malicious users will get used to it eventually.
    – Servy
    Commented Jan 29, 2013 at 19:50
  • 3
    Was this deployed net-wide, or SO only?
    – yannis
    Commented Jan 29, 2013 at 19:55
  • 4
    @Yannis - network wide, due to difference in volume we only have meaningful numbers on SO currently. Commented Jan 29, 2013 at 20:02
  • 4
    I've seen a couple of these so far, and some are sneakier than others. The first one, I was completely baffled when I saw an edit that contained cout V is a root nbuser resultset containable the the in the text with the comment Improved Formatting, Corrected Spelling. I'm looking forward to "Find the honeypots", although I still wish we could flag prior edits for honeypots. Commented Jan 29, 2013 at 20:19
  • 14
    Obligatory: fake CS paper generator from MIT. Commented Jan 29, 2013 at 21:13
  • 3
    @ShaWizDowArd For the suggested edit queue there is no automatic punishment yet, we want to watch these audits for a little bit before we start trusting them (though early signs are good). Same thing we did/are doing for the other queues. Commented Jan 29, 2013 at 22:11
  • 6
    @George: FWIW, we're far more concerned about the handful of "militantly lazy" reviewers than folks engaged enough to recognize audits at this point. There are quite a few "tells" if you actually bother to read the page - but that's the whole idea.
    – Shog9
    Commented Jan 30, 2013 at 15:23
  • 35
    Also, can we please add honeypot examples that change random keywords that aren't code into inline code blocks? These need to be rejected but so many people approve them!!!
    – animuson StaffMod
    Commented Jan 30, 2013 at 18:18
  • 21
    Also edits that add a ton of bolding/italicising.
    – ThiefMaster Mod
    Commented Jan 30, 2013 at 18:31
  • 2
    @KevinMontrose - when are users going to be suspended from the Review Edit queue? There is one user in particular I have sadly been keeping tabs on, who has a 400/10 accept/reject rate, and has missed their last 6 review audits. Isn't that enough to revoke a user's review priveleges? Commented Feb 8, 2013 at 15:19
  • 3
    @LittleBobbyTables - The audits started having teeth over the last week or so. The user you are referring to is now banned from all review queues for a significant duration. I believe they exposed a slight glitch in the process, which I hear is being addressed.
    – Brad Larson Mod
    Commented Feb 12, 2013 at 20:58
  • 2
    Just saw my first honeypot edit. The funny thing was, I already rejected it because the word "iPhone" in the title was all lowercase before I even checked if the edit made sense :-). Commented Mar 12, 2013 at 10:10
  • 5
    @KevinMontrose Do we have animuson's suggestion of turning random keywords into inline code yet? Commented Sep 16, 2013 at 23:28
  • 4
    After auditing a bunch of “Controversial” suggested edits from the Suggested Edits Dashboard, I find myself in vehement and vociferous agreement with @animuson about the need to add review-audit honeypot-traps that add random ˴backticks˴ to non-code spans and leave the entire post looking like some awful ransom note.
    – tchrist
    Commented May 3, 2015 at 16:04
75
+50

I think it should be possible to add a suggest as honeypot button to all the items in all the queues. Maybe only to those with a certain rep, or certain number of helpful flags. I have come across any number of egregiously bad edits, which I've linked to before.

If you wanted to construct some, may I suggest:

  • add a peripherally related tag while leaving terrible grammar and spelling untouched - tons of these every day but here's an example if anyone is skeptical https://stackoverflow.com/review/suggested-edits/1263995
  • add SOLVED to the title while making no other changes
  • add "Need some help with this" at the end of a post leaving Hi, Thanks, and general bad-question content untouched
  • correct jquery to jQuery throughout a post without formatting any of the misformatted code or fixing any of the spelling mistakes
  • add "Thanks in advance" to a post that didn't have it: https://stackoverflow.com/review/suggested-edits/1263994
  • fix a simple typo in the title while ignoring uncapitalized i, thanks in advance, regards, and general wiffle-waffle and repetition in the body https://stackoverflow.com/review/suggested-edits/1342576 (that one was fixed with Improve so you have to look at the edit history to compare what Dom did and what rds did.)

I have seen all of these in the wild, and seen "This edit was already approved, please visit the post" when I tried to reject them.

9
  • 5
    Nice idea, but don't think it will happen this way. Hope to be wrong though! Commented Dec 20, 2012 at 12:10
  • 1
    So the mods would review these suggested honeypots?
    – user200500
    Commented Dec 20, 2012 at 12:58
  • @Asad sure, that's a good idea. I have no clue how the audit items get into the other queues, but whatever that mechanism is, I'd love to see it applied to suggested edits. And for all the queues, I think a good way to get honeypot items is for good reviewers to suggest them Commented Dec 20, 2012 at 14:17
  • 8
    Edits that DO fix some problems, but leave other, while NOT adding any new problems absolutely can't be honeypots. Commented Jan 1, 2013 at 18:25
  • @OlegV.Volkov those are supposed to be rejected as too minor. So fixing jQuery but missing 10 other things should be a fine honeypot. Ditto adding a (possibly even wrong) tag to a horrible post that needs major work. IMO anyway. Commented Jan 1, 2013 at 18:59
  • 4
    @KateGregory, "minor" is too subjective term for my personal liking. Better make honeypots from unquestionably bad edits. Commented Jan 1, 2013 at 19:07
  • 1
    Well, why is this still not added?
    – Cole Tobin
    Commented Jan 28, 2013 at 17:14
  • I'd +1 this more if I could, fantastic idea Commented Jan 28, 2013 at 18:11
  • Another honeypot: Replacing JavaScript with jQuery and / or adding the jquery tag when jQuery isn't mentioned in the body or title.
    – wizzwizz4
    Commented Feb 22, 2017 at 19:31
3

The honeypot edits implemented by the Stack Overflow team are cool, but they have one weakness: as far as I can see, all honeypot edits presented to users are made by "new" user<number> accounts with 1 reputation (which is not that rare) and have never been reviewed before.

That way, somebody who writes an auto-approve bot will only have to wait until a suggested edit gets reviewed by others before approving it to make sure it is not a honeypot edit.

So, honeypot edits shouldn't just have fake "users" proposing them, but also fake users approving them in order to fool smarter bots.

TL;DR: Honeypot edits are great, but they could be made even better by including fake accepts.

4
  • 2
    Nope, from what I've seen the users are real and with more than 1 rep. Can you show example? Edit: hmm.. it appears that all audits are now suggested by Community user which is indeed very bad idea, likely a bug. Commented Apr 8, 2013 at 12:15
  • 1
    @Sha Wiz Dow Ard Take this one for example. (You can't see the user anymore.) After 32 minutes, I was the only one to Approve / Reject it. If it would be a real edit, it would have gotten at least one Approve / Reject within 5 minutes. - Regarding the Community user: you'll only see that after approving / rejecting it. Commented Apr 8, 2013 at 12:20
  • I don't think anyone thinks a robo reviewer is actually a script. It's a person who just goes through clicking Approve the moment it's enabled, without thinking or reading the post at all, grinding for a badge. If they pause long enough to check the rep of the suggester, or whether anyone else has reviewed (which you can't tell easily), or go read the original question, or even -gasp- read the diff of the edit, they'll spot that it's a review. It's super easy to spot if you actually read anything on the screen at all. Yes people fail them. Commented Apr 8, 2013 at 12:22
  • @Kate Gregory: Even for human "robo"-reviewers it would make it more challenging to include fake accepts, because people are more likely to just copy other's decisions than clicking 'accept' without looking. Commented Apr 8, 2013 at 12:25
0

If the team doesn't want to craft their own honeypots we could have a meta question where people put in their suggestions for the suggested edit queue; it'd be amusing and helpful!

To pick a random (shorter) question off my front page (and make it slightly worse):


svn commit throws error

when i try to

commit js code in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed svn: Commit failed (details follow): svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user


The honeypot suggested edit could be:

svn commit throws error

when I try to

commit js code

in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed
svn: Commit failed (details follow):
svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user
6
  • 1
    Sorry, didn't really understand what you mean here.. do you suggest something or did something happen to you that is related to my question? Commented Dec 20, 2012 at 9:11
  • 1
    Sorry, I meant that if the team doesn't want to create their own honeypots we could have a question on meta where we craft some appalling examples for them to put in the queue. Commented Dec 20, 2012 at 9:12
  • 3
    OK, it's clear now but I don't think it's a good honeypot; personally I'll skip it since I don't know if this is proper syntax or not. IMO should add some obvious spam link to the end or something like that. :) Commented Dec 20, 2012 at 9:26
  • @sha, you're a responsible reviewer though. You're trying to find the bad ones... Commented Dec 20, 2012 at 9:32
  • 11
    This isn't obvious enough to be a honeypot.
    – user200500
    Commented Dec 20, 2012 at 9:51
  • I demand that @shog9 craft the honeypots
    – voretaq7
    Commented Jan 29, 2013 at 19:50
-3

Can we not just take suggested edits that were either universally accepted, or universally rejected and use them as the honeypot bait?

To further ensure the good suggestions are good suggestions, we could only use the universally accepted answers in the queue if no further edits have been made to the post since the suggested edit (i.e. the suggested edit fixed everything that needed to be fixed).

Yes, it might not be as obvious as the honeypot questions we're using in other review queues, but people should still be choosing the right answer.

3
  • 15
    Not automatically that's for sure. Crap get approved, that's a fact and the opposite is probably true as well. The honeypot suggestions should at least be chosen manually by moderators if not constructed by them. Commented Dec 20, 2012 at 12:58
  • @ShaWizDowArd: I was going to suggest that having a universal acceptance or rejection would surely be enough to prove that the suggestion was either good or bad, but turns out I was wrong e.g..
    – Matt
    Commented Dec 20, 2012 at 13:14
  • 2
    Yeah, I was also naive a while ago.. welcome to the club! :-) Commented Dec 20, 2012 at 14:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .