Why no honeypot suggested edits?

Question

While the review audits are great and much needed, as far as I can tell they do not apply for the suggested edits queue. Robo reviewers can still approve crap and smile all the way to their badges.

Is there a reason behind this? Constructing dummy suggested edits which are obviously bad shouldn't be too hard, and those who will approve them will get the nice warning message.

Answer 1

We've rolled out audit tasks for suggested edits in our more recent builds. What we're doing here is actually kind of fun. They're also sometimes hilarious.

Since we know suggested edits have really noisy history, the approach we use for other queues (selecting "known good" or "known bad" content to then fake numbers on) won't really work. Instead, we're actually creating new, bad, suggested edits*.

We're building a super simple model (basically a Porter Stemmer + Markov Chains**) of a few thousand posts per-site, and using that to create "looks OK at a glance, but deeply flawed" audit edits. Thus "involve boy code machines".

We'll probably keep tweaking the algorithm, but based on a day or so of data it looks like it's convincing enough to catch really egregious reviewers.

_{*Not in a technical sense, these audits never get into the DB as suggested edits; but in a display sense.}

_{**We're not doing anything fun with Markov Chains, just a random walk through the model to generate text.}

Answer 2

I think it should be possible to add a suggest as honeypot button to all the items in all the queues. Maybe only to those with a certain rep, or certain number of helpful flags. I have come across any number of egregiously bad edits, which I've linked to before.

If you wanted to construct some, may I suggest:

• add a peripherally related tag while leaving terrible grammar and spelling untouched - tons of these every day but here's an example if anyone is skeptical https://stackoverflow.com/review/suggested-edits/1263995
• add SOLVED to the title while making no other changes
• add "Need some help with this" at the end of a post leaving Hi, Thanks, and general bad-question content untouched
• correct jquery to jQuery throughout a post without formatting any of the misformatted code or fixing any of the spelling mistakes
• add "Thanks in advance" to a post that didn't have it: https://stackoverflow.com/review/suggested-edits/1263994
• fix a simple typo in the title while ignoring uncapitalized i, thanks in advance, regards, and general wiffle-waffle and repetition in the body https://stackoverflow.com/review/suggested-edits/1342576 (that one was fixed with Improve so you have to look at the edit history to compare what Dom did and what rds did.)

I have seen all of these in the wild, and seen "This edit was already approved, please visit the post" when I tried to reject them.

Answer 3

The honeypot edits implemented by the Stack Overflow team are cool, but they have one weakness: as far as I can see, all honeypot edits presented to users are made by "new" user<number> accounts with 1 reputation (which is not that rare) and have never been reviewed before.

That way, somebody who writes an auto-approve bot will only have to wait until a suggested edit gets reviewed by others before approving it to make sure it is not a honeypot edit.

So, honeypot edits shouldn't just have fake "users" proposing them, but also fake users approving them in order to fool smarter bots.

TL;DR: Honeypot edits are great, but they could be made even better by including fake accepts.

Answer 4

If the team doesn't want to craft their own honeypots we could have a meta question where people put in their suggestions for the suggested edit queue; it'd be amusing and helpful!

To pick a random (shorter) question off my front page (and make it slightly worse):

---

svn commit throws error

when i try to

commit js code in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed svn: Commit failed (details follow): svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user

---

The honeypot suggested edit could be:

svn commit throws error

when I try to

commit js code

in svn(eclipse)

i got the following error

org.tigris.subversion.javahl.ClientException: RA layer request failed
svn: Commit failed (details follow):
svn: Server sent unexpected return value (400 Bad Request) in response to MERGE request for ''/arty/arty/trunk/src/main/webapp/js/user

Answer 5

Can we not just take suggested edits that were either universally accepted, or universally rejected and use them as the honeypot bait?

To further ensure the good suggestions are good suggestions, we could only use the universally accepted answers in the queue if no further edits have been made to the post since the suggested edit (i.e. the suggested edit fixed everything that needed to be fixed).

Yes, it might not be as obvious as the honeypot questions we're using in other review queues, but people should still be choosing the right answer.