Spam-polishing audits

Question

A truly horrific thing happened on 2014-02-26:

In case you can't see that, that's blatant spam. With a suggested edit to fix its link. That edit was unanimously approved.

A suggested edit to fix the link in a blatant spam post was unanimously approved.

Let that sink in for a second.

Now, we all knew Stack Overflow had problems with robo-approvers, but this... this... I'm not the only speechless one here.

However, we can learn from this horrific approval. We can use what was discovered in the Meta post about it to create a systemic solution to this scourge! It seems to me that attempts to polish blatant spam will be blindly approved by robo-reviewers - so let's take advantage of that!

Add a second type of suggested edit review audit, taking a spam post (real spam-nuked example, or auto-generated mess with links) and polishing it. Reviewers are expected to reject it - why polish turds? (I'm open to suggestions on how to handle improving.) On pass or fail, suggest that the reviewer check out the post in question (not a real one, obviously) and flag it. Something like this:

Congratulations

This was a test. You passed.

These suggested edits - making minor improvements to spam posts - should be rejected so that editors learn not to try to polish turds. (wording up for debate)

In future, when you see an edit like this, open the post in a new tab and flag as spam (but only if it actually is spam)¹.

Stop Look and Listen

This was a test. You failed.

These suggested edits - making minor improvements to spam posts - should be rejected so that editors learn not to try to polish turds.

In future, when you see an edit like this, reject it. Then open the post in a new tab and flag as spam (but only if it actually is spam).

Robo-reviewers get caught, spam gets flagged, everybody wins.

¹ This note is there to try and avoid people mistakenly flagging audit subjects (if they end up being real posts) as spam when they're not. Better ideas welcome.

---

@sth suggests in an answer that it's not worth the time to implement audits showcasing such a rare occurrence. I politely disagree. The indication here seems to be that not only are the reviewers not evaluating the edit, they're not even looking at the post; that's a problem, and this seems a reasonable way to catch them. With some of the suggested edits getting approved these days, something is needed - and this particular something has the benefit of educating users about spam and turd-polishing at the same time.

Answer 1

One thing that we've been talking about amongst the moderators is a way to identify reviewers who approve really troublesome content, even after the fact. Anyone who does anything other than flag obvious spam or trolling is harming the site by letting terrible content survive the review process.

When a post receives a spam, offensive, or not-an-answer flag and is later deleted by a moderator or the community, the system should record anyone who reviewed the post and did not flag it. Moderators should have access to a page of these reviewers, adjustable by the last day, week, and month, and sorted by the number of problematic posts that they did not flag. Their review history should be viewable via one click from this page, with problematic posts that were not flagged being highlighted in some fashion on the list.

If we felt certain enough about the lack of false positives returned by such a metric, the system could potentially use this as a secondary criterion for imposing review bans. We'd have to look this over to see if otherwise good reviewers would be caught by it for some reason, but in my experience few reviewers who upvoted or clicked "No Action Needed" on spam were paying attention on anything else.

Such a review statistics page would have clearly shown the reviews here, once that post was deleted by the community as spam, and would have made it easy for us to step in and take action (like we did on the users involved).

It would also help us to identify other problematic posts that might have slipped through review, because these same poor reviewers might have approved other spam or nonsense that we might have to clean up from their review history. Again, this is something I often have to do once I find a poor reviewer via a flag, our other tools, or spam that's been sitting on the site for days.

Answer 2

I was going to post this as a comment to sth answer, but the length of the post quickly escalated.

Let's start by examining the effect of the behavior described above. First, the original offender that suggested the edit. Basically he flooded an possibly overflowing review queue with a proposal that even if approved will not resolve the problem in the post. So, to be clear he:

• wasted the time of the reviewer.
• actually gave more visibility to the spam by promoting it for view to anyone that will browse the review queue.
• failed to do the only needed thing: remove the ads

That said, I would conclude that this is a behavior we wouldn't want to encourage.

So, now that we have concluded that proposing an edit wasn't the action we wanted - because we want the spam to disappear - let's analyze what the approvers did by approving that edit.

If the above assumption stands, the approvers also enforced an unwanted behavior. By approving the proposal, they are basically saying "Good work, keep doing that. Here are 2 rep points for your time". So again:

• they failed to remove the spam
• they encouraged keeping spam

I believe that anything that can help correcting this should be welcome. The real question is another: it is easy enough for the staff to put such restriction in place? As said, we do want to avoid such situation arise, but we must do a cost/benefice estimate to see what we can do to prevent it. This is a question only the staff can answer: is the cost for implementing a system that will work to discourage the above behavior reasonable (I would expect that adding some new audit questions isn't to difficult)?

That said... rejecting that edit cost nothing and will in turn teach the offender a lesson: flag spam and do not polish it. So, I totally agree on your horror.

Answer 3

Can we put this edit in as an audit and fail those who approve.

I think we need to manually pick some of the cases that have come up on meta and use those as examples as the current examples are usually rather easier to spot.

Answer 4

It's technically not the point of a suggested edit review to judge the correctness/appropriateness of an answer. Especially if the edit suggestion is purely syntactic and not semantic, like in this case. If a bad post gets improved formatting the resulting situation is still not worse than before the edit suggestion.

The edit is unnecessary, but not such a big problem. Also I don't think something like that happens often enough to make a whole new class reviews necessary.

Implementing special measures addressing suggested edits on spam posts seems like a waste since only a tiny percentage of spam posts will ever get a edit suggested.