54

A truly horrific thing happened on 2014-02-26:

In case you can't see that, that's blatant spam. With a suggested edit to fix its link. That edit was unanimously approved.

A suggested edit to fix the link in a blatant spam post was unanimously approved.

Let that sink in for a second.

Now, we all knew Stack Overflow had problems with robo-approvers, but this... this... I'm not the only speechless one here.

However, we can learn from this horrific approval. We can use what was discovered in the Meta post about it to create a systemic solution to this scourge! It seems to me that attempts to polish blatant spam will be blindly approved by robo-reviewers - so let's take advantage of that!

Add a second type of suggested edit review audit, taking a spam post (real spam-nuked example, or auto-generated mess with links) and polishing it. Reviewers are expected to reject it - why polish turds? (I'm open to suggestions on how to handle improving.) On pass or fail, suggest that the reviewer check out the post in question (not a real one, obviously) and flag it. Something like this:

Congratulations

This was a test. You passed.

These suggested edits - making minor improvements to spam posts - should be rejected so that editors learn not to try to polish turds. (wording up for debate)

In future, when you see an edit like this, open the post in a new tab and flag as spam (but only if it actually is spam)1.

Stop Look and Listen

This was a test. You failed.

These suggested edits - making minor improvements to spam posts - should be rejected so that editors learn not to try to polish turds.

In future, when you see an edit like this, reject it. Then open the post in a new tab and flag as spam (but only if it actually is spam).

Robo-reviewers get caught, spam gets flagged, everybody wins.

1 This note is there to try and avoid people mistakenly flagging audit subjects (if they end up being real posts) as spam when they're not. Better ideas welcome.


@sth suggests in an answer that it's not worth the time to implement audits showcasing such a rare occurrence. I politely disagree. The indication here seems to be that not only are the reviewers not evaluating the edit, they're not even looking at the post; that's a problem, and this seems a reasonable way to catch them. With some of the suggested edits getting approved these days, something is needed - and this particular something has the benefit of educating users about spam and turd-polishing at the same time.

  • 10
    Turd polisher could be an interesting badge. – Jason Sturges Feb 26 '14 at 9:04
  • 11
    @JasonSturges: Or a positive one, Turd Flusher. – George Duckett Feb 26 '14 at 9:04
  • @BlueIce not a dupe, as another question merely raises concern while this suggests a systemic solution – gnat Feb 26 '14 at 13:45
  • 1
    Ah. That makes sense. – Blue Ice Feb 26 '14 at 14:04
  • 1
    A canonical "don't polish turds" link would be a good thing to have for the banner. Jeff's introduction of the phrase for this issue is too tangential to be clear guidance for an auditee. – Josh Caswell Feb 26 '14 at 19:39
  • 2
    It got approved even without the </a> being removed. :( – Troyen Feb 26 '14 at 19:53
  • 4
    I am upvoting you but I insist that neither the word "turd" nor "polish" appear in the audit text. Something like "missing the need to remove the spam" is more important than that the edit was a kind of polishing. I think especially ESL reviewers will have a problem with the idiom, and I find it puerile and would rather not read it. But the key is, the bad behavior was LEAVING THE SPAM there, not whether or not the better-making was sufficient. – Kate Gregory Feb 26 '14 at 22:43
  • 2
    @KateGregory Yeah, that was all I could think of for the audit text on short notice. As noted in the examples, the wording is up for debate, and better ideas will be gladly plagiaris ^W accepted. – michaelb958--Reinstate Monica Feb 26 '14 at 22:48
33

One thing that we've been talking about amongst the moderators is a way to identify reviewers who approve really troublesome content, even after the fact. Anyone who does anything other than flag obvious spam or trolling is harming the site by letting terrible content survive the review process.

When a post receives a spam, offensive, or not-an-answer flag and is later deleted by a moderator or the community, the system should record anyone who reviewed the post and did not flag it. Moderators should have access to a page of these reviewers, adjustable by the last day, week, and month, and sorted by the number of problematic posts that they did not flag. Their review history should be viewable via one click from this page, with problematic posts that were not flagged being highlighted in some fashion on the list.

If we felt certain enough about the lack of false positives returned by such a metric, the system could potentially use this as a secondary criterion for imposing review bans. We'd have to look this over to see if otherwise good reviewers would be caught by it for some reason, but in my experience few reviewers who upvoted or clicked "No Action Needed" on spam were paying attention on anything else.

Such a review statistics page would have clearly shown the reviews here, once that post was deleted by the community as spam, and would have made it easy for us to step in and take action (like we did on the users involved).

It would also help us to identify other problematic posts that might have slipped through review, because these same poor reviewers might have approved other spam or nonsense that we might have to clean up from their review history. Again, this is something I often have to do once I find a poor reviewer via a flag, our other tools, or spam that's been sitting on the site for days.

  • 2
    This tracking needs to take into account the edit history. Somebody who reviewed a post that was later vandalized shouldn't pay for that. – Monica Cellio Feb 26 '14 at 20:24
  • @MonicaCellio - Those shouldn't be caught by this, because I can't think of a scenario where a post was fine to begin with, somehow got edited into spam, then was spam-flagged and deleted. If it does happen, it's exceedingly rare, because I can't think of a recent case where that occurred. The closest situation would be where someone replaced the content of their own question with nasty stuff as part of a rage-quit, but that's also not very frequent and few of those questions were reviewed to begin with. – Brad Larson Feb 26 '14 at 20:38
  • @BradLarson - only similar situation that comes to mind is the case of a robo-accepted vandalism edit. Vandalism is not so uncommon, so I suppose that it might get accepted if enough robo-reviewers are around at the time. – Hitodama Feb 27 '14 at 8:36
17

I was going to post this as a comment to sth answer, but the length of the post quickly escalated.

Let's start by examining the effect of the behavior described above. First, the original offender that suggested the edit. Basically he flooded an possibly overflowing review queue with a proposal that even if approved will not resolve the problem in the post. So, to be clear he:

  • wasted the time of the reviewer.
  • actually gave more visibility to the spam by promoting it for view to anyone that will browse the review queue.
  • failed to do the only needed thing: remove the ads

That said, I would conclude that this is a behavior we wouldn't want to encourage.

So, now that we have concluded that proposing an edit wasn't the action we wanted - because we want the spam to disappear - let's analyze what the approvers did by approving that edit.

If the above assumption stands, the approvers also enforced an unwanted behavior. By approving the proposal, they are basically saying "Good work, keep doing that. Here are 2 rep points for your time". So again:

  • they failed to remove the spam
  • they encouraged keeping spam

I believe that anything that can help correcting this should be welcome. The real question is another: it is easy enough for the staff to put such restriction in place? As said, we do want to avoid such situation arise, but we must do a cost/benefice estimate to see what we can do to prevent it. This is a question only the staff can answer: is the cost for implementing a system that will work to discourage the above behavior reasonable (I would expect that adding some new audit questions isn't to difficult)?

That said... rejecting that edit cost nothing and will in turn teach the offender a lesson: flag spam and do not polish it. So, I totally agree on your horror.

  • 1
    +1 A good explanation of why turd-polishing is bad. – michaelb958--Reinstate Monica Feb 26 '14 at 22:32
  • given that they were robo approved at least in the mentioned case I would somewhat doubt the "wasted the time of the reviewer". But in general, yes it wastes the time of those reviewers that review to make the site(s) truly better. – PlasmaHH Jun 6 '14 at 14:57
4

Can we put this edit in as an audit and fail those who approve.

I think we need to manually pick some of the cases that have come up on meta and use those as examples as the current examples are usually rather easier to spot.

-11

It's technically not the point of a suggested edit review to judge the correctness/appropriateness of an answer. Especially if the edit suggestion is purely syntactic and not semantic, like in this case. If a bad post gets improved formatting the resulting situation is still not worse than before the edit suggestion.

The edit is unnecessary, but not such a big problem. Also I don't think something like that happens often enough to make a whole new class reviews necessary.

Implementing special measures addressing suggested edits on spam posts seems like a waste since only a tiny percentage of spam posts will ever get a edit suggested.

  • 1
    I really do not understand why this answer has been such heavily downvoted... +1 from my I see valid points – user221081 Feb 26 '14 at 10:08
  • 15
    it's not about spam posts, but about careless reviewers whose brainless approvals damage regular, non-spam posts and, which is maybe even worse, mislead users making edit suggestions. I am still thankful to reviewers who blocked few my edit suggestions 2 or 3 years ago and triggered suspension that taught me how to do edits responsibly and thoroughly – gnat Feb 26 '14 at 10:09
  • 3
    I do not think about this suggestion as a way to prevent unneeded but not harmful edits, or a way of spam prevention. I think about it as a way to catch robo-reviewers, and they really are a problem, problem you ignored in your answer. Everything else is just a side effect, one I find positive, but side effect anyway. – Mołot Feb 26 '14 at 10:10
  • but also It's technically not the point of a suggested edit review to judge the correctness/appropriateness of an answer. Especially if the edit suggestion is purely syntactic and not semantic, like in this case. which is why I gave +1. I agree the robo-reviewers were and are the problem but how many suggestions were there already trying to stop them... Also Implementing special measures addressing suggested edits on spam posts seems like a waste since only a tiny percentage of spam posts will ever get a edit suggested. reflects the truth. – user221081 Feb 26 '14 at 10:13
  • 11
    @mehow did you read what was in suggested edit? blah-blah-seo.com/buy-email-database Buy Email Database if this slips through reviewer eyes, do you really want them continue reviewing? – gnat Feb 26 '14 at 10:18
  • 12
    Also, it is responsibility of everybody, all the time, to recognize and flag spam. Of course suggested edit system is not a tool for that. But also newest question list is not a tool for that. There is no specific tool for that. We do it when it happens, no matter which tool on this site we are currently using. – Mołot Feb 26 '14 at 10:28
  • @gnat yes I have seen it and honestly that should have been picked up. But the only action taken by mods should be to BAN those who accepted, who suggested. Less robo-reviewers may increase the amount of suggested edits to be handled but would definitely increase the quality of moderation. – user221081 Feb 26 '14 at 10:39
  • 1
    @gnat: The problem are robo reviewers which approve edits which damage regular, non-spam posts. This suggestion here for some reason focuses on decent edits to spam posts, which seems to miss the point. Suggested edits that damage regular non-spam posts would make obviously much better reviews to combat the problem of those edits getting approved. – sth Feb 26 '14 at 10:43
  • @Mołot: Wouldn't robo-reviewers be much more effectively caught by audits that check for the actually harmful edit approvals instead of audits that check for appropriate spam flagging? – sth Feb 26 '14 at 10:46
  • 11
    @sth reviewers who aren't capable to recognize "Buy Email Database" in the text (no matter what are the diffs) would better be discovered and suspended. We have enough reviewers already there, queue is always empty or single digits, it's not like in CV queue – gnat Feb 26 '14 at 10:46
  • 1
    @sth maybe you're right, maybe you're wrong, but that aspect is not evaluated in the answer we are commenting at all, now is it? – Mołot Feb 26 '14 at 10:48
  • 7
    @mehow "It's technically not the point of a suggested edit review to judge the correctness/appropriateness of an answer". True, but only because the point is to judge the appropriateness of the edit. And in this case the appropriate action was to flag, not to give more visibility to the spam by editing it. Allow me a paradox here... Suppose someone post a vulgar image on an answer but the image isn't visible because he used the wrong syntax in the post: do you really want that some genius correct the image tags and also force at least 4 different reviewers to view it? – Hitodama Feb 26 '14 at 11:54
  • @SPArchaeologist this is one specific case. How often do you reckon this actually happens? My point is that no matter if it's spam or not if someone is robo-reviewing we should ban them from reviewing. Have more to do but walk-away at the end of the night knowing we done some quality work. – user221081 Feb 26 '14 at 12:02
  • @mehow Yes, I got your point about "how often does it happens". What I was saying is that this is not an excuse for approving the aforementioned edit. I agree on the other point - a staff member should elaborate on the cost of implementing an audit specific for this case: only based on the cost you may decide if it is something worthy to have. – Hitodama Feb 26 '14 at 12:17
  • 9
    Someone sees that a burglar has broken the window to his neighbor's apartment, so he sweeps up the glass. Bystanders pat him on the back and give him a thumbs up. No one calls the police. – Josh Caswell Feb 26 '14 at 19:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .