This isn't a regular, let's stop curse words post. Please read on.

This post was precipitated by a report that came to my attention from the Smoke Detector in Charcoal HQ which is a bot that, I am grateful, alerts users to possible spam and abusive posts on the site.

This post could easily have been filtered out on the keywords alone, let alone how they were put together.

This is a link that will not be able to be observed by most people
WARNING: NSFW, de-linkified to avoid accidental clicking.

It is so offensive it really caused pain for the handful of people who read it. It is taking a while to recover. This post is at the very least a fantasy of written child pornography, at the worst a story of horrific child abuse.

I had difficulty finding someone or enough people to flag it to delete it. Which leads me to the question.

Can we increase the filters of what is posted on these sites to assist in preventing this kind o f thing occurring again rather than wait to be alerted to it after the fact?

This post isn't about preventing curse words on the site, it's about setting a filter to prevent extreme NSFW and material that is so abhorrent it should not be too difficult to filter. As the post I am using as an example, would have been relatively simple to filter on. Or at least make the poster task of posting such matter more difficult. Within the content of that post there were multiple words and phrases that could easily be flagged and especially when put together in such number, there is no reason such content should make it on the site. At least if content needs to be re-written in such a way that it is not so graphic, it will assist in keeping the site cleaner from the outset.

FYI I am forwarding a link to this post and the parenting post to SE employees for forwarding to the appropriate authorities.

  • 5
    This is the same as profanity, technically speaking - I don't think SE are going to add any extra filters. Like any other site which allow people to post freely (even YouTube), bad stuff can be posted, and people should know this. That is what we have flags for, and in case of repeated offense from same IP, SE can impose IP block. – Shadow The Princess Wizard May 17 '16 at 13:15
  • 1
    @ShadowWizard Respectfully disagree. This is much, much worse then profanity. – Magisch May 17 '16 at 13:15
  • @ShadowWizard it's not the same as profanity, it's child pornography/ and/or child abuse – Nobody May 17 '16 at 13:16
  • 8
    But we can't tell that from words only. Words can be manipulated and it's impossible to catch them all, not to mention false positives. /cc @Magisch. – Shadow The Princess Wizard May 17 '16 at 13:17
  • @ShadowWizard I guess. Im a little shaken, forgive me. :/ – Magisch May 17 '16 at 13:18
  • @ShadowWizard you're not giving it a chance, as you've closed any discussion on ti – Nobody May 17 '16 at 13:18
  • 4
    @ShadowWizard This is a whole new category of stuff. Profanity needs filters; this needs reporting to the police. – ArtOfCode May 17 '16 at 13:19
  • 1
    besides that post had MANY keywords that should've prevetned it being on the site – Nobody May 17 '16 at 13:19
  • And it had SO much in it, it shouldn't have made it on the site! It can easily be filtered out – Nobody May 17 '16 at 13:19
  • OK, reopened, let's see how it will go. But like I said, don't think it matters in the bottom line, purely technically speaking. – Shadow The Princess Wizard May 17 '16 at 13:19
  • @ShadowWizard if you can read the post, dear God, you would understand. It was clearly unfit for the site – Nobody May 17 '16 at 13:21
  • @Yvette no doubt in that, and never doubted that. I'm just coming from the technical point of view, that is all. – Shadow The Princess Wizard May 17 '16 at 13:25
  • I'd be interested in what the procedure of dealing with content this illegal is btw. Do we just pass it to a CM. Do the viewers have to file a police report? Is it even legal to allow 10k users to still view this? – Magisch May 17 '16 at 13:38
  • 1
    It has to be purged from the database @Magisch. Until that time (maybe for police investigation, etc.) it should be locked away beyond reach of normal users. I hope SE will report it itself since it will be hard for most of us to make a police reports that actually sticks. – Patrick Hofman May 17 '16 at 13:39
  • @PatrickHofman that is why I linked it here and then put this link and that link in an email to SE. so it wouldn't be lost. I closed the tab and was like doh but the link was still in chat – Nobody May 17 '16 at 13:44

After some discussion, @AndrewT pointed me in the direction of @Shog9's Triage Post over on meta.so and this looks like a possible way to implement a catch for posts which score way too high for something like obscenity, sexual terms or other relevant words or phrases.

The idea is to send all posts that do pass a threshold to a staging point where only mods and 10k-ers can see them, allowing a decision to be made on validity before the post getting to the front page and possibly indexed by Google.

The thresholds could be set high enough that sites like Parenting or Health can avoid impact to valid questions, as these sites do get questions on sexual health, abuse, violence etc.

The basic triage diagram on that post shouldn't need a huge amount of change - procedurally this is very similar:

enter image description here

You'd just have the two options of full visibility on homepage or close and delete.

  • 1
    Should we change this to a feature request? – Nobody May 18 '16 at 8:41
  • 1
    @Yvette yes, this way it has bigger chance to draw official response. (for good or for bad, it can also be declined) – Shadow The Princess Wizard May 18 '16 at 12:53
  • @ShadowWizard meh declined or no, it makes it official and saves another post. – Nobody May 18 '16 at 13:10
  • 1
    Reposting what I said in chat: "Realistically... (1) you'll never have a perfect system, especially with such a small sample to test against, and (2) it honestly isn't worth the development effort. This is the first time I've seen something like this in a few months of moderating SO and years of running Smokey. It's rare enough to qualify as "meh"" "I'm not saying that it isn't a serious case, but I am saying that it's not practical to block these automatically" – Undo - Reinstate Monica May 18 '16 at 13:12
  • @Undo won't hurt to go through the official process, it will put the matter to bed for the next year or so at least. – Nobody May 18 '16 at 13:14
  • I unaccepted the answer, as the community clearly doesn't want this. So I'll let the community float the answers with votes. – Nobody May 29 '16 at 2:48

The problem here is not the filter. No matter how strict you set a filter, there will always be a way to go around it. Or, in the worst case you end up blocking a lot of false positives.

The problem in my opinion is the activity on some sites. Some sites simply don't have enough (moderating) users that are on the site very frequently. If you would have had that, one or two abusive votes would have triggered a moderator and it would have been deleted before anyone could see that. With 6 community flags it would have been deleted too.

If you can't bring up 6 users to flag a very abusive post, the community has some work to do on their commitment. I would bring this up on their meta in the first place. Second, maybe on low-traffic sites, they need to lower the number of required flags to get an abusive post deleted.

  • 1
    well I re-reported it on meta tavern and went to parenting chat and it was empty. You raise a good point. I had difficulty finding someone with the power to delete it or enough users to assist – Nobody May 17 '16 at 13:37
  • 6 votes from any user within the community? If you go to the Tavern and ping some active users, you have 6 within no-time. – Patrick Hofman May 17 '16 at 13:38
  • ah any community. Good to know. But they have to join that community first? – Nobody May 17 '16 at 13:43
  • They just need to click "Join this community" in order to create a profile, so who care a user is a regular on SO? I have created quite some accounts just to be able to flag. – Patrick Hofman May 17 '16 at 13:44
  • 7
    "I am not sure if the person was posting a real problem or not." And then you expect a filter to make a better decision than you? – Patrick Hofman May 17 '16 at 13:47
  • 5
    Also - this is nowhere near as bad as some of the stuff we see on Parenting. Smoke Detector or ordinary user flagging is all it needs. Parenting is nowhere near big as SO or one of the main sites, so things may exist for a while longer than you may be used to, but we are all in multiple times a day, and as Patrick said - a highlight on meta will speed things up :-) – Ro-Reinstate Monic-Alsop May 17 '16 at 14:21
  • 2
    Usually SmokeDetector and the high rep users catch them very quickly - before the majority of folks see them. The problem with your suggestion is that all the words in there could be used in perfectly valid questions - we cannot blanket ban all words related to sex, although in this case SmokeDetector did flag on some of the content, which in addition to the manual flagging got it caught. – Ro-Reinstate Monic-Alsop May 17 '16 at 15:06
  • 1
    As to police intervention - discussion is ongoing. Problem is: it's text based, from an unregistered user, from an IP address that may or may not be spoofed... – Ro-Reinstate Monic-Alsop May 17 '16 at 15:07
  • 4
    Side point: FWIW, I don't think insufficient moderation (community or diamond) is an issue on Parenting specifically. I'd argue that for a site with complex, difficult issues, it's very well moderated. I agree that community action is much more likely to work in cases like this than anything automated, but in some cases, it's just not possible to totally eliminate exposure to problem content for some period of time. – Jaydles May 17 '16 at 15:19
  • 6
    Also, if one willingly opts to go and look at the content Smokey reports, they have to accept the fact that they have opted to be among the first few to look at bad/very bad posts, and that requires one to develop thick skin (like the mods). It is a choice a user makes. – Infinite Recursion May 17 '16 at 15:59
  • 1
    Pekka - please no. That has so many downsides I can't think where to start (not picking on SO specifically, but on large groups being drawn by flags. The phrase mod-swarm was coined in various chat rooms specifically because of the impact...quite often negative) – Ro-Reinstate Monic-Alsop May 17 '16 at 17:08
  • 2
    almost all the abuse is targeted at one of the mods, and it is thoroughly horrible. It gets caught, and binned. The post Yvette is talking about is nasty, but reads like a teen wrote it. It got caught, and binned - ideally could have been faster. – Ro-Reinstate Monic-Alsop May 17 '16 at 17:22
  • 3
    The majority of the community never has to see them, because the system works pretty well. – Ro-Reinstate Monic-Alsop May 17 '16 at 17:23
  • 1
    @Rory I'm thinking about a super specific queue for offensive-flagged stuff from smaller sites where the only job is to determine whether the post is unambiguously offensive. If five users vote yes, the post gets hidden until a mod can cast a final verdict. There wouldn't be even the need to link to the question, just a yes/no button. But maybe there's too many complications – Pekka supports GoFundMonica May 17 '16 at 23:35
  • 1
    @Yvette: Not being harsh, it's just a fact. The current SE moderation model works fine for flags raised by users organically. Use of bots or SEDE for flagging is different. As a regular user, this Meta post would be like this: I am a user on parenting, I saw a post that should not have been posted, I flagged it and it took some time to handle the flag and the post to be removed. So I propose the software filters are increased. Refer what the mod said - meta.stackexchange.com/questions/279369/… – Infinite Recursion May 18 '16 at 6:58

I'm really sorry that you saw this post, and I hope that you, and everyone else who saw the post, is OK.

From a technical standpoint, I'm not quite sure how the filtering would work. It seems like the filtering works 99.99% of the time, and this post represents the 0.01% of instances where the filtering didn't work.

Some people have mentioned false positives (e.g. someone posting on Parenting.SE to ask a real question about child abuse). I don't think this would be a problem. If a new user has a serious question, but they word it using inappropriate language, then a moderator/community member will need to edit it anyway (and probably contact the authorities if there's an indication that someone could be in danger). So I don't think there's anything wrong with putting such a post in a queue where it will be reviewed by a moderator.

  • I totally agree with this – Nobody May 17 '16 at 23:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .