A machine can flag spam automatically. Can it do better?

Question

Since this has been posted, we've done a few things:

1. We now cast four flags on posts that pass an accuracy threshold (current 99.9% historical confidence)
2. SmokeDetector now casts the first flag on all autoflagged posts. That should help address concerns about transparency in the flag queue & on post flag timelines.

3. There's a userscript exposing auto flagging activity inline on the post. It's pretty:

   

4. We're trying to get an RSS feed of posts deleted with autoflags for chat consumption, but chat RSS is fraught with peril.

TL;DR: Charcoal is the organisation behind SmokeDetector. Since January 2017, we've been casting up to 3 flags automatically on posts that our systems are confident are spam. We'd like to increase that to 5 automatic flags on posts we're even more certain of to reduce the time spam spends alive on the network.

Who are you?

Charcoal is a user-run organisation that is primarily responsible for the spam-detecting bot, SmokeDetector. Over the past four years, with the aid of SmokeDetector, we've looked for spam on the Stack Exchange network to flag and destroy manually. In January 2017, with the blessing of Stack Exchange, we started running an "autoflagging" project, wherein our systems automatically cast up to three flags on a post if they're confident that it's spam. If you missed that happening entirely, we wrote a meta post on Meta Stack Exchange - or there's a slightly more concise explanation on our website.

How's that been going for you?

Good. We currently have 215 users who have opted into the autoflagging system (you can sign up too, if you're interested). We've flagged around 30 000 (29 592) posts, of which the vast majority (29 526) were confirmed spam - that's 99.7% accurate.

What are you proposing?

We'd like to expand our autoflagging system. At present we cast up to 3 automated flags on posts we're confident are spam; we'd like to increase that number to 5 flags on a subset of those autoflagged posts where we're even more certain they're spam.

Why?

Just so we're up-front about this: this is an experiment. Ultimately, we're trying to do these things:

• Reduce the time that spam spends on the sites before being deleted;
• Lower the number of humans who involuntarily have to see or interact with spam.

Increasing the number of flags we cast automatically on spam should accomplish both of these things:

• Automatic flags are near-instant; manual flags take multiple minutes to be cast - that means that increasing the ratio of automatic to manual flags results in a shorter time before 6 flags accumulate and the spam is deleted.
• Automatic flags are not cast by a human. Fewer humans, therefore, are forced to see/interact with the spam.

The data we have backs this up. In terms of time to deletion, we saw a significant drop in the time it took to delete spam when we started our autoflagging project. Take a look at this graph from the meta post on the subject for an excellent visual representation of that. Before we started autoflagging, spam spent an average of 56 hours per day alive across the network; with autoflagging in place, the average is much less, at around 7 hours per day.

What would this change mean for sites?

If this change goes ahead, these things are likely to happen:

• It will only take 1 or 2 manual flags from users to spam-nuke an autoflagged post, instead of the current 3. Posts that are not autoflagged will, of course, still require 6 flags to nuke.
• There may be an increase in posts spam-nuked entirely by Charcoal members, who may or may not be active on the site.
• You will see a reduction in the time spam spends on the site before being deleted.
• Fewer humans will have to involuntarily see each spam post.

The last two of those are indisputably good things. The first two, however, are more controversial, and are the reason we want to have a discussion here on meta before we make this happen. What follows are the major concerns we've seen, and what we can do about them or why we don't think they're an issue - we'd like to hear your thoughts.

The major thing we're looking for out of this is a reduction in time to deletion. The following graph shows how long spam currently spends alive on the top few sites; we're hoping to see a moderate reduction in the average times, and a significant reduction in the top outliers.

The following graph is from an experiment we've been running over the past week, casting between 1 and 5 flags randomly on each post matching the settings we're considering.

In raw numbers, that's this:

PostCount  FlagCount  ATTD      StdDev  CommonMax
173        1          191.474   243.63  678.73
166        2          98.7831   127.88  354.55
167        3          69.2814   156.62  382.51
194        4          33.3196   61.46   156.23
177        5          13.5254   12.43   38.39

PostCount is the sample size; FlagCount the number of flags cast on each post in the sample; ATTD the average time to deletion, and CommonMax is the maximum of a 95% confidence interval. The major takeaway from these stats is that we're likely to see a ~5x drop in the average time to deletion, and a ~10x drop in the outliers.

Accuracy & false positives

Spam flags are a powerful feature that need some care in applying correctly. This is a concern that came up when we originally built the autoflagging system, so we already have safeguards built in.

• We only flag a post if we're more than 99.75% sure it's spam. (Technically, the precise certainty varies by conditions set by the users whose accounts we use, but it's always above 99.75% - more detail on that on our website).
• If the system breaks down or bugs out and starts flagging things it shouldn't, all Charcoal members and all network moderators have access to a command that immediately halts all flagging activity and requires intervention from a system administrator to re-enable. Outside of testing, that kill-switch has never had to be used.
• We never unilaterally nuke a post. There are currently 3 manual flags required in addition to the automatic flags to nuke a post; this increase proposal still retains at least one manual flag.

We also make sure that everything has human oversight at all times. While only 3 humans currently have to manually flag the post, there are always more users than that reviewing the system's decisions and classifications; if a post is flagged that shouldn't have been, we are alerted and can alert the relevant moderators to resolve the issue. Again, this is very rare: over the past year, we've flagged 66 posts that shouldn't have been, compared to 29 592 spam posts (that's 99.7% accurate overall). We allow users to set their own flagging conditions, provided they don't go below our baseline 99.75% certainty. We recommend, however, a higher value that has a certainty of 100.00% - those who set their conditions below that are likely to see more false positives flagged using their account.

This proposal decreases the required manual involvement to nuke a post; to compensate for that lower human-involvement barrier, we will correspondingly increase the required accuracy before casting the extra automatic flags. For example, we currently require 99.75% accuracy before casting autoflags; we could require 99.9% accuracy for 4 autoflags, and 99.99% accuracy for 5 autoflags. (For reference, humans are accurate 95.4% of the time, or 87.3% on Stack Overflow - those are stats that jmac (a former Community Manager) looked up for us last year when we started autoflagging).

In the rare event of a legitimate post getting autoflagged, we also have systems in place to ensure it isn't accidentally deleted and forgotten about. Multiple people review each post we catch, whether it's autoflagged or not, and classify it as spam or not; if an autoflagged post is classified as not-spam, the system posts an alert to chat to let us know. That lets us ping the necessary people to retract their flags, and keep an eye on the post to make sure it doesn't get deleted.

To make it starkly clear how accurate this could be, here's a visualisation:

That's a chronological representation (left-right, top-bottom) of every post that would have been flagged under the settings we're considering for 5 flags, and whether they were spam (green squares) or legitimate (red squares).

Community agency & involvement

As I said earlier, this proposal reduces the required manual involvement to nuke a post. Since Charcoal members also cast manual flags on top of the automatic flags cast by the system, that's also likely to increase the number of posts that are nuked entirely by Charcoal members, without involvement from users who are active on this site. Some posts already have 6 flags cast on them by Charcoal (including autoflagging and manual flags), but the proportion of posts that applies to is likely to increase.

We don't think this is an issue in terms of subject matter expertise: the spam we see on the Stack Exchange network is broadly the same wherever you go - you don't need any subject matter expertise or activity on a particular site to be able to tell what's spam and what's not. We do, however, recognise that it's possible that a site's community may want to handle its own spam; if that's the case, we're happy to turn the autoflagging system off on that site or to retain it at its current levels. Just post something on your site's meta and link us to it.

What now?

We want to increase the number of automated flags from 3 to 5 to reduce the time spam spends alive on the network. We'd like to hear your thoughts. We appreciate that quite a lot of the stuff we do at Charcoal is fairly invisible to the sites, so we want to be as open as possible. If you'd like data or specific reports, let us know and we'll try to add them in - we already have a lot of reporting around autoflagging, so it may already exist. If there's other stuff we can do to explain or to help you make an informed decision about whether you want this, drop an answer or a comment on this post. Charcoal members will be hanging around this post to respond to your concerns, or you can also visit us in chat.

Answer 1

I see a very big difference between moving to 4 or to 5 automatic flags. I wouldn't have any issue with 4 flags, but 5 flags means that only a single user is required to nuke the post. This would probably still be safe if that single user organically stumbles upon the spam post, in this case the odds of a user falsely flagging a false positive are very low. But that isn't the case I'd be worried about. As far as I understand, you keep links to auto-flagged posts in various places, which is pretty much necessary for this system to be administered. A single overzealous user monitoring flagged posts and adding their own would be rather dangerous, if they are careless.

The odds of actually getting a non-spam post nuked are incredibly low if you assume that all flags are independent. But this assumption isn't necessarily true if you have a list of possible spam posts that users use to find posts to flag.

I also see seriously diminishing returns here. Making sure that most spam is removed in a few minutes is a worthy goal. I'm not convinced pushing it below a minute is worth additional risk.

One other concern of mine would be how Charcoal handles the far more murky area of self-promotion. My understanding is that there aren't any automatic flags that target posts that are likely to be self-promotion, but not blatant spam. But this is an area that confuses real users as well, so it would be particularly dangerous in combination with the 5 flag option.

In general, I like the project and do support automatically flagging posts on all sites. My major concern is that it isn't transparent at all unless you know it exists. You're using real user account to flag, a moderator that doesn't follow the main meta might never know there are any automatic spam flags, as they only ever see real users. This is not a new concern, but I think transparency is even more important if you increase the automatic flagging in this way.

Answer 2

The biggest advantage I see in how the Smoke Detector project has operated thus far is its transparency: one of the most frustrating aspects of using these sites is how difficult it can be to determine precisely what happened in a given scenario (much less why it happened) - even for moderators, there's so much data that's either unavailable or more often available but not clearly connected that making sense of the sequence of events that underpin a complaint can be a real chore. For ordinary users, deciphering something odd they encounter can require knowledge of hidden tools and a 4-year degree in Meta Minutiae.

But Smoke Detector? It's all right there, out in the open and heavily crosslinked. What got reported, why / who reported it, what the post said and who wrote it, when it was created, when it was deleted, who flagged it... A massive record of spam in all its gory detail, laid out for review by anyone with an interest in doing so.

Entrypoints

With that in mind, it might seem ironic that most of the concerns expressed here relate in some way to accountability. But it's not: for all the data (and powerful searching / filtering exposed in the web interface), Smoke Detector suffers from one of the same problems that afflicts so many otherwise-powerful systems on Stack Exchange - discoverability!

• There's no obvious way for me to see (as a user or moderator browsing my site) what - if anything - has been identified, flagged or deleted by the Smoke Detector project.

• If I do come across a spam-deleted post, there's no obvious indicator that Smoke Detector had any involvement in this. The most likely route for me to discover this would be to observe a comment from someone not normally active on my site, but this rare occurrence lends itself to observing only activity on borderline posts; my perception of both the scope and worth of the project is inevitably skewed.

Fortunately, I think these concerns can be addressed fairly easily. My initial recommendation would be to generate a monthly report for every site on which Smoke Detector has identified posts, summarizing statistics for the post(s) identified and linking to a search that provides the full list of posts for perusal by those who might be interested. Such a report could be posted on each meta site on a regular basis, thus providing both transparency and awareness of the project without intruding on the day-to-day operations of the site itself.

For moderators researching specific complaints, a userscript to look up SD data on any spam-deleted post should suffice to allay concerns about false-positives going unnoticed. This would potentially require a bit of extra support from the Smoke Detector project, but I don't think it would need much work.

Thoughts?

Answer 3

The thing that concerns me about 5 auto-flags is that sometimes users are wrong and now it only takes one user to spam-delete a post. A statistic that would help me better evaluate this is: on each site, how many organic spam flags are dismissed or disputed? If even site users sometimes get it wrong -- and they do, particularly on questions where product links in answers are expected -- then allowing a single user, native or (especially) visiting from Charcoal, to cast binding spam flags gives me pause. But I have anecdotes, not data; can we get data? (It would have to come from SE, I know.)

I'm glad to see your team working on tools to help moderators monitor what's going on. I'm going to need push notifications, not just pull (the dashboard), which I understand is possible.

Answer 4

Art mentioned this in a comment:

However, while some people are more zealous than others in flagging

I dunno whether bad luck, or good that it occurred, this happened yesterday:

The red color marked user is a well known overzealous flagger. I've seen many of their spam flags being retracted just a few seconds after flagging and then using a NAA flag. The blue marked user is a highly respected member of Charcoal (or atleast, I respect them a lot), and who is quite good at taking decisions.

In this case, the overzealous user flagged the post almost immediately. A few seconds later, the trusted user naa- reviewed the smokey post. After that the first user retracted the flag immediately and added a naa flag.

This wouldn't be a problem usually, but it would make a difference if that user is given the task of being the 6th flagger on a post with 5 auto flags.

Some suggestions to prevent this:

• Check if a user has lots of invalidated feedback and flag it from their account, so that you can be sure that they are not the 6th flagger.
• Check if a user has lots of invalidated feedback and stop/scold/educate the user to be more careful.
• Use just 4 instead of 5, but this has the disadvantage (from the data) that it doesn't help much. So I'm not much attached to the "use 4 instead of 5" statement.

You people are quite brilliant at this, and can figure out a better way. My only request is to somehow make sure that a overzealous users isn't the deciding factor, if you are flagging 5.

Good luck!

Answer 5

Smoke detector is a fantastic project, it has inspired many of us to do other type of moderation bots.

However once Brad Larson stated in this comment on a question that I made

I'm not terribly excited about a chatroom gathering together to coordinate flags on content in order to automatically delete it

I realize that the main objective remove spam quickly is important but I think we that coordinate flags/actions in chatroom has a great responsibility to listen to moderators on specific site (they have been elected) and we need to assure the members who do not participate in chat/flagging are never targeted or gamified.

With that said, these are some ideas to leverage the great capability of SmokeDetector to find spam and keep community moderators involved.

• Get moderators involved on each site, listen and let'em decide how many auto-flags you can cast, as a side note I would be ok that moderator directly cast flags if they think that it's the best solution, they will be responsabile since community has put their trust in them.

• Instead of using other users "real" accounts, create SmokeDetector flag account eg. SD_FLAGGER1, this will both remove the idea of user participating to get "free" flags and badges (I know they get very few flags but just the idea taste bad for the community). More important, it will be clear to moderators who is flagging the post.

Keep up your excellent work!

Answer 6

I feel that the increased automated flags should be opt-in with the agreement of the moderation team, and community involved. At a broader level, I think the individual communities ought to have a say when any project may affect how it runs.

To me, the right workflow here would be to let the moderators be the point of contact, have the option of discussing and making a decision as representatives of the community and/or bringing it on meta. I don't feel comfortable with letting any organisation (Other than SE) making this decision for us.

And I think this is a big part of the issue here - that it takes things out of the hands of the individual community to a bigger extent than I'm comfortable with.

There's been a case of voting without knowing you did, I'm assured that its unlikely but its possible someone can opt in and forget. I'd prefer some more direct human input, personally.

While getting rid of individual cases of spam is a worthwhile endeavor, it also means that mods are less likely to notice and nuke the accounts. It's not just about time to deletion.

So, if it's a small community with a ton of spam, and they want it? This might work.

It's worth considering whether we're optimising for paperclips. As a moderator I have a ton of questions about this:

A. Does optimising for shorter lifetime for spam reduce the chances of them getting picked up on search engines?

B. What are the benefits for community outside the outcome of spam getting deleted faster?

C. How does this compare in time to deletion to fewer auto flags, or more precisely, what's the minimum number of flags at which auto flagging is useful?

D. As a downside, does this discourage people outside Smokey from flagging spam (since they don't get into the habit of it)?

Answer 7

I am grateful to the Charcoal and SmokeDetector developers and users for their substantial, commendable efforts to fight spam on Stack Exchange sites.

However, in relation to this specific proposal, I want to comment on a specific point, which is made twice by the OP:

It will only take 1 or 2 manual flags from users to spam-nuke an autoflagged post, instead of the current 3. [...]

There are currently 3 manual flags required in addition to the automatic flags to nuke a post; this increase proposal still retains at least one manual flag.

(Emphasis mine.)

A reduction to two manual flags would still permit human peer review, i.e. confirmation or disputation of a human's decision by another human. This seems to me to be the minimal essential safety check against humans going rogue. It is less safe than requiring three manual flags, but at least it does not forego peer review altogether; and of course it would cut the humans' workload by (about) a third. I would be supportive of this.

A reduction to one manual flag would eliminate this crucial safety check. I would not be supportive of this.

Edit: I see that Monica Cellio has already made essentially the same point, and so has Mad Scientist.

Answer 8

Related to the issues of transparency, perhaps an automated comment under a post that's been autoflagged would be helpful in understanding and sighting Smoke Detector activity in situ so to speak, as opposed to needing to make a search of Meta Smoke or a monthly report, as suggested by Shog.

Of course these tools could still be made available, but an auto comment sits with the post and is more easily accessible. It would also show any potential manual flaggers, that it's been auto-flagged. To assist user's not knowing about the project the auto comment could include a link to the Charcoal wiki or a canonical post on Meta, perhaps the FAQ, when/of Smokey is added to the FAQ.

Answer 9

^{Note of warning I have not read all of the answers here. I'm sure what I wrote is duplicated somewhere.}

I like smokey, it works amazingly well. Yes I would trust it to fully automatically delete spam.

I'm going to point out some "issues" and then speek to them (we are on meta after all).

There are two issues with moving from 3 flags to 5.

1. Do we really need the extra speed? On the site I moderate, we do not get that much spam. (Smokey has auto flagged 8 posts, total). With the user base, and the amount of questions we get per day we really don't need a machine gun on our spam. A friendly bear watching over the site seems sufficient for us.

2. Humans are far more likely to get something wrong then a machine. Yes smokey is correct somewhere around 99.7%, but what is the human error rate? (I've made mistakes, clicked the wrong button etc.) Now with only one human vote, you've removed the human self check. A human error could get by with just one person voting. I don't like that idea. (There are users on BSE I would not trust with that responsibility, and I know there are some on your sites too.) We would essentially be giving whoever came across that auto flagged post moderator powers.

I understand that smokey would be auto flagging only the most obvious spam posts. Posts that SE should be filtering out.

In that light I have no problem with smokey moving to 5 flags. I trust smokey, and if it is that good at finding the really bad spam I think there is no way of a user to mess up. Say we have a spam post, if the user thinks it is not spam (somehow) then the post waits in the queue for the next guy.

About the speed, It's spam. The quicker we get rid of it the better. We may not "need" the speed, but there is no harm in having spam disappear quicker.

Go for it.

Remember... Only YOU Can Prevent Spam Posts.

Answer 10

Suggestion #1: site transparency

Provide a clear list of what was flagged and removed as spam (either as a separate review queue, or just a list). This way the site community will have an option to gamify getting another Gold badge for reviews review the spam removal history for accuracy.

Suggestion #2: user transparency

Make it crystal clear to a user that their post was removed as spam, with an obvious link of how to dispute that (and a solid workflow procedure to dispute). This may not address 100% of false positives some of which may be from new/unregistered users; but it will at the very least hopefully reduce the amount of grief and upset from false positived users.

Answer 11

I'm not against this idea - but using actual user accounts to do botted automation is a dodgy hack that should be rectified before pushing this further in my opinion. Fair enough while it was an experiment - but now this is getting a little bit beyond the experiment phase.

How reasonable would it be to create a SmokeDetector user that has the privilege to cast multiple spam votes? I think if we got that happening we would fix the transparency issues highlighted in other answers.

Once this transparency is achieved, I would feel comfortable with the idea of bumping it up to 4 votes (ie, 2 manual votes still required). Once we trial that for a while, running the graphs again should give us insight as to how we should proceed.

Answer 12

My concern is alerting users in general to this. Perhaps there is advice to avoid getting one's question/post inadvertently marked as spam. This may be especially important to any links posted ... Such as 'do' this or 'don't do' that, etc....

Answer 13

I also oppose having it down to only one human. Keep it two or more.

Do spam-flags put the post/answer on a review queue? That would (I think) get those last two votes in more quickly, if they are needed.

Answer 14

On the single piece of "autoflag transparency", a thought, or several

Rather than needing accounts with special privileges such as "multiple flags per post" make, and register system-wide, 5 accounts. CharcoalAutoflag1 through 5, say, or come up with something better. They would perhaps need extra flagging volume, but nothing special flag-per-post. They simply pile on in the same way that real people do. Indeed, if they are accurate, they could even get their extra flagging volume the same way real users do, by having their flags classified as helpful.

If it's:

99% spam, CharcoalAutoflag1 flags it.

99.5 CharcoalAutoflag2 also flags it

99.75 CharcoalAutoflag3 also flags it

99.95 CharcoalAutoflag4 also flags it

99.99 CharcoalAutoflag5 also flags it (or 99.999 or however many decimal places you want to put there.)

I agree with the general idea that the automated flagging seems good from a spam-removal view, but if it's pretending to be real users doing the flagging, it's not at all apparent if a post is being flagged by people or bots. Having whichever account correlates to the highest level of certainty post a comment to the effect that "Charcoal's automated spam detection evaluates this post as 99.5% likely to be spam" or similar message would also be helpful (perhaps you are doing this, I've hardly spent the evening combing though the whole backstory to formulate the commentary on a small part of the process I had not known about until today.)