11

This is not a duplicate of Should Stack Exchange have a spam filter? (Does it already and I’m not aware?) which is asking if there's already a system in place.
I am asking for a review of the existing structure, which relies heavily on volunteers, and breaks abysmally when those volunteers withhold their efforts.


I'm aware of our current anti-spam structure - Charcoal, Charcoal-HQ & Metasmoke. I'm also aware of large explanatory info sources about this - A machine can flag spam automatically. Can it do better? and similar.
Also ref recent questions from other users who weren't aware of this setup - such as How can we handle spam faster?

And, before anyone else links to it - this is the SE blog telling us how good the spam protection already is - How does spam protection work on Stack Exchange? - written by one of Charcoal's leaders, ArtOfCode.

What the astute reader may also notice is

  1. In addition to SpamRam [about which I can find very little] it already relies on Charcoal, and
  2. When Charcoal is off you get this, multiple times a day…

enter image description here

Click for full size

What I wasn't aware of until only this week, is that it seems to be entirely volunteer-run.
That the volunteers don't currently wish to engage is already part of a larger issue, which I don't want to add to here.

It's time Stack Exchange had its own spam filtering.
How could this be practically achieved? Does the company have any plans to implement a viable structure, rather than relying solely on the goodwill of volunteers?

The first thing that springs to mind is that we already benefit from a system that works reasonably well under normal circumstances. So, my first suggestions would be…

License the structure & pay for it.

Alternatively, or in addition,
Change the number of human flags to just three, or even fewer, if detection heuristics put it above say 98% spam already.

I am not, btw, advocating we just hand this whole task, lock, stock & barrel to any form of 'AI' & remove human checking.

I'm open to other suggestions.

28
  • 5
    They do already have a system - see stackoverflow.blog/2020/06/25/…
    – greg-449
    Jun 24, 2023 at 11:58
  • 38
    For the record, Philippe was recently promising to compensate Charcoal for our hosting costs. That's obviously a fair bit less than the objective cost of actually maintaining the system; we normally have a substantial number of volunteers who donate their time and know-how in the form of updates to the patterns, the actual code, our documentation, etc etc.
    – tripleee
    Jun 24, 2023 at 11:59
  • 5
    @tripleee - thank you for the additional information. I have to admit, I didn't know who you were three days ago - all this 'behind the scenes' discovery is very recent for me; even though I've been here 9 years. Thank you for the work you & the other volunteers do. :)
    – Tetsujin
    Jun 24, 2023 at 12:17
  • 17
    And, at the risk of restating the obvious, github.com/Charcoal-SE is entirely open source. If the company wanted us to run ops for them, it's not clear how you would compensate a loosely-knit network of volunteers on at least four continents to render a service, or create a legal agreement to govern this arrangement. Given recent devolopments, it is uncertain that the volunteers who are currently on strike would be willing to return.
    – tripleee
    Jun 24, 2023 at 12:26
  • 2
    Amusingly, being a single point of failure and much too efficient was an early reservation I had of charcoal. And don't forget even if they ran the software, and did flagging themselves, they'd still need to have the 'rules' for detection volunteers provide. This won't work without the community, and... a couple of years on, maybe that's a good thing. Jun 24, 2023 at 12:30
  • 29
    That screenshot is precious. It proves the strike works, and should be sent to the board of managers of Stack Overflow Inc. They should see what their actions cause, and even them with their narrow view should understand people don't want to visit such sites, and they will lose lots of money if this will be going on. So no, I'm against SE making their own anti spam system. Jun 24, 2023 at 12:33
  • 18
    Re "Stack Exchange needs its own spam system": Sure. Just let ChatGPT generate questions and answers. It will get the numbers up. Jun 24, 2023 at 12:55
  • 15
    Around 6AM British Summer Time (05:00 UTC) there was a massive wave of spam. I was honestly impressed at the sheer quantity of it. I think spammers have taken note that the effectiveness of Stack Exchanges spam filtering system has decreased somewhat and are taking advantage. I have no current way of investigating how they're doing it, whether by multiple IPs or other means as I'm effectively locked out of looking at PII (Personal Identifiable Information) such as emails and IPs by the strike, but I'm worried how much SE relies on community effort and they don't realise.
    – Mokubai
    Jun 24, 2023 at 13:11
  • 3
    'Why are we reliant on the goodwill of a volunteer-based system?', people like free whenever possible; regardless of the real cost. --- and some people have worked for free for over a decade: meta.stackoverflow.com/questions/tagged/moderator-resignation that's the business model, the company pays for the equipment and supplies the software (and intervenes where moderation is insufficient) and everyone helps themselves for free (or by being subjected to ADs). There is a Satisfaction Survey.
    – Rob
    Jun 24, 2023 at 13:42
  • 16
    @Tetsujin Old fashioned community flagging needs a motivated community to pitch in. The sort of folks who would be unhappy with SO inc's current actions. We'd be able to manage say charcoal being down a week or so - but it would be a mix of people like me who are less involved in these things, and people who would be on charcoal anyway. Every solution still needs the community somehow Jun 24, 2023 at 15:06
  • 2
    @JourneymanGeekOnStrike - I should have stuck with my original title, "Why doesn't SE have its own effective spam system?" to which we all already know the answer. It was cheaper to let someone else do it.
    – Tetsujin
    Jun 24, 2023 at 15:12
  • 4
    ‘the existing structure, which relies heavily on volunteers, and breaks abysmally when those volunteers withhold their efforts’ — which is what gives the strike any chance of being effective. Why are you proposing to take that away? Jun 25, 2023 at 9:41
  • 2
    Your latest edit basically describes how Charcoal's autoflagging feature works. Even with a higher threshold than you propose, it gets false positives from time to time. Some of this could be remediated with a redesign to reduce overlap between detection reasons (for example, a post in Chinese will trigger two heuristics, "majority non-Latin text" and "Chinese text", which for legacy reasons count as distinct). It submits multiple flags automatically when a particular threshold is exceeded, rather than reduces the number of required flags, which of course it can't; the net effect is the same.
    – tripleee
    Jun 26, 2023 at 7:58
  • 8
    I love that image of the homepage in your question because while SO inc is out chasing AI in any possible way, like implementing some fancy AI formatter, it can't even detect the exact same title being posted 5 times in less than 5 minutes...
    – Tomerikoo
    Jun 26, 2023 at 14:18
  • 2
    And the system is also so helpful, limiting us to 25 flags a day... so if there is a community member trying to assist by flagging they should quickly be outrun by the spammers 😢 Jun 26, 2023 at 14:21

3 Answers 3

45

I'd argue that it would be very difficult to replicate Charcoal without a fair amount of effort.

You can automate flagging and removal. However the 'rules' used to detect spam are manual, community-generated, and verified, and since spammers are consistently creating new patterns, you'd still need to be able to accurately identify spam, and verify to ensure there are no (or at least minimal) false positives, and take the appropriate action.

While an 'official' tool would have certain advantages, such is potentially being able to use 'historic' information and PII such as email and IP addresses, and since SE's so very enthusiastic about GenAI and machine learning maybe they could write one, The company hasn't yet, and you'd still need training data - both historic and current to make better matches. Which might involve needing people to flag anyway.

Basically, it comes down to whether the cost of winning over the community is cheaper than writing a new tool or letting the network get overrun by spam.

Or we have a tool that literally costs the company nothing but treating its most enthusiastic members right (but of course, support would be welcome) and has proven itself to be so efficient the network suffers spam outbreaks when it's down. And its devs have worked with the company in the past.

It's their call, really.

1
  • 19
    It’s trivial to replicate all the Charcoal systems as they exist today. It’s expensive to keep them up to date as spam evolves.
    – Undo
    Jun 24, 2023 at 19:07
36

SpamRam is a system that I designed which works on an account and network level. From what I understand, it has received some upgrades over the years, but it's essentially fed from user and moderator activity such as flags, account destruction and other events. It also has some "honeypots".

None of these systems function unless humans are pointing at things and saying "That's bad!" and people with the most privilege that a community can bestow are following through on those reports.

I don't think it's even feasible to create something autonomous at the scale of Stack Overflow because the investment required would be indistinguishable from a seed round to develop that kind of product.

0

At the time of this edit, the question state is closed as a duplicate of Should Stack Exchange have a spam filter? (Does it already and I’m not aware?). I don't think that it should be closed as a duplicate of such a question, as this question already mentions SpamRam, the Q&A public platform system used to fight spam, among other things.

It looks that the purpose of this question is to propose to the company to take measures to have an anti-spam system that doesn't depend on volunteers, more specifically, that doesn't depend on Charcoal and the moderators, understanding moderators as community members that have earned moderation privileges, in this case, the privilege to flag posts as spam.

Why limit the public Q&A platform to getting its anti-spam free of volunteer participation? Why don't you go full throttle and suggest using GenAI to make the public Q&A platform free of volunteer posts?

To be clear, the above are rhetorical questions to express that if the company does something like that, then the company might still own brands (Stack Overflow, Stack Exchange), domains (stackoverflow.com, stackexchange.com, superuser.com, askubuntu.com, etc.), software and hardware, but Stack Overflow and the Stack Exchange network, as they exist nowadays, will no longer exist. Period.

Eliminating the flag privilege and preventing volunteer participation led by Charcoal members to fight spam in the Q&A public platform will change the core of Stack Overflow and Stack Exchange.


Explanation of the hypothetical suggestion

By "going full throttle" to "take advantage of using GenAI", I mean to use it to post questions and answers; no "Ask Question" or "Answer" forms for user input will be required anymore, and spammers will not be able to post spam directly.

Using content crawlers / miners and GenAI, the company might scan real existing communities, and code repositories, among other sources, find out what questions are in the wild and answer them, posting questions and answers on their platform.

The company will no longer need a community team to handle the relationship with question-and-answer posters and moderators.

If the company finds a way to do it right, it might not need an anti-spam system...

... but they should find a way to handle AI hallucinations.

Please note that I'm not suggesting the above to the company; it's just a thought exercise to help understand that the company should not prescind volunteer participation in fighting against spam.

7
  • 8
    Because, that's not how "genAI" works and because genAI is a completely inappropriate tool for this application. "GenAI", or generative AI, is a very specific type of AI (i.e., machine learning). "GenAI" has very specific applications, which, frankly, it does a good job at. But those very limited jobs are not general purpose artificial intelligence (and don't included even generating factually accurate output text, at least at the current level of the technology).
    – Makyen
    Jun 24, 2023 at 17:26
  • 8
    So, yes, AI/machine learning could be quite useful (and some effort has been made to use AI/machine learning by people associated with Charcoal, with mixed results), but it's not "genAI". Please don't fall into the hype that "genAI" is universally applicable to every situation
    – Makyen
    Jun 24, 2023 at 17:26
  • @Makyen, I'm sorry, but I am not following you. Probably it's my fault, and the breach between my thoughts and words is causing a misunderstanding. I edited the answer to clarify the use of GenAI that, in a rhetorical sense, could someone might suggest. I'm not saying that this hypothetical suggestion is viable or has sense.
    – Rubén
    Jun 24, 2023 at 17:42
  • 7
    Rubén, I don't understand how the company using genAI to post questions and answers has anything to do with preventing/combating spam, unless you're also saying that the company should completely stop taking questions, answers, and comments from users. If there's an input box that people can use to have their custom text shown to other people, particularly shown to the general public, then there will be spammers and trolls using it to post inappropriate things. The overall system needs to be able to handle that by removal/prevention/sanctions, or just not allow any input from the public.
    – Makyen
    Jun 24, 2023 at 17:51
  • @Makyen Yes, the hypothetical full-throttle suggestion implies that the company removes the input boxes, hence, SO/SE will no longer exists, even if the URLs are still used to publish content.
    – Rubén
    Jun 24, 2023 at 17:52
  • btw, no-one ever mentioned preventing community participation at all.
    – Tetsujin
    Jun 25, 2023 at 9:52
  • Well, sure, removing user input from the equation will remove spam. But then we might as well stop eating food to reduce waste. It's not a solution.
    – Cerbrus
    Jul 3, 2023 at 8:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .