228

A recent statement from the SE company to the press suggests that company spokespeople (the CEO or whoever writes his statements) believe that moderators have been relying on ChatGPT detection tools to find and suspend people who post plagiarised AI-generated text:

Stack Overflow ran an analysis and the ChatGPT detection tools that moderators were previously using have an alarmingly high rate of false positives. [...] We stand by our decision to require that moderators stop using the tools previously used. We will continue to look for alternatives and are committed to rapid testing of those tools.

The purpose of this meta post is to make, very clearly and distinctly, a point that has been made before (in the middle of longer meta posts) but perhaps not noticed by SE upper management. Making it the title of a meta post will make it much harder to ignore or miss.

Moderators do not (and never did) rely blindly on automatic GPT detector tools.

The decision to delete content or suspend users is always taken seriously, and experienced SE moderators are not foolish enough to rely blindly on ... any automated tool, really. Just like we don't automatically suspend everyone who shows up in "suspicious votes" tools, we don't automatically delete every post that automatic GPT detector tools claim is AI-generated, nor suspend every such poster. We use a variety of heuristic methods to judge such cases, together with good old human judgement. Wouldn't it be ironic if we used an automated process to ban people who used an automated process to create their posts? We don't, and we never did.

Moderators have been secretive about the exact methods used to detect AI-generated posts. Yes, but that's the same with any type of site rule-breaking. We're also secretive about the exact methods used to detect voting fraud. If we announced publicly how we detect any kind of plausibly-deniable misbehaviour, people who wanted to misbehave would simply adjust their technique to continue misbehaving while avoiding our methods. That's why we haven't announced the exact process we use to figure out whether someone is posting GPT copy-pastes. But the process is not, and never has been, "run it through a GPT text detector and then immediately reach for the ban hammer". If the company thinks it is, that's a major misunderstanding which probably contributed to the disastrous recent announcement. Perhaps this PSA can help to clear up the misunderstanding? As a relatively short post making a single clear statement, it could easily be conveyed by CMs to upper management (hint, hint).

Don't believe me? Check it out: Stack Overflow moderator sideshowbarker has handled ~10,000 ChatGPT-flagged posts but never used any detection tools to do so.

15
  • 9
    Is there a way to ensure this question, and the other one previously posted on this topic, are reliably relayed to the press? Maybe they will want to rebound on this. Maybe. Jun 5, 2023 at 16:11
  • 4
    @FrédéricHamidi Most press organisations have a contact line. Alternatively, many journalists can be contacted personally: Twitter used to be the go-to, but these days email, ActivityPub or LinkedIn might be more effective.
    – wizzwizz4
    Jun 5, 2023 at 16:12
  • 18
  • 1
    To play the devil's advocate: If usage of detection tools isn't a major factor in detecting AI generated content, one could maybe leave it out without much impact on the success quality. What is the value of these detectors really? Maybe they are only giving leads? Jun 5, 2023 at 17:31
  • @Trilarion, I actually tried one of these on some of my more terse, to-the-point answers from way back and it consistently ranked them as 80~100% written by a human... even with a single sentence followed by ten lines of code. I was quite impressed. Then again, the confidence percentage is there for a reason, and using AI to defeat AI cannot mathematically suffice alone. Jun 5, 2023 at 17:34
  • 1
    @FrédéricHamidi What else do you want to use? AI is almost the same as statistics and whatever the mods use additionally to these programs is also just statistics. Do you want to come to my home and watch me typing in that answer instead? I really need to read more about Platon. Last heard about him in school with the allegory of the cave. It seems relevant to the modern time. Jun 5, 2023 at 19:20
  • @Trilarion Re: your first comment; My assumption is that the automatic tool has high recall, and the moderators' judgement has high precision. Neither one on its own is good enough. Jun 5, 2023 at 20:11
  • 2
    @TheGuywithTheHat This would mean that you should use an AI detector to automatically screen for promising candidates first and then use your personal moderator magic to only pick those you are sure of is something like this. That's probably what the mods mean and it's also likely what the company doubt that it works very well. The question is simply whom to believe. My gut feeling is that the company is mostly wrong but it may have a point or two. The policy needs to be reverted, maybe refined and the communication must be strongly improved next time. Let's keep the fingers crossed. Jun 5, 2023 at 20:20
  • 5
    @Trilarion We've already been leaving them out. That's the whole point. Most of the SO mods who have been handling the bulk of these flags never used any sort of detectors. Others of us have consulted them, especially in the early days before we knew from experience how badly inaccurate they were, but we never relied solely on them (or even primarily), and we definitely don't anymore. That's literally the whole point of Rand's post. Thus, your comment makes no sense. We don't use detectors to screen, either (except possibly indirectly via user flags, and we validate each manually). Jun 6, 2023 at 11:02
  • @CodyGray-onstrike I took Rand's statement as detectors are used but not exclusively and your statement in this comment as the are not used now. This is not completely identical to me and I find the various statements (which appeared only in the last days) slightly inconsistent. I'm learning something new everyday about the use or not-use of various tools. I wonder if working on flags that may have been raised with the help of detectors is kind of outsourcing the the usage of detectors. All in all I think now that AI detectors aren't an issue because mods don't use them anyways. ... Jun 6, 2023 at 12:17
  • @CodyGray-onstrike ... So I wonder what is really left of the controversy. I don't expect detectors to be an issue anymore because as you say, mods don't use them anymore. But what exactly is the issue that the company has then? I'm sorry for asking again and again for more details. In principle I'm just waiting for what happens next. Jun 6, 2023 at 12:20
  • 7
    @Trilarion What we've got here is failure to communicate. The company apparently believes that moderators are relying heavily on detectors (hence this meta post), and instead of asking us, they pushed out a policy that we can't moderate AI-generated content at all any more. Hence the strike. Jun 6, 2023 at 14:13
  • 6
    @Trilarion: There are a large group of moderators who don't all practice the job in exactly the same way. It is entirely unsurprising to me that they describe their work in ways that are slightly different from one another.
    – Kevin
    Jun 6, 2023 at 14:32
  • 12
    The issue, @Trilarion, is that the company has told us we're not allowed to use any heuristics to justify the deletion of AI-generated posts, including detectors (which nobody cares about, b/c we didn't use those anyway) or any other "tells" that we think have a high degree of confidence. Staff has been unable or unwilling to provide us even a single example where they do not share our confidence. This is the central problem: we want to remove content from our sites that we think is beyond reasonable doubt generated by an AI. The company is saying no. The issue of detectors is a red herring. Jun 7, 2023 at 9:55
  • 9
    While that's trivially true, @Kevin, the primary issue here (for staff) is what Stack Overflow moderators are doing, purely because of the vast scale at which SO operates. And we know well what SO mods are doing, simply by asking them (which staff didn't do, and still hasn't). Active SO mods are, in general, remarkably well-aligned in our moderation and flag-handling strategies. On this particular issue of AI-generated content, we're even more closely aligned than anything else. Besides, certain mods have been specializing in this, amassing staggering amounts of experience & consistency. Jun 7, 2023 at 9:59

1 Answer 1

11

My impression is that moderators and users are using a kind of "mental Bayes' Theorem" to detect AI answers. We answer questions like the following, repeatedly updating our probability:

  • Is the answer written in the style of a native-English-speaking professor with 10+ years professional experience? (Is the user a native-English-speaking professor with 10+ years professional experience?)

  • Is it written with a level of personal closeness and friendliness unexpected between users of Stack Exchange?

  • Is it unnecessarily verbose?

  • Is the answer wrong?

  • Can I generate a similar answer using ChatGPT?

  • Has it been significantly edited?

  • Does it contain correct citations and/or links?

  • If it contains code, does it also contain the code's output?

  • Does the author appear aware of what site they're on?

  • Does the author appear to know other users exist?

  • Is the author a new user? Or do they have a history of AI-suspected answers?

  • Has the author meaningfully responded to comments?

  • Does the answer contain typos, slang, smileys, jokes, punctuation and grammar errors, etc.?

  • Does it contain images, tables, markdown, etc.?

  • Does the author write as if they're a human (e.g. "I'm not sure, but...", "This worked for me...")?

  • Does its overall structure resemble that commonly generated by AI?

  • Does it use phrases commonly used by AI?

(Oh, and: Does the answer say e.g. "ChatGPT wrote this"?)

So if the answer says e.g. "as an alternative to @user213's answer, ...", it's unlikely to be AI generated, so we increase our mental probability towards "human generated", and if the answer says e.g. "As an AI language model, ...", it's unlikely to be human generated, and increase the probability towards "AI generated".

Given that I've tried detecting AI answers myself now, I feel you can, in most cases, reasonably deduce whether an answer is or isn't human-written this way (assuming a binary yes/no answer to the question), especially if there's a lot of anomalies one way or the other. This doesn't always work; there are going to be cases where the result is "I don't know for sure", especially for short answers where there's just not enough data.

4
  • 4
    There is also the answer rate (say, an average of 7 minutes for 10 lengthy answers) and number of answers per day. Especially compared to the same before December 2022 (if available). For instance, two lengthy answers separated by a few minutes. Preparing answers offline (or in several browser tabs) is theoretically possible, but very unlikely (e.g., it requires more work and organisation). Jul 2, 2023 at 9:59
  • 2
    Oh great, a ready-made checklist for LLM plagiarisers to help them avoid detection. Jul 2, 2023 at 11:21
  • 4
    @user3840170 Unlikely. There is a mixture of positive- and negative-polarity items here, and things that most plagiarizers won't be able to do anything about. (It doesn't seem practical to feed ChatGPT all your previous answers and tell it to mimic your writing style, for example.) By the time enough effort is put into tweaking the text to make it suitably "human", it would have been more easily written from scratch. Hopefully, anyway. Jul 2, 2023 at 12:00
  • 5
    @user3840170 If anyone manages to train a model according to this checklist so that it produces answers that aren't wrong, correctly cites the sources, and responds appropriately to comments, everyone will be much better off since the experts will suddenly have lots of spare time on their hands while the answers will be exactly as useful as ever before. Sadly, LLMs still have a decades long way to go from "glorified autocomplete" to "artificially intelligent domain expert".
    – TooTea
    Jul 3, 2023 at 13:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .