-14

In How can we determine whether an answer used ChatGPT? (SO specific; it might be different for other sites), moderators initially didn't disclose how they detect AI generated content, probably in order not to make it too easy for those wanting to circumvent the AI content ban. Later they divulged that they do not only rely on tools and that indeed they mostly train themselves on ChatGPT content. But in the implementation phase they didn't have complete information about the ground truth, i.e. they wouldn't really know for sure if deleted content really was AI generated. Indeed the company seems to strongly doubt that, given their alleged ban on banning at the end of May the exact wording of which might be published soon.

If only there was a direct way to solve this controversy and get to know how accurately moderators can really detect AI content.

Oh wait. I think there is. Take some random questions from before ChatGPT, take a random answer from them (negatives) and add an equal amount of ChatGPT-generated answers to these questions (positives), then let volunteering moderators classify them.

The true false positive rate for bans would still depend on the true rate of AI generated posts (which is unknown) and on the exact way how moderators focus their attention on answers and some context would be missing, but it would end the discussion about the accuracy of detection tools and it might give the company an acceptable off-ramp (if only they knew how accurate moderators can detect AI content).

It's some work to set up and collect the data, but only a fraction of the work that was done during implementation of the ban and I think it will help being more safe about any statements like "we can accurately detect AI content". It would probably be worth it.

It's also reported that it was offered by moderators multiple times to the company, but such a test doesn't depend on the company.

10
  • 36
    Moderators have already offered to do exactly this multiple times in response to staff doubts. Jun 29, 2023 at 16:27
  • 3
    It won’t give them an off ramp if it ends up being shown that they were wrong to begin with (aka what most of us already suspect to be the case). I doubt they would risk losing face like that
    – Clive
    Jun 29, 2023 at 16:29
  • 29
    So, how much time and effort do you feel that the completely unpaid volunteer moderators should be putting out in order to convince the people paid by the company as to what the ground based reality is, when the company completely refused to provide even a single case of where SE thought that moderators were wrong in their evaluation as to what was AI-generated content?
    – Makyen
    Jun 29, 2023 at 17:22
  • 1
    The linked post belongs to Stack Overflow Meta. Bringing that post here and claiming it represents the "moderators" voice looks misleading. This post might belong to Stack Overflow Meta or should be edited. Remember, this site is Meta Stack Exchange, there are ~180 communities each of them having their own moderators
    – Rubén
    Jun 29, 2023 at 17:23
  • 15
    Note: What you're asking to have done is hard and a lot of work to do in a realistic manner. While moderators do look at the specific text content of a post, we rely on a lot more information than just the content of a single post when evaluating if a user is posting AI-generated content. Thus, for any such test to be realistic it would need to create a body of work and history for the user. That's ... really difficult to do in a realistic manner.
    – Makyen
    Jun 29, 2023 at 17:28
  • 1
    @doppelgreener "offered to do exactly this" Right, I remember. But it's independent of staff, one could do it without any staff, although the company should also have an interest in finding that out. Jun 29, 2023 at 19:38
  • @Makyen "how much time and effort do you feel" I feel that if you do thousands of these checks anyway, you could also do a hundred more just to be sure that the whole work before wasn't in vain. If it's a fraction of the work you do anyway it's still tolerable. For example we also do regular tests in the review queues, don't we. On the other hand you can say I don't want to but then at the same time you cannot say that you know that it works flawlessly. If you ever wondered about that I feel you should at least consider it. Jun 29, 2023 at 19:49
  • @Rubén-PeopleFirst "The linked post belongs to Stack Overflow Meta." I'm not so familiar with all the SE sites. I will edit and mark that link as SO specific and add a note saying that for other SE sites it may vary. The question itself isn't specific to SO, it applies to all sites that banned GPT. Jun 29, 2023 at 19:54
  • 8
    You missed my point. It is significant additional work to take such a test, but it's more work to create the test. We, moderators, know we're not perfect, and we make mistakes. We can identify a very small number of such actions which were incorrect. I first mentioned that in a public comment back in January. SE has made the extraordinary claim that moderators are massively wrong in identifying AI-generated content. Extraordinary claims require extraordinary evidence. SE has refused to supply any examples of incorrect action. The burden of proof that it's huge is on them.
    – Makyen
    Jun 29, 2023 at 21:49
  • 13
    To try to put it a bit more clearly, SE claims in their data post that moderators are incorrectly identifying 230 to 260 posts as AI-generated every single week for months, but have not pointed to a single example. If that error rate was true, there should be a flood of examples. That's a very extraordinary claim that's based entirely on a metric only they can even check. Yet, there are other metrics that you (and others) have identified which indicate there's a considerable amount of AI-generated content that has never been identified and deleted.
    – Makyen
    Jun 29, 2023 at 22:13

4 Answers 4

15

When the strike started, the position by SE was that it is impossible for humans to detect AI-generated content with sufficient accuracy. That was an absolute statement SE made and on which the new policy is based on.

Of course it would be interesting to do this kind of blind test. But there are several problems with this:

  • it's trivial to cheat if you want to. You can just search or look up the existing posts to figure out if they're human- or AI-generated

  • it's an unfair test if you want to judge the ability of mods to detect AI-generated content as it's much harder than the real thing. You do not have patterns of behaviour and multiple posts, which tend to reveal a lot.

It would be a very large amount of work to set up this kind of test and to have a sufficient number of mods take part. And in the end SE could just say that the mods might have cheated, so it doesn't mean anything. Or the accuracy is not as good as you'd ideally like, but it ignores that the test is much harder than judging real situations with multiple posts and additional information. So it's hard to determine the right target accuracy here because the test is different than the real thing.

In the end SE has been frustratingly stubborn on this point and argued in circles. So nobody really wants to spend that amount of effort on something that might not even convince SE to do anything. If SE were to set this up I think you'd find quite a few moderators to take part, if only out of curiosity.

2
  • If "SE could just say that the mods might have cheated, so it doesn't mean anything" then really nothing means anything. Then the strike lasts until one side gives up. I see your point about the missing context that makes it harder, but I think it still might be impressive as a lower bound and ideally we should judge content by its own. If someone just needs to set it up so that others take part in it, that might be possible. Jun 29, 2023 at 20:11
  • 1
    You'd have to get some human users to write totally new answers. Still, there's a host of other issues. Jun 30, 2023 at 0:49
12

The evaluation of whether something is AI generated or not is not done with no context. Moderators look holistically at the user posting it, prior posting patterns and other information. (Yes, I'm guessing, but it's an informed guess and a current moderator can correct me if I'm wrong). Once a moderator has a strong suspicion that a user is posting content they didn't author, it doesn't necessarily mean they immediately suspend that user. Sometimes a private warning is enough. Sometimes it just puts a user or the post on the mod team's radar. It's a judgement call and that is hard to measure with the sort of test you're proposing.

I wonder, what is the point of trying to precisely measure the false positive rate of a human moderator when the consequences of a false positive are very low and mostly reversible? Is it good for a community to cater to people who post a lot of answers that are so poor quality they can be mistaken as generated by AI?

The problem isn't how well or how poorly moderators detect AI-generated text. The problem is much bigger and a lot harder to solve than that.

3
  • Context would only make it easier but on the other hand content should mostly be judged by its own. One could see the results as a lower bound on the accuracy. And for a newly registered user that posts the first few times you haven't much context. But what is the real problem in your opinion? I didn't understand it completely. I actually like the writing style of GPT, apart from the content problems I wish I could write like that. The only problem is always the content and moderators can hardly detect content problems. But what is the bigger problem? Jun 29, 2023 at 20:03
  • 2
    @NoDataDumpNoContribution There’s a difference between judging the quality of content and attempting to determine if it was generated by AI. Plenty of human written content fails my quality standards. The bigger problem is that the mechanisms Stack Exchange uses to curate content don’t scale up to the traffic larger sites are getting. If they did, AI wouldn’t be a problem. I despise the tone of ChatGPT’s writing. It’s flavorless pap. It’s fine for generating code and maybe for writing prompts I guess.
    – ColleenV
    Jun 29, 2023 at 20:40
  • @NoDataDumpNoContribution "[...] content should mostly be judged by its own." <-- Content is judged on it's own for quality. If there's any thing suspicious that suggests it might be AI generated, then checking other heuristics to confirm/deny the suspicious activity is a lot better than quasi-guestimate about something in a complete vacuum. Imagine user A that uses the website since 2018 and always wrote quality content that feels like AI (for whatever reason). Should that user be punished for it? Based on your comment, yes! Because, by it's own, it seems generated by AI. Jun 30, 2023 at 0:33
3

It seems very hard to design an experiment along these lines—one whose results would be useful. I think both sides would reject such an experiment's conclusion, and argue poor experimental design, were it to go against their expectations.

I think we can all agree that some posts are blatantly obviously in need of deleting/moderation. E.g. yesterday, I saw a user at Ask Ubuntu who listed what to do if the OP's operating system were Windows and macOS; this is not the kind of error a human would make. So there's no real point in testing these super-easy instances—we can just assume the mods will get these correct 100%.

So do we make the test harder? We can include human-AI coauthored posts, ranging from 0% AI-generated to 100% AI-generated posts. And include things like AI-translated and AI-polished answers. And all the answers will be correct and useful.

But if we do this:

  • Does the difficulty reflect the real situation?
  • Would moderators deliberately moderate more conservatively knowing what's at stake?

I think "woah, can we slow down and think about this more carefully?" not because of obvious cases of AI plagiarism—it can be treated much like spam—but because of posters who use ChatGPT to improve their posts and make their research more efficient. Or non-native English speakers who get ChatGPT to translate and/or correct their posts. This, in my opinion, is what needs testing.

Besides, even assuming flawless AI detection, are human-AI coauthored (useful, correct) answers also getting deleted? This is not a binary (AI vs. non-AI) situation.

And besides, it would be unprecedented and rude for Stack Exchange to subject its moderators to such testing.

1

The network has ~180 sites. Stack Overflow, besides being the network flagship and the greatest "victim" of the company, is the community that has suffered the worst from the flood of posts that occurred after the launch of ChatGPT by a very large difference.

Stack Overflow has the largest number of community moderators. Also, it is the main referral for new users for other communities and the source of multiple community lead initiatives. What make you think that SO moderators have not already done their best to keep SO healthy? What make you think that the company has proceeded correctly?

Seriously, bringing here such a post without talking first with SO mods looks to be a great disrespect.

On the other hand, even though many SE users are SO members, they have agreed that each community has its topic, scope and workings. The SO workings can't be directly transferred to all communities as the circumstances differ, starting with the activity volume.

  • What led you to think that this should be discussed here?
  • How will the other communities get enough posts to test their AI-generated-text detection abilities?

Please, let's be a bit emphatic with SO moderators and all network moderators participating in the moderation strike.

Related

2
  • "looks to be a great disrespect" Hi Rubén. With all respect I think you're overreacting and your attacking ad hominem instead of focusing on content. You just attack the author of a question you happen to not like. I couldn't disagree more but I also do not wish to comment anything anymore. Jun 29, 2023 at 21:34
  • 8
    Don't play the victim. I haven't attacked you, I haven't mentioned any attributes as a person or stuff unrelated to the act of posting here and now. The criticism is centered on posting here-and-now content without considering the people being criticized.
    – Rubén
    Jun 29, 2023 at 21:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .