56

It's been about a week since the community and moderators lost the ability to effectively moderate AI content. Given that time, we should start to see some data/patterns emerge that could tell us (and the SE Staff) how effective (or not) the previous policy was. What objective trends have you seen that could demonstrate that the previous heuristics that were being used were effective?

We're all aware that there is potential for false positives in any detection, but can we perhaps come up with some data that helps demonstrate the rate at which we were successful?

3
  • I'm not sure how you want to prove that there were only a low number or none falsely as AI generated detected but instead written by human posts, by comparing a state where AI generated posts are moderated vs. one where they aren't. Less complaints maybe. But otherwise I think that this "experiment" doesn't give much insight into this specific value. We can maybe learn something about the abundance of AI generated posts or how they fare, i.e. what their real quality is, but not really about the false detection rate, because we don't moderate them anymore. Jun 7, 2023 at 10:46
  • 2
    Be careful sharing ChatGPT-identifying patterns. If those identifying telltales land in the wrong hands, they can be used to patch the source, and make ChatGPT identification all the more harder. By their actions since the Monica times, SE does not introduce themselves as trustworthy allies, do they.
    – Levente
    Jun 7, 2023 at 12:42
  • 3
    @Levente I'm sure SE already has complete access to all of the flag comments used to identify ChatGPT content, so I'm not that concerned with that aspect personally. Jun 7, 2023 at 13:09

1 Answer 1

65

I'll hopefully have more data as time goes by, but here's one that stands out as obvious. For those that haven't seen my other posts on this topic, note that I've flagged over 1,200 answers that I've suspected as being ChatGPT-generated in whole or part. Most of these have been on Stack Overflow, with some other sites as well.

In reading literally thousands of Stack Overflow answers, I've noticed that there's a phrase that GPT sometimes uses in them that indicates at a very high signal-to-noise (signal being GPT; noise being false positives) ratio.

No, I'm obviously not going to tell anyone outside of a very select, private group what that phrase is, since people will actively avoid it in the future. I honestly haven't even told any moderators about it up to this point. They know it's one of the phrases that I believe points to a possible GPT-answer (but it would never be the sole indicator that I rely on, either), but they don't know how much I believe it is such a great GPT-indicator over most all others.

Edit: And yes, as @MissSkooter points out in the comments, this makes the following data unverifiable to most readers here. You simply have to trust me (or choose not to) based on the reputation I've developed flagging and posting here. I'm happy to share the data with SE Staff, if requested, though. Or they can just look at my "Saves" list for "Suspected ChatGPT" on SO. I've made notes regarding the phrase in some recent (at the time of this answer) private notes. I have to assume these aren't that private ;-), and I'm okay with the SE staff reviewing them.

This particular phrase occurs in only 251 Stack Overflow (undeleted) answers ever. I suspect it's even often left in answers where a user has attempted to edit the content to hide ChatGPT use.

Eight of the 251 answers that use this phrase were written yesterday.

Now three of those answers (yes, three yesterday) were written by a single very high-rep user that I have trouble believing was using GPT (or would even need to). There are also things that I see in that user's answers that indicate to me that it was either human written, or at least edited. So yes, we have false positives on it already, but they are easy to spot for multiple reasons. I typically can eliminate those that we believe are false positives before even flagging.

If this phrase was never used by ChatGPT, then we should theoretically see a fairly consistent use of the phrase over time. This assumes, of course, that it isn't a new idiom that has become popular recently, which it isn't, other than ChatGPT seemingly making it popular.

But the rate of usage yesterday, even throwing out the known false-positives, was roughly 133 times higher than the average SO day.

Some more analysis:

Stack Overflow has been receiving answers for 5,424 days. With ChatGPT’s introduction on November 30th, 2022, we had 5,235 “pre-ChatGPT” days and 189 days since ChatGPT’s introduction.

  • Prior to November 30th, this phrase had been used 218 times. That’s an average rate of 4.16 occurrences of the phrase every 100 days.

  • Between November 30th and May 31st, the phrase was used 20 times at the rate of 10.81 uses per 100 days. Keep in mind that there have been many more uses in this timeframe, but the (perhaps vast) majority have been deleted under the previous “detection” policies. This increase just shows that there are likely some ChatGPT uses in the timeframe that we have missed, and by eyeballing a few of them, I can confirm that this is the case.

  • In the last week (essentially since we stopped moderating the use of AI), the phrase has been used 13 times, which is a rate of 185.71 uses per 100 days. There has actually been at least one more, but the user deleted the answer when I pointed them to the AI Policy.

Reputation analysis1:

  • For the 10 most recent answers that used this phrase before ChatGPT was available:

    • The median reputation of the user who posted was 15,435.
    • Trimming the lowest rep and highest rep 2, the mean was 25,266.
  • For the 10 most recent answers that used this phrase as of today:

    • The median rep of the user who posted was 44.5.
    • The mean, again trimming the outliers, was 98.

    1 Of course, reputation grows over time, but not this drastically over a six month span.
    2 The outliers were so extreme (on the high end) that the results would have been meaningless if they weren't trimmed.

    So is this heuristic biased against newer users? Or is it that new users are more likely to use ChatGPT?

Let's look at the five suspected ChatGPT uses today. In reviewing these answers, I do believe that they are GPT, and I probably would have flagged most, if not all of them:

Of the five users who posted these answers:

  • User #1: Had only the one answer, but it does appear to be GPT and I typically would have flagged it. It just has too much in it that indicates it was likely written by ChatGPT.

  • User #2: Had 5 answers on May 16th, 4 answers today, and all appear GPT. One of them has a comment that the answer didn't help, but another of the (likely GPT) answers was accepted and upvoted. With four total downvotes, one upvote, and one accept, the net rep is still an 18 point gain.

    Another user noticed that this user was using AI and commented on several answers. I believe they attempted to downvote them as well, but they ran into the same problem I've seen -- If a community member suspects someone of using AI, they can only downvote 3 answers before the "targeted voting" algorithm rolls back votes.

  • User #3: Has 4 answers today. Two appear to be GPT. Another is code only, and the other appears to be user-written, but could simply be edited GPT output (closer analysis of the code not completed). I personally tend to suspect that if a user used ChatGPT on two answers in one day, they likely used it on all four.

  • User #4: Has 3 likely-GPT answers dating back to April 7th. Two of them have comments indicating they have issues.

  • User #5: Had 3 likely-GPT answers yesterday; perhaps another on May 12th. All four answers have comments indicating potential issues with the quality.

So following that one phrase could have unearthed a likely 17 (at least) AI-generated answers that now remain. Seven of those (so far) have been identified as having potential problems. While the new policy seems to say that users who repeatedly post "very low quality" content can be reprimanded, the moderators that I've talked to seem to lack the confidence that these meet the bar of "repeatedly" or VLQ.

And under the new moderation policies, "writing style" cannot be used to show that something might be AI-generated, so these answers will all remain.

Side-note: While this particular phrase is the most drastic example, there are certainly other "GPT-tells" that have increased in the last week.

19
  • 24
    Regenerate upvote
    – Glorfindel Mod
    Jun 6, 2023 at 18:06
  • 4
    This is interesting, but I can't shake the feeling of you haven't really given us anything testable here. I get why you don't want to share the phrase, and I'm not asking you to, but there is a lack of evidence. To be clear, I agree with you about the signs of drop in quality and appreciate your post, but I just don't feel like it's particularly convincing the way it is Jun 6, 2023 at 18:08
  • 13
    @MissSkooter You are absolutely correct. It's unverifiable for readers here, but could be verified by SE Staff, and I would be happy to share the data with Staff privately. Jun 6, 2023 at 18:14
  • 5
    So the phrase is something other than "As a large language model, I..."?
    – Mark
    Jun 6, 2023 at 18:21
  • 3
    @Mark Lol, yes ;-). Although we've seen and flagged that one, as it seems you have. Forgot about that, though - That might just be a 100% hit rate, but uncommon enough to not be terribly useful most of the time, of course ;-) Jun 6, 2023 at 18:27
  • 3
    I think our brains have evolved to detect AI, but I don't know how. Because we can sense the difference between ai generated ones and the real ones Jun 6, 2023 at 18:37
  • 19
    @Exampleperson Basic pattern recognition. The human brain, especially subconscious, is quite good at it. As you mention, it's harder, though, for most people to determine why they recognize the pattern. Jun 6, 2023 at 18:37
  • 1
    @NotTheDr01ds, People try hard to create AI and stuff. But our natural brain was developed without us doing anything. It is a remarkable feat of nature. And that too - consciousness. Crazy. Jun 6, 2023 at 18:39
  • 15
    We can't verify this, naturally, but SE staff can and, assuming it's true, it's pretty compelling. Please update us if SE staff does reach out to you.
    – Chris
    Jun 6, 2023 at 18:54
  • 1
    For what it's worth, I think that I have independently stumbled upon your phrase. So actually, we probably can independently verify this. No, I'm not going to give any hints or discuss this with anyone. Nor am I going to spend time gathering statistics. But consider this an indicator that probably someone else is going to stumble across this too, in due course. A clock is ticking.
    – JdeBP
    Jun 7, 2023 at 3:08
  • 3
    Currently two posts with "cutoff in September 2021" on Stack Overflow.
    – tripleee
    Jun 7, 2023 at 11:36
  • Perhaps the reason you find your trigger phrase in GPT answers is that GPT is preferentially copying/plagiarizing content and/or style from that high-rep user because of their high rep. It regards them as a 'trustworthy' source due to their high rep, so why not copy what they do ...
    – brhans
    Jun 7, 2023 at 20:54
  • 2
    @brhans It's true, and I considered that, but oddly the first time that high-rep user used the phrase was all three times the other day! I feel that scenario has definitely happened before, which is why we do see many of our high-rep users test with false positives (or they're time travelers who went back before GPT was invented to post) ;-). Jun 7, 2023 at 21:03
  • 1
    And to any future readers, no, this metric will rapidly degrade into uselessness and should not be considered useful whatsoever in the future.
    – Passer By
    Jun 9, 2023 at 11:06
  • @PasserBy No, the 10.81 number is correct. There is a ~ doubling of the use of that phrase from the period of time ..2022-11-30 and 2022-11-30..2023-05-31. This is because most, but not all, of the uses of it were caught (not just by me) and already removed. Since the new policy went into effect that says Mods aren't allowed to take action based on "writing style", however, the number has skyrocketed. Jun 9, 2023 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .