With AI answers now being essentially unmoderated, what data might be surfacing that demonstrates the old policy was effective?

Question

It's been about a week since the community and moderators lost the ability to effectively moderate AI content. Given that time, we should start to see some data/patterns emerge that could tell us (and the SE Staff) how effective (or not) the previous policy was. What objective trends have you seen that could demonstrate that the previous heuristics that were being used were effective?

We're all aware that there is potential for false positives in any detection, but can we perhaps come up with some data that helps demonstrate the rate at which we were successful?

Answer 1

I'll hopefully have more data as time goes by, but here's one that stands out as obvious. For those that haven't seen my other posts on this topic, note that I've flagged over 1,200 answers that I've suspected as being ChatGPT-generated in whole or part. Most of these have been on Stack Overflow, with some other sites as well.

In reading literally thousands of Stack Overflow answers, I've noticed that there's a phrase that GPT sometimes uses in them that indicates at a very high signal-to-noise (signal being GPT; noise being false positives) ratio.

No, I'm obviously not going to tell anyone outside of a very select, private group what that phrase is, since people will actively avoid it in the future. I honestly haven't even told any moderators about it up to this point. They know it's one of the phrases that I believe points to a possible GPT-answer (but it would never be the sole indicator that I rely on, either), but they don't know how much I believe it is such a great GPT-indicator over most all others.

Edit: And yes, as @MissSkooter points out in the comments, this makes the following data unverifiable to most readers here. You simply have to trust me (or choose not to) based on the reputation I've developed flagging and posting here. I'm happy to share the data with SE Staff, if requested, though. Or they can just look at my "Saves" list for "Suspected ChatGPT" on SO. I've made notes regarding the phrase in some recent (at the time of this answer) private notes. I have to assume these aren't that private ;-), and I'm okay with the SE staff reviewing them.

This particular phrase occurs in only 251 Stack Overflow (undeleted) answers ever. I suspect it's even often left in answers where a user has attempted to edit the content to hide ChatGPT use.

Eight of the 251 answers that use this phrase were written yesterday.

Now three of those answers (yes, three yesterday) were written by a single very high-rep user that I have trouble believing was using GPT (or would even need to). There are also things that I see in that user's answers that indicate to me that it was either human written, or at least edited. So yes, we have false positives on it already, but they are easy to spot for multiple reasons. I typically can eliminate those that we believe are false positives before even flagging.

If this phrase was never used by ChatGPT, then we should theoretically see a fairly consistent use of the phrase over time. This assumes, of course, that it isn't a new idiom that has become popular recently, which it isn't, other than ChatGPT seemingly making it popular.

But the rate of usage yesterday, even throwing out the known false-positives, was roughly 133 times higher than the average SO day.

Some more analysis:

Stack Overflow has been receiving answers for 5,424 days. With ChatGPT’s introduction on November 30th, 2022, we had 5,235 “pre-ChatGPT” days and 189 days since ChatGPT’s introduction.

• Prior to November 30th, this phrase had been used 218 times. That’s an average rate of 4.16 occurrences of the phrase every 100 days.

• Between November 30th and May 31st, the phrase was used 20 times at the rate of 10.81 uses per 100 days. Keep in mind that there have been many more uses in this timeframe, but the (perhaps vast) majority have been deleted under the previous “detection” policies. This increase just shows that there are likely some ChatGPT uses in the timeframe that we have missed, and by eyeballing a few of them, I can confirm that this is the case.

• In the last week (essentially since we stopped moderating the use of AI), the phrase has been used 13 times, which is a rate of 185.71 uses per 100 days. There has actually been at least one more, but the user deleted the answer when I pointed them to the AI Policy.

Reputation analysis¹:

• For the 10 most recent answers that used this phrase before ChatGPT was available:

  • The median reputation of the user who posted was 15,435.
  • Trimming the lowest rep and highest rep ², the mean was 25,266.

• For the 10 most recent answers that used this phrase as of today:

  • The median rep of the user who posted was 44.5.
  • The mean, again trimming the outliers, was 98.

  _{¹ Of course, reputation grows over time, but not this drastically over a six month span.
  ² The outliers were so extreme (on the high end) that the results would have been meaningless if they weren't trimmed.}

  So is this heuristic biased against newer users? Or is it that new users are more likely to use ChatGPT?

Let's look at the five suspected ChatGPT uses today. In reviewing these answers, I do believe that they are GPT, and I probably would have flagged most, if not all of them:

Of the five users who posted these answers:

• User #1: Had only the one answer, but it does appear to be GPT and I typically would have flagged it. It just has too much in it that indicates it was likely written by ChatGPT.

• User #2: Had 5 answers on May 16th, 4 answers today, and all appear GPT. One of them has a comment that the answer didn't help, but another of the (likely GPT) answers was accepted and upvoted. With four total downvotes, one upvote, and one accept, the net rep is still an 18 point gain.

  Another user noticed that this user was using AI and commented on several answers. I believe they attempted to downvote them as well, but they ran into the same problem I've seen -- If a community member suspects someone of using AI, they can only downvote 3 answers before the "targeted voting" algorithm rolls back votes.

• User #3: Has 4 answers today. Two appear to be GPT. Another is code only, and the other appears to be user-written, but could simply be edited GPT output (closer analysis of the code not completed). I personally tend to suspect that if a user used ChatGPT on two answers in one day, they likely used it on all four.

• User #4: Has 3 likely-GPT answers dating back to April 7th. Two of them have comments indicating they have issues.

• User #5: Had 3 likely-GPT answers yesterday; perhaps another on May 12th. All four answers have comments indicating potential issues with the quality.

So following that one phrase could have unearthed a likely 17 (at least) AI-generated answers that now remain. Seven of those (so far) have been identified as having potential problems. While the new policy seems to say that users who repeatedly post "very low quality" content can be reprimanded, the moderators that I've talked to seem to lack the confidence that these meet the bar of "repeatedly" or VLQ.

And under the new moderation policies, "writing style" cannot be used to show that something might be AI-generated, so these answers will all remain.

Side-note: While this particular phrase is the most drastic example, there are certainly other "GPT-tells" that have increased in the last week.