What is the network policy regarding AI Generated content? [closed]

Question

Earlier this week, Stack Exchange released guidance to moderators on how to moderate AI Generated content. What does this guidance include?

Answer 1

The policy given to moderators in private, which was later made public:

May 29, 2023

Please stop use of GPT detectors for content moderation on Stack Exchange

In light of significant research on the efficacy and flaws of GPT detectors, we’ve been studying the impacts of these detectors on Stack Exchange. And I’ve got some bad news to share.

We made two major discoveries, which I’ll discuss in more detail in this post. To summarize, though, we have strong evidence that:

1. none of the GPT detectors work as advertised on Stack Exchange data and have egregious false detection rates on our platform, and
2. they are creating an alarming native language bias in suspensions on the platform.

The tl;dr: We are now asking you to stop using GPT detectors to substantiate claims of GPT usage and to stop suspending due to AI-generated content in general. Based on what we now know, we can’t admit GPT detectors’ results as reasonable evidence of GPT usage. To be clear: as of today, Philippe has instructed our team that GPT detectors will not be acceptable to establish reason-to-block. We will revisit this decision if evidence emerges that the issues with these tools have been resolved.

GPT detectors do not work as advertised and produce unreliable results

Based on significant and credible research that implies GPT detectors return excessive false positives, we conducted our own survey of GPT detector efficacy on Stack Exchange posts authored before ChatGPT’s November release. This survey concluded that GPT detectors misclassify 32% (+/- 6%) of non-GPT posts on Stack Exchange sites as having been written by GPT. This error rate is too high to be tenable for day-to-day use, and is certainly too high to establish cause for messaging or suspension.

We also discovered that GPT detectors often disagree with each other as to whether a post is GPT created. This means that strategies relying on one of two detectors returning a positive result could lead to a baseline error rate of around 50% - and the incorrect detection rate gets worse the more detectors are added. In other words, 50% of posts detected as positive using an “either detector” strategy could be incorrectly classified as AI-written.

However, we did not find a way to reduce the error rate far enough by combining GPT detector results. Relying on multiple GPT detectors returning positives at the same time will return an error rate in the range of 12% (+/- 7%). We expect that combining the results of additional detectors will not return more meaningful results. This is still too high for operational usage, particularly at the volume GPT messages and suspensions are issued on the platform today.

If the situation changes and a GPT detector is released that meets our criteria for operational efficacy, we will revisit this policy change.

GPT detectors are creating bias against non-native English speakers

Even more alarmingly, recent research shows that GPT detectors exhibit overwhelming bias against non-native English speakers, and misclassify their writing as authored by GPT at a rate of ~45-80%. Available detectors are also unanimous in this misclassification around ~19% of the time. This risks unjustly targeting non-native English speakers on the platform, which comprise a significant majority of participants on some of our largest sites. (I would also hazard to say that non-native English speakers comprise the majority of participants on a majority of sites across the platform.)

We conducted a survey of the site using our best available information, on the assumption that biases against non-native speakers (a feature we cannot analyze) would show up as biases for and against specific countries (a feature we can analyze).

Our results strongly suggest that actual mod messages sent about GPT to users on the platform are significantly and inappropriately biased, likely driven by biases in the underlying tools used to flag GPT answers. To be clear, we do not believe that moderators are consciously acting with any sort of bias, but that the tools that they are using for this task have that bias built into them.

Regions of the world that appear to be unjustly targeted by GPT detectors in use on our platform include southeast Asia, the Middle East, and most of Africa. Regions of the world unjustly benefited by GPT detectors include North America, Europe, and Oceania.

The effect sizes are not small. For example, Pakistan receives 3.6x more suspensions for GPT usage than their baseline participation rates should imply. Bangladesh receives 2.7x more suspensions than the base participation rate justifies. India receives 1.75x more suspensions. On the other hand, the United States receives 0.6x the suspension rate, alongside Sweden, Great Britain, and Australia. While we unfortunately can’t share the raw data from our conclusions, we hope that these data points help to convince you that our alarm is justified.

Bear in mind that some of the reason we suspect GPT detectors create this bias is because they are detecting patterns in language that non-native speakers rely on. Therefore we also cannot accept similarities between the structure of a user’s answers alone as an indicator of GPT usage.

Only self-admission, freely given, can count as evidence of GPT usage on the platform

In light of the above, there are no tools currently available (at least so far as we are aware) that can identify GPT usage successfully. Therefore, the scope of admissible evidence for establishing GPT usage is, in the current state of things, very thin.

• GPT detectors cannot be counted as evidence for the reasons above.
• User behavior, such as answer timing, could be correlated with GPT usage, but a hunch alone does not a suspension make. Instead, this signal has to be taken as a prompt to review the objective quality of the user’s contributions.
• You may consider things that have obviously been copy pasted from ChatGPT such as literally mentioning the knowledge cutoff date from ChatGPT or phrases that include things like "If this didn't help, you can use a forum like Stack Overflow and ask for help there" to be self-admission and AI generated.

Therefore, the only admissible evidence we can currently permit for GPT usage is self-admission by the author of the posts, freely given. Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

As a final reminder, suspensions should only be issued for real behavior, actually known to be malfeasance. We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics.

---

We are aware that this leaves the state of GPT enforcement on the site in pretty rough shape. The silver lining is that these tools don’t really seem to have been working as designed anyway.

Here are the key points we’d like you to take away from this post:

• We are now requiring you to ignore and/or decline reports using online GPT detectors or intuition as their basis. Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.
• Do not direct users to GPT detectors or encourage them to act on their results in any way. Users, as well as moderators, should not be using these tools to determine post and user outcomes.
• The only admissible evidence for GPT usage is self-admission; however, you should not solicit self-admission from users.
• Remember that mod messages and suspensions are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unvetted heuristics.
• Effective immediately, Philippe has revoked the temporary policy change allowing moderators to jump to 30 day suspensions for GPT usage. In light of the current evidence, we simply do not trust that the tooling exists to support suspensions with the required degree of confidence. Although this policy was only formally enacted on Stack Overflow, any network sites that have informally been working under such a policy should also now consider that policy overturned, and should no longer be acting on it.

As far as the existing suspensions go, we will let active suspensions that were issued for a time period of less than 90 days expire organically. We’ll only review these suspensions if users contact us. For suspensions that were issued for a time period of 90 days or greater, we’ll manually review these to determine if they can be removed (they’re not just flooding bad content, for example).

Low quality content

You may still naturally deal with content abuse in ways we always have: if someone is posting a flood of poor quality content, suspend them for low quality content. Just don’t make it about GPT or AI generated content - make it about the quality being poor on a repeated basis. This includes things like repeated answers that don't address the question.

If someone is posting a flood of content to promote their tool/site/blog, use the astroturfing policy we have.

One final note: The amount of content deleted for GPT usage is truly astounding across the network. While we don’t doubt users have been using GPT and other language models to post answers, if you’ve been using GPT detectors to mark individual answers for deletion, we’d ask you to consider carefully whether any such content needs undeleting. Keep in mind, false positives are dramatically more likely for authentic writing by users who do not natively speak English, and may run as high as 80% for some detectors; in these cases, users may not deserve to have their content removed. Additionally, it remains possible that a false positive on one answer means false positives on other answers by that user are more likely to occur.

If you have questions, let us know in the answers below and Philippe is also willing to schedule a meeting in the TL if needed to discuss it further.

Answer 2

We recently performed a set of analyses on the current approach to AI-generated content moderation. The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high. Through no fault of moderators' own, we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts. Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written. This standard would exclude most suspensions issued to date.

We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.

We've reminded moderators that suspensions (and typically mod messages as well) are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics. Therefore, we are not confident that either GPT detectors or best-guess heuristics can be used to definitively identify suspicious content for the purposes of suspension.

As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.