Meta notes: This was originally posted as an answer in response to What is the network policy regarding AI Generated content?. Some of the requests for more information have been addressed in GPT on the platform: Data, actions, and outcomes, and I have written my response to that in an answer post there.
We recently performed a set of analyses on the current approach to AI-generated content moderation.
How exactly did you perform it? (partially status-completed)
The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high.
Can you please show us the data? (partially status-completed)
And what does hard data have to do with "potential for false-positives"? (I don't see what you could possibly have to extrapolate such a conclusion based on) Are you speculating whether deleted content was really AI-generated? Are you also speculating that the only reason deleted content with AI-related flag reasons was deleted was because it was AI-generated? There are other reasons, such as not following the rules for referencing content written by others.
we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts.
Can you please spell out what exactly is substantiating your suspicions? Is it Usernames? Avatar images? Profile location fields?
If a large percentage of the flags are on content written by people suspected to be from a specific set of countries, how exactly does that indicate antagonistic bias against those users? Have you ruled out the possibility that users from those countries are just proportionally violating the current AI-generated-content policies more than people from other countries? Or that a larger proportion of users from those countries are violating the policy compared to proportions of users in other individual countries? The reality is that there are countries where lying and fraud are just part of business / culture. Why would you rule out the possibility that certain cultures or subcultures care less about following the current AI-generated-content policy and won't mind violating our policies to do... whatever it is they're trying to?
For example, Wikipedia has a page on List of countries by intentional homicide rate. If you look at that page, you will find that the rate distribution by country is not- in fact- uniform (flat)... Does that make the people who sourced or compiled that that information... racist? Obviously not. So even if I observe that usernames or profile pictures of users who I have strong reason to suspect for violating per-site policies- such even as self-admittance/"confession"- are coming from a particular demographic (which I do), neither does that make me racist- as long as those usernames and profile pictures do not form part of my analysis (which they do not).
Adding to that, there is data that shows that developers in different countries have different sentiments about AI tools: Your 2023 SO dev survey.
- 83.6% of respondents from India use or plan on using AI tools, and from Brazil, 78.0%. (the top two countries in that response category)
- 82.2% of respondents from Brazil view AI tools favourably. (the top country in that response category)
- 15.4% of respondents from India think the most important benefit to AI tools is improved accuracy in coding, and from Brazil, 13.4%. (the top two countries by proportion of that response)
- 55.2% of respondents from India trust in the accuracy of AI, and from Brazil, 45.0%. (the top two countries in that response category)
Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.
The purpose of the suspensions and ban policies on Stack Overflow is rate-limiting bad content. If those people want to contribute "legitimately" (in accordance with site-policies), they can wait for the suspension to pass. I'm pretty sure the people who actually want to do the right thing will generally care enough and have enough grit to actually learn from their mistake and try again. The people who don't will give up. I see that as an absolute win. A one-week suspension is less than a slap on the wrist.
Would you please consider making that "internal evidence" less internal? (attempted status-completed. I find the attempt unsatisfactory / flawed / incomplete)
In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user.
This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written.
What exactly is this standard? You say what it excludes, but not what it includes.
Considering that the question post is titled "What is the network policy regarding AI Generated content?", and the body asks "Earlier this week, Stack Exchange released guidance to moderators on how to moderate AI Generated content. What does this guidance include?", the instant-self-answer doesn't really answer the question.
Using scanning tools may not be highly accurate, but it had at least one good property of being concrete.
There are good, non-biased ways to evaluate probability of using AI-generated-content that are not based on hunches: look at the writing style, phraseology, and compare it with the post owner's past writing. If there's a huge difference, that's a big red flag.
Also,... why do you point at problems with "bias", and then ban an approach that you haven't made any argument about being biased? What's the connection? (you later say scanners have rates of false-positives, but I don't see any clear connection between false-positives and bias).
... If anything, a scanner would prevent bias against users based on things like their username, avatar image, and profile location field, (assuming you don't plug that info into the scanner, which I'm almost certain nobody does).
And if people don't plug that info into the scanner, what else is there except the content written in near-perfect English by the one-and-the-same ChatGPT? Unless someone prompted along the lines of: "Please write me and answer to the following question in the style of a <ethnic/racial/geographical group> person". And let me tell you as matter-of-factly as I can that from what I've seen in usernames, I don't think that these policy violators would feel very inclined to do that.
This standard would exclude most suspensions issued to date.
Again, can we please see the numbers and facts?
We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.
Can you please show us the numbers and your methodology? (some attempt made at status-completed)
As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.
A friendly reminder to readers that we have a list of ChatGPT or other AI-related discussions and policies for our sites, and that at the time of this writing, ChatGPT is banned on SO, and this decision is (was?) supported by SE. Quoting from the SO banner announcement:
We’ve just published a new Help Center article outlining our expectations and rationale for GPT-generated content on Stack Overflow and decided, together with moderators, to add a banner for all users pointing to it. We've also explicitly allowed more leeway for moderators in how they handle suspensions for this matter.
It seems you've changed your tune now.
I suspect there's a deeper reason behind what you've given in your new policy post and I'm curious about what it is... traffic dropping?
Why not make us part of the conversation? Maybe we could help. A post here on MSE like- "We're concerned about traffic dropping ever since the release of ChatGPT. How can Stack Exchange and Stack Overflow for Teams continue to stay relevant and competitive in the knowledge-sharing space?" is one way to do that.
