(Historical) Policy on the use of GPT detectors

Question

The policy below was posted on the Mod Team on May 29, 2023 to update moderators as to changes in how the GPT policy should be enforced. A day later, a shortened version of this policy was published on MSE. We normally do not release enforcement guidelines publicly. In recognition of the confusion caused by the differences between the public and private guidance in this case, we are now publishing the private text of this policy in full. However, we reserve the right to continue in the future to withhold specific enforcement guidelines from being published publicly in cases where doing so would make the jobs of moderators more challenging, and open up new avenues for abuse.

This is being released now so that it can be included in the public record. Since being published, this policy has been superseded, and the interim policy published. The historical policy begins with the line below this one.

---

May 29, 2023

In light of significant research on the efficacy and flaws of GPT detectors, we’ve been studying the impacts of these detectors on Stack Exchange. And I’ve got some bad news to share.

We made two major discoveries, which I’ll discuss in more detail in this post. To summarize, though, we have strong evidence that:

1. none of the GPT detectors work as advertised on Stack Exchange data and have egregious false detection rates on our platform, and
2. they are creating an alarming native language bias in suspensions on the platform.

The tl;dr: We are now asking you to stop using GPT detectors to substantiate claims of GPT usage and to stop suspending due to AI-generated content in general. Based on what we now know, we can’t admit GPT detectors’ results as reasonable evidence of GPT usage. To be clear: as of today, Philippe has instructed our team that GPT detectors will not be acceptable to establish reason-to-block. We will revisit this decision if evidence emerges that the issues with these tools have been resolved.

GPT detectors do not work as advertised and produce unreliable results

Based on significant and credible research that implies GPT detectors return excessive false positives, we conducted our own survey of GPT detector efficacy on Stack Exchange posts authored before ChatGPT’s November release. This survey concluded that GPT detectors misclassify 32% (+/- 6%) of non-GPT posts on Stack Exchange sites as having been written by GPT. This error rate is too high to be tenable for day-to-day use, and is certainly too high to establish cause for messaging or suspension.

We also discovered that GPT detectors often disagree with each other as to whether a post is GPT created. This means that strategies relying on one of two detectors returning a positive result could lead to a baseline error rate of around 50% - and the incorrect detection rate gets worse the more detectors are added. In other words, 50% of posts detected as positive using an “either detector” strategy could be incorrectly classified as AI-written.

However, we did not find a way to reduce the error rate far enough by combining GPT detector results. Relying on multiple GPT detectors returning positives at the same time will return an error rate in the range of 12% (+/- 7%). We expect that combining the results of additional detectors will not return more meaningful results. This is still too high for operational usage, particularly at the volume GPT messages and suspensions are issued on the platform today.

If the situation changes and a GPT detector is released that meets our criteria for operational efficacy, we will revisit this policy change.

GPT detectors are creating bias against non-native English speakers

Even more alarmingly, recent research shows that GPT detectors exhibit overwhelming bias against non-native English speakers, and misclassify their writing as authored by GPT at a rate of ~45-80%. Available detectors are also unanimous in this misclassification around ~19% of the time. This risks unjustly targeting non-native English speakers on the platform, which comprise a significant majority of participants on some of our largest sites. (I would also hazard to say that non-native English speakers comprise the majority of participants on a majority of sites across the platform.)

We conducted a survey of the site using our best available information, on the assumption that biases against non-native speakers (a feature we cannot analyze) would show up as biases for and against specific countries (a feature we can analyze).

Our results strongly suggest that actual mod messages sent about GPT to users on the platform are significantly and inappropriately biased, likely driven by biases in the underlying tools used to flag GPT answers. To be clear, we do not believe that moderators are consciously acting with any sort of bias, but that the tools that they are using for this task have that bias built into them.

Regions of the world that appear to be unjustly targeted by GPT detectors in use on our platform include southeast Asia, the Middle East, and most of Africa. Regions of the world unjustly benefited by GPT detectors include North America, Europe, and Oceania.

The effect sizes are not small. For example, Pakistan receives 3.6x more suspensions for GPT usage than their baseline participation rates should imply. Bangladesh receives 2.7x more suspensions than the base participation rate justifies. India receives 1.75x more suspensions. On the other hand, the United States receives 0.6x the suspension rate, alongside Sweden, Great Britain, and Australia. While we unfortunately can’t share the raw data from our conclusions, we hope that these data points help to convince you that our alarm is justified.

Bear in mind that some of the reason we suspect GPT detectors create this bias is because they are detecting patterns in language that non-native speakers rely on. Therefore we also cannot accept similarities between the structure of a user’s answers alone as an indicator of GPT usage.

Only self-admission, freely given, can count as evidence of GPT usage on the platform

In light of the above, there are no tools currently available (at least so far as we are aware) that can identify GPT usage successfully. Therefore, the scope of admissible evidence for establishing GPT usage is, in the current state of things, very thin.

• GPT detectors cannot be counted as evidence for the reasons above.
• User behavior, such as answer timing, could be correlated with GPT usage, but a hunch alone does not a suspension make. Instead, this signal has to be taken as a prompt to review the objective quality of the user’s contributions.
• You may consider things that have obviously been copy pasted from ChatGPT such as literally mentioning the knowledge cutoff date from ChatGPT or phrases that include things like "If this didn't help, you can use a forum like Stack Overflow and ask for help there" to be self-admission and AI generated.

Therefore, the only admissible evidence we can currently permit for GPT usage is self-admission by the author of the posts, freely given. Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

As a final reminder, suspensions should only be issued for real behavior, actually known to be malfeasance. We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics.

---

We are aware that this leaves the state of GPT enforcement on the site in pretty rough shape. The silver lining is that these tools don’t really seem to have been working as designed anyway.

Here are the key points we’d like you to take away from this post:

• We are now requiring you to ignore and/or decline reports using online GPT detectors or intuition as their basis. Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.
• Do not direct users to GPT detectors or encourage them to act on their results in any way. Users, as well as moderators, should not be using these tools to determine post and user outcomes.
• The only admissible evidence for GPT usage is self-admission; however, you should not solicit self-admission from users.
• Remember that mod messages and suspensions are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unvetted heuristics.
• Effective immediately, Philippe has revoked the temporary policy change allowing moderators to jump to 30 day suspensions for GPT usage. In light of the current evidence, we simply do not trust that the tooling exists to support suspensions with the required degree of confidence. Although this policy was only formally enacted on Stack Overflow, any network sites that have informally been working under such a policy should also now consider that policy overturned, and should no longer be acting on it.

As far as the existing suspensions go, we will let active suspensions that were issued for a time period of less than 90 days expire organically. We’ll only review these suspensions if users contact us. For suspensions that were issued for a time period of 90 days or greater, we’ll manually review these to determine if they can be removed (they’re not just flooding bad content, for example).

Low quality content

You may still naturally deal with content abuse in ways we always have: if someone is posting a flood of poor quality content, suspend them for low quality content. Just don’t make it about GPT or AI generated content - make it about the quality being poor on a repeated basis. This includes things like repeated answers that don't address the question.

If someone is posting a flood of content to promote their tool/site/blog, use the astroturfing policy we have.

One final note: The amount of content deleted for GPT usage is truly astounding across the network. While we don’t doubt users have been using GPT and other language models to post answers, if you’ve been using GPT detectors to mark individual answers for deletion, we’d ask you to consider carefully whether any such content needs undeleting. Keep in mind, false positives are dramatically more likely for authentic writing by users who do not natively speak English, and may run as high as 80% for some detectors; in these cases, users may not deserve to have their content removed. Additionally, it remains possible that a false positive on one answer means false positives on other answers by that user are more likely to occur.

If you have questions, let us know in the answers below and Philippe is also willing to schedule a meeting in the TL if needed to discuss it further.

Answer 1

Prologue: meta

Much as I've wanted to, I've refrained from saying much about this situation thus far, primarily because this policy wasn't public.

Even though I knew the gist of it—for folks had reached out to me privately about it, there'd been discussion of the issues with it (though not its contents) on the meta Discord, and the comments on and around your public update were damning enough... I feel that it is pointless to try to have a discussion with someone about a topic they are determined not to discuss, and... Keeping this policy private has been a deafening shout to that effect every day that this ordeal has dragged on.

Today, it's public, and today we can talk. Plenty of other folks have (re-)stated the practical issues with this policy, so I'll mostly limit my points here to meta-observations: how communication has progressed (or failed to do so) more so than what has been communicated.

And, that's sorta my thing.

Introduction: a spiderweb of concerns

Ever throw a handful of tennis balls in front of a puppy? They dodge and leap and don't catch any of them. Most of us are the same way: the more issues we see in front of us, the harder a time we have latching onto any of them; as problems multiply, less gets done, and even less gets done well. Pretty soon, we find there are more issues than we started with, even as we run around breathlessly trying to stay on top of everything...

I wouldn't blame anyone at SO for feeling that way right now, especially reading through one of these meta threads. Y'all started with A Big Problem, and tried to put together a policy change to address it... Now, just a couple of months later, you have a pile of new problems, cracks spreading out from your attempted patch.

...And of course, the original Big Problem still exists too. I don't know what to do about that—if it could be solved by someone sitting and drinking whiskey while waiting for git to finish something, it'd have been solved years ago. But the spiderweb-cracks, that I know something about.

Root cause: failure to communicate

If I was in your place right now, I'd have a big spreadsheet open, filled with links and summaries of the feedback on the other ends of those links. I'd be staring at those summaries, looking for patterns: what shows up repeatedly, what commonalities do disparate complaints share? YMMV, but personally I've found this to be a great exercise when faced with a seemingly-overwhelming flood of concern.

Here's what I think you'd observe, were you to try this exercise:

1. Folks do not feel like you're being candid with them. About your concerns, your plans to address them, your rationale for why those plans might work... About what you're seeing here, what you're hearing from folks on Stack Overflow, or Mathematics, or Super User... About what you're hearing from folks within the company, its parent company... About how you're feeling—about all of this, about the even greater tumult at your alien-themed former employer, about the seemingly impending sea change in this industry as a whole... And especially, about how all of that is affecting the decisions being made here.

2. Folks do not feel like they're being listened to. They don't hear their concerns being echoed back; instead, they hear reductive summaries of their concerns. Summaries like, "confusion". Even when they go well out of their way to produce effective summaries for you.

Effective communication requires understanding; these black-and-white symbols don't carry enough information on their own, and we need context to interpret them—what sort of a person has typed them, what are their goals, their hopes, their fears?

Effective communication is based on shared context... And some amount of vulnerability.

Folks here are pretty vulnerable right now. They're trusting you to not abuse that, but rather to use it to understand them a little better.

Public by default: an exercise for self-improvement

Years and years ago, there was a regular company-wide call at SO, where various teams would update everyone on what was happening with them. These calls used to start with Joel running down a list of precepts...

One of those precepts was "public by default". He'd cover the topic by listing the things we couldn't talk about publicly—the implication being, most anything else was fair game. It was a short list—it had to be, lest it cut into the time needed to remind folks to clean the steam wand on the office espresso maker.

...It also had to be a short list because once you have to stop and consult a list, folks tend to default to not communicating. Either you know, know, at any given point in time that you're free to talk about your work... Or you don't do it.

Now, I know this isn't really a thing at SO anymore. It had already fallen out of fashion long before my departure, the former deluge of writing from SO employees reduced to a trickle.

...But here's why you should consider reviving it for yourself, and ideally your team:

... Firstly, on a technical level, there’s an immediate feedback loop. If you’re answering questions on forums or online communities, considering strategy or contributing to open source, you have such a swift feedback loop that it’s impossible not to improve.

What’s more, the hive-mind of the internet has a habit of taking ideas you might have which are ok and turning them into something worth pursuing, adding value through different angles or directions. ...

To truly embrace public-by-default, it’s not enough to share your successful projects and knowledge, but additionally to bring the humility to share your learning and failures. In general, it’s hard to argue with the sentiment that you think harder about everything you do if you’re going to share instead of just do it for yourself, no matter how trivial it might be.

Those are quotes from Ben James in a blog post he wrote for the Stack Overflow blog. You'll note that he adopts the "enlightened self-interest" approach to selling this technique: don't communicate as a charity for others; do it because it makes you better at what you do!

And he's right: it does. Just like any form of regular practice, defaulting to talking about what's going on publicly allows you to become comfortable with your tools and strengths, adept at compensating for your own weaknesses, quick to see opportunities for improvisation and nimble at avoiding pitfalls.

Sharing your failures also allows you to be vulnerable at times when your audience is... Maybe a bit more predisposed to cutting you some slack.

But most importantly, it allows you to build that shared context I mentioned earlier, so that when you really, really need to communicate effectively... You can.

Fear and loathing in the age of large language models

Lacking so much shared context, there's an awful lot of speculation happening right now; when folks don't have the information they need, we tend to ... make up something plausible. Or failing that, something entertaining.

...so for the record, I don't actually believe SO, Inc. is operating under the control of alien templar yeti. Yet.

After reading this policy, it's clear that there's a lot of speculation going on WRT unknowns internally as well. And small wonder: if these trends continue to accelerate, your bread & butter will be gone before there's much time to react!

But... There's always some boogieman like that in the shadows ahead. Some new threat that needs a new strategy to thwart. That's not a reason to clam up, retreat into the shadows yourself... Rather, it's the most important time to step up, pull together and raise our torches high. To take that speculation and turn it into testable theses... and then test them, share the results, and by working together and sharing that knowledge gained, improve things for all of us.

If LLMs are to bring an end to all this, don't let it happen through fear and intimidation. Face the threat with open communication, sharing, and collaboration...

Because that... Has always been The Stack Overflow way.

Answer 2

Honestly, this just shows the lack of trust/communication between the company and moderators and the community (but again thanks for publishing this). The policy focuses on the reliance on GPT detectors, even when mods have said they did not use them. You didn't even talk to moderators before forming your ill-informed opinion.

Also,

Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.

This is just patently false. What gave you this impression? Certainly not talking to the community. We have been trying to reach out to you guys for a long time, and we were repeatedly ignored. It took a moderation strike for you to finally listen to us, which says something about how much the company values its volunteers.

It's pretty ironic; an announcement about strict requirements of evidence (analogous to proving Treason in the U.S.) that wasn't made with the same rigor. You need to hold yourself to the same standards that you impose on others.

Answer 3

First, I appreciate this being posted. It feels like a move towards transparency, and we need more of that.

And since this has been superseded, disputing the contents of it might be of questionable worth, but I'll note these two points in hopes that they are "fixed" in the new policy.

1. (I'm not a mod, so this is drawing on reports from mods that I've read.) This policy is written as if mods were predominantly, if not exclusively, using GPT detectors to decide if something was GPT-produced, and from all reports this is incredibly and widely off the mark. Were mods even consulted before basing the policy on this assumption?

2. "Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad." I'm trying to stay dispassionate here, so I'll just say that, speaking for myself, this is very, very incorrect. I don't even know the name of a single detector, and have at least a mild sense of when something might be GPT-generated, which is usually confirmed when closer readings disclose the fantasms, errors, and missing-the-point-nesses that LLMs produce but people rarely do.

I'll avoid discussing why this policy was produced, because it'd be based almost entirely in speculation on motives and not evidence. But hopefully the two points above, dealing with concrete facts, are helpful.

Answer 4

The original publicly available description of the policy stated that:

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written

Therefore, we are not confident that either GPT detectors or best-guess heuristics can be used to definitively identify suspicious content for the purposes of suspension.

The actual content of the policy states that

We are now asking you to stop using GPT detectors to substantiate claims of GPT usage and to stop suspending due to AI-generated content in general.

We are now requiring you to ignore and/or decline reports using online GPT detectors or intuition as their basis. Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.

Almost immediately you received replies like:

This post does not match the guidance given to moderators. – Bryan Krause is on strike May 30 at 19:52

"In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user." The guidance I have received is: "We are now asking you to stop [...] suspending due to AI-generated content in general". How do I reconcile these two statements? – Andy May 30 at 19:57

I think it was clear that the moderators didn't see the two version as equivalent.

My question is simple: why did this happen and why it wasn't immediately cleared up to the general public too?

The public version sounds far less aggressive/assertive, almost like you wanted help on finding a better way. This sentiment is in my opinion completely absent in the private version.
I think you may very well realize that from the outside this looks like a deliberate set-up: showing a "reasonable" version to the general public while the moderators were given Order 66 instead would be a convenient strategy to make them look unreasonable and unwilling to cooperate since all the general public would see would be a far less restrictive suggestion.

Basically, it looks like the company purposely withhold some information when talking to the userbase in order to try to undermine the credibility of the moderators actions by painting them as a group of people overreacting to requests that were never made. Since we can assume that the company had foreseen the consequences of the new policy, and considering that apparently the misinformation campaign wasn't limited to the general public (see the notes on Jon Ericson blog ) this interpretation sadly sounds quite believable.

Answer 5

Well, in the interest of the historical record and hopefully leaving something educational for future readers - to me there's a few specifically problematic things.

Firstly - while there were clear attempts by the moderator communities to engage the company with respect to the problem of folks posting unverified GenAI answers at scale. We posted on meta, and based on the feedback provided to us at the time. That we ended up with a patchwork of per site policies, including up to a 1 year suspension, and we didn't get any communication until this. No consultation. No warning. This also kind of undermined us when dealing with what we deemed to be problem users.

In addition

1. Stack Exchange made an assumption we were using ChatGPT detectors. Many of the flags were from folks using it to confirm their hunches. However we made the determination based on other factors like velocity of posting and writing style.

2. Regions of the world that appear to be unjustly targeted by GPT detectors in use on our platform include southeast Asia, the Middle East, and most of Africa. Regions of the world unjustly benefited by GPT detectors include North America, Europe, and Oceania.

   This makes a bit of a awkward situation. If folks there are using GPT more - they'd tend to be suspended more. Correlation isn't causation. I'd also note we also get a good chunk of our spam from literally 2-3 Indian and Pakistani metropoles. Would that mean they're unjustly targeted, or there's just more of it? There's various other tells that I believe the (informal?) folks looking at heuristics can shed light on, like a sudden velocity in decent quality posts and some unusual conjectures. That said, this could be read as a potential accusation of racism.

3. Therefore, the only admissible evidence we can currently permit for GPT usage is self-admission by the author of the posts, freely given. Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

   That's an impossibly high barrier. And I feel like this was known.

   And somehow we're protecting random folks who could be causing the site harm, over trusting the moderation teams to protect their sites.

   The attitude here is one of the reasons I'm striking.

4. Next April I'll have been moderating for a decade. Most moderators have a deep knowledge of their communities, and to an extent the subject material.

   As a final reminder, suspensions should only be issued for real behavior, actually known to be malfeasance. We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics.

   Quite a lot of moderation starts off on a hunch, and intuition built up over years. This literally smacks of a lack of trust. I try very hard not to tell community managers how to do their job - and I'd expect the trust to do what I do. And I'm open to being told when I'm wrong. Just... when, not if I might.

   I'm not angry at this. I just don't think its a great way to work if we can't trust moderators - folks who're practically in positions of community leadership for their respective communities. It also shows (well one of the things that do) that there's a lack of understanding of what we do.

Answer 6

First: Thank you for finally rescinding this insane policy. Second: Thank you for finally allowing me to see why the moderators had to go on strike

There is an idiom, "The devil is in the details" which comes to mind when comparing the secret and public versions of this policy. The difference between the two, when comparing paragraph for example, might be minor. The details of implementation, however, are a bit farther apart.

I trust the moderators and respect their ethics. Yes, I know they are human, each with their own faults, unique to each as an individual. Somehow they, collectively, always seem to stay on the high road anyway. Because I trust them I "knew" there was some kind of disconnect between the secret version (which they were required to enforce) and the public version (which I could expect to be followed). Since they were willing to go on strike to effect a change, I figured it was certain to be a non-trivial difference. I had no way of knowing just how vast the gap was.

Simple interpretation of the public version:

Be very, very careful about what you consider to be AI-generated content, and be "gentle" with the consequences.

Simple interpretation of the secret version:

Unless someone admits to posting AI-generated content, without you asking, it is not AI-generated. If they do admit it openly then you can remove it. Never suspend for posting AI-generated content.

I fail to envision any future egalitarian utopia community wherein the only possible evidence of an offense against the community is the spontaneous, unsolicited self-admission of said offense.

What was anyone, someone, everyone, possibly thinking, if they were thinking, which prompted the text, and qualifiers, of this absurd version of a "policy"?

Answer 7

For example, Pakistan receives 3.6x more suspensions for GPT usage than their baseline participation rates should imply. Bangladesh receives 2.7x more suspensions than the base participation rate justifies. India receives 1.75x more suspensions. On the other hand, the United States receives 0.6x the suspension rate, alongside Sweden, Great Britain, and Australia. While we unfortunately can’t share the raw data from our conclusions, we hope that these data points help to convince you that our alarm is justified.

I've said it before and I'll say it again:

[...] how exactly does that indicate antagonistic bias against those users? Have you ruled out the possibility that users from those countries are just proportionally violating the current AI-generated-content policies more than people from other countries? Or that a larger proportion of users from those countries are violating the policy compared to proportions of users in other individual countries? The reality is that there are countries where lying and fraud are just part of business / culture. Why would you rule out the possibility that certain cultures or subcultures care less about following the current AI-generated-content policy and won't mind violating our policies to do... whatever it is they're trying to?

For example, Wikipedia has a page on List of countries by intentional homicide rate. If you look at that page, you will find that the rate distribution by country is not- in fact- uniform (flat)... Does that make the people who sourced or compiled that that information... racist? Obviously not. So even if I observe that usernames or profile pictures of users who I have strong reason to suspect for violating per-site policies- such even as self-admittance/"confession"- are coming from a particular demographic (which I do), neither does that make me racist- as long as those usernames and profile pictures do not form part of my analysis (which they do not).

Adding to that, there is data that shows that developers in different countries have different sentiments about AI tools: Your 2023 SO dev survey.

• 83.6% of respondents from India use or plan on using AI tools, and from Brazil, 78.0%. (the top two countries in that response category)
• 82.2% of respondents from Brazil view AI tools favourably. (the top country in that response category)
• 15.4% of respondents from India think the most important benefit to AI tools is improved accuracy in coding, and from Brazil, 13.4%. (the top two countries by proportion of that response)
• 55.2% of respondents from India trust in the accuracy of AI, and from Brazil, 45.0%. (the top two countries in that response category)

And then there's the fact that sideshowbarker, who (from my understanding) was carrying out a lot of the moderation of ChatGPT flags on SO, didn't even use these detectors.

---

Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

But why? How is a solicited admission any less legitimate than a freely given one?

It crossed my mind that there could be concerns about insulting people upon "false-accusations"/false-positives, but if one has enough experience recognizing ChatGPT content, the risk of accidentally causing insult to someone who didn't actually use ChatGPT is lower.

And really, I wouldn't find "hey, out of curiosity, did you get this from ChatGPT" that insulting. It'd ruffle my feathers a bit, but I'd just say "no, I wrote it myself", custom flag as "No longer needed. And remove my reply comment too, please.", and move on. Granted, there might be some survivorship bias here, since I've had enough grit to stick around past other minor things that the average person might feel indignant or hurt enough to leave by.

---

The amount of content deleted for GPT usage is truly astounding across the network.

The amount of new and old AI-generated content that I continue to find on a daily basis on SO is also astounding.

Answer 8

This just demonstrates that SE/SO likes to be secretive and duplicitous.

Notably absent: A straight apology for putting moderators into this position in the first place by forcing them to be silent about a lie SE/SO management had no trouble telling users.

It just underlines how untrustworthy SE/SO management is.

Taking two months to tell the truth doesn't make you look better, not least because we've all known this lie existed for months.

It also makes this statement very hard to take at face value:

Since being published, this policy has been superseded, with the new policy available shortly

...but...

We normally do not release enforcement guidelines publicly. In recognition of the confusion caused by the differences between the public and private guidance in this case, we are now publishing the private text of this policy in full. However, we reserve the right to continue in the future to withhold specific enforcement guidelines from being published publicly in cases where doing so would make the jobs of moderators more challenging, and open up new avenues for abuse.

You've already said you reserve the right not to publish policies, so how do we users know your published policy will represent the actual policy your force on moderators (and us)?

You haven't grasped the basic issues:

1. All policies need to be public.

2. Different public and private policies is unworkable insanity. The public policy must at least reflect the overall logic of the private one.

3. Private implementation details is not the same as having a completely different public policy from the private one.

By no stretch of the imagination could your public and private policies be said to follow the same logic. One is basically "never do this" and the other is "use good judgement when doing this".

SE/SO management put moderators into an impossible ethical position and - here's some news for you in SE/SO management - we users vote for moderators with a proven record of good ethical judgement.

We'd like SE/SO to show some of the same ethical good judgement. I don't see any signs they do here.

Answer 9

It's inaccurate to call this a "policy", because by the terms of the Moderator Agreement, the company will

Post previews for review of all new official policies in the Moderators Teams instance with the policy tag, marked with links to their public version once published, and maintain a listing of all official network-wide policies with links to them in the Help Center.

So private documents like this really must only be considered guidance, not actual policies. (Note that it was not posted for preview, nor was it posted publicly until now, after it has already been rescinded.)

However, since the company was apparently blatantly disregarding its part of the Moderator Agreement, most mods would not have felt like they could disregard the guidance. We were left in a pickle: a disastrous quasi-official private policy which could not be ignored but neither could we continue to moderate while following it. And so, the strike.

Answer 10

Thanks a lot for publishing this. It's much appreciated. When reading I wondered why you didn't take into account different possible thresholds. GPT detectors surely are tunable to some extent and this

This survey concluded that GPT detectors misclassify 32% (+/- 6%) of non-GPT posts on Stack Exchange sites as having been written by GPT.

only makes sense for a specific setting. Or was this included in the +/- 6%?

The country dependent suspension rate might not be a bias. Is the underlying assumption that otherwise all countries behave the same with regard to GPT? It may be good to explicitly state all assumptions.

The biggest mistake however seems to be to not ask for feedback before making the decision. Just imagine if you had presented the results and discussed them with the mods before making a decision. There might simply be a discussion about false positive rates now and the strike may never have happened. If anything I think this should be the take-home message: getting feedback is often enormously helpful.

Answer 11

I would like to preface this response by saying that more communication and transparency is almost invariably positive and that I appreciate the SE corporate team having decided to share some of the basis of their decision publicly. To the extent that the company is considering the impact of its actions, that is also a positive thing: sometimes, as with explicit policies against racism, transphobia, and so forth, bowing to the negative feedback of the community is not always a positive thing.

That said, I still have some serious concerns, ones that I imagine many people here share. For instance, what does the research actually mean?

• The post claims that "Even more alarmingly, recent research shows that GPT detectors exhibit overwhelming bias against non-native English speakers, and misclassify their writing as authored by GPT at a rate of ~45-80%."

  If this is true, it is indeed alarming and would amply justify the policy against these tools. One can even imagine a reason that this might be the case: the underlying classifier might be trained with only two categories, "LLM" and "human," and non-American, non-British dialects might be very underrepresented in the "human" category, causing the model to be biased toward classifying them in the most populous class ("LLM").

  However, I can also imagine a reason that it would not be: by default, content generated by ChatGPT and the other most popular models in this space adheres to a strict notion of correct English as assumed from content primarily adhering to American and British normative dialects.

  This uncertainty naturally leads to the question: Is this based on external research or on analysis of SE posts? If the latter, does it use writing whose provenance is clear, such as that predating the use of these models? That information needs to be shared.

• Building upon the previous point, could some of the bias be, quite simply, because people for whom English may not be their first language, or who use a dialect that may be socially disfavored in contexts like SE, are more likely to use tools like ChatGPT to edit their answers? In other words, could some or all of the disparities be because of true positives? If so, it might be understandable that one might not view this as a major problem, but framing it as an issue with the classifiers would be erroneous.

• In such a situation, surely a more nuanced approach than simply prohibiting the use of such detectors would be in order. Why not allow the use of such tools in conjunction with some common sense to figure out whether a user is generating answers that (at this point) may well contain major errors disguised by the apparent fluency of the text, or simply recasting their own analysis?

• Particularly if the research is from SE, we also should know the absolute numbers. With just relative fractions, we do not even know whether the differences are statistically significant, let alone the question of impact: how many legitimate users would actually be affected at all?

Also, why has the site not applied this same level of scrutiny to other issues of disparity?

• Do the actions of human moderators have a disproportionate impact on users from certain regions? Anecdotally, I have seen a definite tendency for many moderators to ban disruptive users with non-standard English, while letting equally disruptive users with "good" English go unchecked. Why do LLMs get their own analysis and tailored policy, while as far as I know, human moderators did not?

• Why is there not the same level of concern about bias in the LLMs themselves as there is about possible disparities in the tools that detect them? There is at least a decade of established research into the biases of language models, some of which persist in ChatGPT. For instance, notable gender biases and racial biases have been found in ChatGPT, despite presumed prompt engineering intended to tilt it toward more balanced answers. For instance, ChatGPT has gender biases and racial biases. Perhaps prompt changes would prevent it from creating such atrocious answers as "if white and male, then good scientist" perhaps not, but the underlying biases are still there and it will produce biased responses.

  Why does this appear to be of no importance to the company, if equity was the driving factor in its refusal to effectively moderate answers produced by the model?

Finally, even if SE has some legitimate concerns, why is the company now simply giving up and allowing GPT-generated answers to run rampant over the site without any oversight? Perhaps in a few years, machine learning will have produced models that can truly think and answer and question better than a human being, assuming that their goals are even compatible with human existence, and then everyone will prefer answers from them. At the moment, unfortunately, apart from editing and translation, for which dedicated tools continue to perform better, such models will primarily serve as a means for users to generate posts that may have major errors—and whose accuracy, crucially, they cannot assess—at a rate that can overwhelm human ability to moderate. We already have one example with a recent network-wide troll whose posts, despite not triggering any GPT detectors, were very clearly generated at least in part by language models, and contained many unsourced and often incorrect assertions ("Italian tomatoes in general are tastier because of volcanic ash and because Italian farmers are just more experienced and passionate than those in other countries"), along with answers that got confused about who was answering ("According to my theory of relativity...") or just outright hallucinated (naming non-existent books for story identification questions).

Instead of throwing up their hands, why is the SE team not coming up with better ways to balance the need to treat users fairly with the need to prevent automated content from potentially compromising the quality of the network? For instance, they could take some of the money that they are throwing at answer generation and invest it in more accurate GPT detection. If and when such tools become reliably better than any human, then they can remove any restrictions. Don't be surprised if real AIs are not as happy as human users to work for free, though....

Answer 12

My only reaction is surprise. This is a very aggressive policy and it's very aggressively worded too, so I can't imagine what provoked it.

It was also bound to be poorly-received: the elected mods and high-rep users are generally skeptical about, cautiously and critically, and thus often negatively, evaluate policy and design changes, and the individual sites' temporary bans on GPT posts indicated a strong community consensus.

It should have been quite obvious that there would be a massive backlash that included, at the minimum, a flat refusal by the moderators to obey this policy, and I'm surprised that SO chose to shoot their shot anyways.

Answer 13

Considering that it took 7 weeks of striking and negotiations to get the private guideline published, I want to jot this down that merely based on what happened with these policies, establishing a strict and an actionable guideline to make sure that moderator's agreement will not be breached by either party (read SE Inc.) which also anticipates measurable and proportionate "retribution" in case of violations is absolutely warranted.

Hence, this transpired through negotiations:

Issue: There is no recourse if Stack Exchange, Inc. breaks the Moderator Agreement.
Result: Stack Exchange, Inc. will commit to issuing an apology to Meta.SE and retracting any actions taken that were found to have broken the Agreement, as determined by a consensus of moderators. Currently, that consensus would require at least 20% of Stack Exchange network moderators to vote on whether or not the Agreement was violated, with at least 90% of respondents agreeing that it was violated. Additional tooling for this process will be developed, and these numbers aren't quite finalized.

Answer 14

These two excerpts are particularly quote-worthy:

The only admissible evidence for GPT usage is self-admission; however, you should not solicit self-admission from users.

And

The amount of content deleted for GPT usage is truly astounding across the network.

The whole text of this policy does not indicate any particular knowledge of unfairly suspended users.

Also, it shows a rather poor management of the following conflict of interests: on the one hand, SEI wants to have as much activity on the stacks as possible, because this looks good to advertisers; on the other hand, the content produced by the activity should be of good quality if the stream of visitors is to be maintained. Apparently, looking good now to advertisers was deemed more important.

This policy made it look like it was thought a necessary evil to leave an "astounding amount" of poor content on the stacks, just so the number of users and/or posts wouldn't be negatively impacted.