105

The policy below was posted on the Mod Team on May 29, 2023 to update moderators as to changes in how the GPT policy should be enforced. A day later, a shortened version of this policy was published on MSE. We normally do not release enforcement guidelines publicly. In recognition of the confusion caused by the differences between the public and private guidance in this case, we are now publishing the private text of this policy in full. However, we reserve the right to continue in the future to withhold specific enforcement guidelines from being published publicly in cases where doing so would make the jobs of moderators more challenging, and open up new avenues for abuse.

This is being released now so that it can be included in the public record. Since being published, this policy has been superseded, and the interim policy published. The historical policy begins with the line below this one.


May 29, 2023

In light of significant research on the efficacy and flaws of GPT detectors, we’ve been studying the impacts of these detectors on Stack Exchange. And I’ve got some bad news to share.

We made two major discoveries, which I’ll discuss in more detail in this post. To summarize, though, we have strong evidence that:

  1. none of the GPT detectors work as advertised on Stack Exchange data and have egregious false detection rates on our platform, and
  2. they are creating an alarming native language bias in suspensions on the platform.

The tl;dr: We are now asking you to stop using GPT detectors to substantiate claims of GPT usage and to stop suspending due to AI-generated content in general. Based on what we now know, we can’t admit GPT detectors’ results as reasonable evidence of GPT usage. To be clear: as of today, Philippe has instructed our team that GPT detectors will not be acceptable to establish reason-to-block. We will revisit this decision if evidence emerges that the issues with these tools have been resolved.

GPT detectors do not work as advertised and produce unreliable results

Based on significant and credible research that implies GPT detectors return excessive false positives, we conducted our own survey of GPT detector efficacy on Stack Exchange posts authored before ChatGPT’s November release. This survey concluded that GPT detectors misclassify 32% (+/- 6%) of non-GPT posts on Stack Exchange sites as having been written by GPT. This error rate is too high to be tenable for day-to-day use, and is certainly too high to establish cause for messaging or suspension.

We also discovered that GPT detectors often disagree with each other as to whether a post is GPT created. This means that strategies relying on one of two detectors returning a positive result could lead to a baseline error rate of around 50% - and the incorrect detection rate gets worse the more detectors are added. In other words, 50% of posts detected as positive using an “either detector” strategy could be incorrectly classified as AI-written.

However, we did not find a way to reduce the error rate far enough by combining GPT detector results. Relying on multiple GPT detectors returning positives at the same time will return an error rate in the range of 12% (+/- 7%). We expect that combining the results of additional detectors will not return more meaningful results. This is still too high for operational usage, particularly at the volume GPT messages and suspensions are issued on the platform today.

If the situation changes and a GPT detector is released that meets our criteria for operational efficacy, we will revisit this policy change.

GPT detectors are creating bias against non-native English speakers

Even more alarmingly, recent research shows that GPT detectors exhibit overwhelming bias against non-native English speakers, and misclassify their writing as authored by GPT at a rate of ~45-80%. Available detectors are also unanimous in this misclassification around ~19% of the time. This risks unjustly targeting non-native English speakers on the platform, which comprise a significant majority of participants on some of our largest sites. (I would also hazard to say that non-native English speakers comprise the majority of participants on a majority of sites across the platform.)

We conducted a survey of the site using our best available information, on the assumption that biases against non-native speakers (a feature we cannot analyze) would show up as biases for and against specific countries (a feature we can analyze).

Our results strongly suggest that actual mod messages sent about GPT to users on the platform are significantly and inappropriately biased, likely driven by biases in the underlying tools used to flag GPT answers. To be clear, we do not believe that moderators are consciously acting with any sort of bias, but that the tools that they are using for this task have that bias built into them.

Regions of the world that appear to be unjustly targeted by GPT detectors in use on our platform include southeast Asia, the Middle East, and most of Africa. Regions of the world unjustly benefited by GPT detectors include North America, Europe, and Oceania.

The effect sizes are not small. For example, Pakistan receives 3.6x more suspensions for GPT usage than their baseline participation rates should imply. Bangladesh receives 2.7x more suspensions than the base participation rate justifies. India receives 1.75x more suspensions. On the other hand, the United States receives 0.6x the suspension rate, alongside Sweden, Great Britain, and Australia. While we unfortunately can’t share the raw data from our conclusions, we hope that these data points help to convince you that our alarm is justified.

Bear in mind that some of the reason we suspect GPT detectors create this bias is because they are detecting patterns in language that non-native speakers rely on. Therefore we also cannot accept similarities between the structure of a user’s answers alone as an indicator of GPT usage.

Only self-admission, freely given, can count as evidence of GPT usage on the platform

In light of the above, there are no tools currently available (at least so far as we are aware) that can identify GPT usage successfully. Therefore, the scope of admissible evidence for establishing GPT usage is, in the current state of things, very thin.

  • GPT detectors cannot be counted as evidence for the reasons above.
  • User behavior, such as answer timing, could be correlated with GPT usage, but a hunch alone does not a suspension make. Instead, this signal has to be taken as a prompt to review the objective quality of the user’s contributions.
  • You may consider things that have obviously been copy pasted from ChatGPT such as literally mentioning the knowledge cutoff date from ChatGPT or phrases that include things like "If this didn't help, you can use a forum like Stack Overflow and ask for help there" to be self-admission and AI generated.

Therefore, the only admissible evidence we can currently permit for GPT usage is self-admission by the author of the posts, freely given. Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

As a final reminder, suspensions should only be issued for real behavior, actually known to be malfeasance. We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics.


We are aware that this leaves the state of GPT enforcement on the site in pretty rough shape. The silver lining is that these tools don’t really seem to have been working as designed anyway.

Here are the key points we’d like you to take away from this post:

  • We are now requiring you to ignore and/or decline reports using online GPT detectors or intuition as their basis. Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.
  • Do not direct users to GPT detectors or encourage them to act on their results in any way. Users, as well as moderators, should not be using these tools to determine post and user outcomes.
  • The only admissible evidence for GPT usage is self-admission; however, you should not solicit self-admission from users.
  • Remember that mod messages and suspensions are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unvetted heuristics.
  • Effective immediately, Philippe has revoked the temporary policy change allowing moderators to jump to 30 day suspensions for GPT usage. In light of the current evidence, we simply do not trust that the tooling exists to support suspensions with the required degree of confidence. Although this policy was only formally enacted on Stack Overflow, any network sites that have informally been working under such a policy should also now consider that policy overturned, and should no longer be acting on it.

As far as the existing suspensions go, we will let active suspensions that were issued for a time period of less than 90 days expire organically. We’ll only review these suspensions if users contact us. For suspensions that were issued for a time period of 90 days or greater, we’ll manually review these to determine if they can be removed (they’re not just flooding bad content, for example).

Low quality content

You may still naturally deal with content abuse in ways we always have: if someone is posting a flood of poor quality content, suspend them for low quality content. Just don’t make it about GPT or AI generated content - make it about the quality being poor on a repeated basis. This includes things like repeated answers that don't address the question.

If someone is posting a flood of content to promote their tool/site/blog, use the astroturfing policy we have.

One final note: The amount of content deleted for GPT usage is truly astounding across the network. While we don’t doubt users have been using GPT and other language models to post answers, if you’ve been using GPT detectors to mark individual answers for deletion, we’d ask you to consider carefully whether any such content needs undeleting. Keep in mind, false positives are dramatically more likely for authentic writing by users who do not natively speak English, and may run as high as 80% for some detectors; in these cases, users may not deserve to have their content removed. Additionally, it remains possible that a false positive on one answer means false positives on other answers by that user are more likely to occur.

If you have questions, let us know in the answers below and Philippe is also willing to schedule a meeting in the TL if needed to discuss it further.

41
  • 251
    Upvote to indicate satisfaction that you are publishing this, obviously not so much with the policy itself.
    – tripleee
    Jul 26, 2023 at 16:04
  • 104
    Reading over this now, I can understand why mods were upset by it. I wonder if the company understands why this policy is so offensive?
    – ColleenV
    Jul 26, 2023 at 16:19
  • 82
    I chose to upvote @tripleee's comment instead of the post to avoid confusing anyone into thinking I like the policy. Jul 26, 2023 at 16:27
  • 46
    For the record, the policy was published under the title "Please stop use of GPT detectors for content moderation on Stack Exchange" and they lit up the diamond orange on the day it was published to try to get the attention of every mod.
    – Laurel
    Jul 26, 2023 at 16:37
  • 9
    Voting on this post is extremely confusing. Should one upvote to appreciate the disclosure of the secret version of the policy or downvote the fact that it was secret and until now the company position was that they never asked the mod to de-facto stop moderating AI generated content altogether? Sorry, but I had to choose the second option
    – SPArcheon
    Jul 26, 2023 at 16:48
  • 8
    I can see how the incorrect stance of "moderators don't want to explain their decisions" could cause such a misguided policy. Hoping the superceding one is more grounded in reality, but we will see.
    – GammaGames
    Jul 26, 2023 at 16:50
  • 32
    "We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics." That is indeed setting the bar very high. How is it with spam detection for example? Aren't there a lot of moderation cases that are basically decided heuristically. If this is still the position of the company I foresee difficult discussions ahead about how to detect AI content. But it looks as if another policy will be published shortly with this one already been superseded. Let's see what is in there about "kicking users on the basis... of guesses". Jul 26, 2023 at 16:52
  • 27
    Thanks for making the old policy public. What a sh**y policy indeed. :/ Jul 26, 2023 at 17:42
  • 33
    I have upvoted for two reasons: First because this post marks this extremely bad policy as historical and second because it was finally made public. I only wish this was done sooner. But better late than never. Jul 26, 2023 at 17:49
  • 20
    Interesting how the company readily admits GPT detector accuracy is horrible, but in the same stroke permits these unreliable GPTs to be used to write answers on the network. In other words, moderation accuracy abruptly became more valuable than content accuracy. Worse yet, this was done on an incorrect assumption, without consultation. As a result, I don't think the rationale provided for the policy change was in good faith by the company. I appreciate this step to fix the situation, though.
    – ggorlen
    Jul 26, 2023 at 19:54
  • 68
    A two times higher GPT suspension rate for a country doesn't have to be a bias, it could simply be that there are more GPT enthusiasts in that country. There was an assumption (all countries behave equally with respect to GPT) in the logic of the company, or wasn't there? Jul 26, 2023 at 20:27
  • 8
    The type of bias SE alleged in the question quoted here, if it's actually bias and not caused by a difference in actual usage, is definitely important and something that we should work hard to stamp out. There should be more investigation. However, only Stack Overflow, the company, can perform that investigation, because they are the only ones who can correlate users to geographical locations in bulk. Now that we're moving towards agreement on ways for humans to identify AI-generated content, rather than only self-admission, it might be possible to investigate potential bias further.
    – Makyen
    Jul 26, 2023 at 22:13
  • 69
    I can see why the moderators went on strike: this policy stops just short of openly accusing them of being racist robots blindly applying the results of GPT detectors.
    – Mark
    Jul 27, 2023 at 1:32
  • 15
    As an AU Moderator I have upvoted this post, not because I agree with any of the words in it, but because it has finally and belatedly revealed the pressure applied on Moderators by Stack Overflow, Inc. away from the public gaze...
    – andrew.46
    Jul 27, 2023 at 4:39
  • 8
    Also ".. in pretty rough shape .." was probably still an understatement. In complete shambles may have been more appropriate because it basically means that the very popular ban on AI generated content was basically completely canceled on May 30. Since then it's a ban in name only and this state now endures for two months already. Depending on how one much value one attributes to such content a lot of damage will have been done in the mean time. It will be very interesting to see how any superseding policy will try to uphold the ban or not. Jul 27, 2023 at 7:53

14 Answers 14

132
+100

Prologue: meta

Much as I've wanted to, I've refrained from saying much about this situation thus far, primarily because this policy wasn't public.

Even though I knew the gist of it—for folks had reached out to me privately about it, there'd been discussion of the issues with it (though not its contents) on the meta Discord, and the comments on and around your public update were damning enough... I feel that it is pointless to try to have a discussion with someone about a topic they are determined not to discuss, and... Keeping this policy private has been a deafening shout to that effect every day that this ordeal has dragged on.

Today, it's public, and today we can talk. Plenty of other folks have (re-)stated the practical issues with this policy, so I'll mostly limit my points here to meta-observations: how communication has progressed (or failed to do so) more so than what has been communicated.

And, that's sorta my thing.

Introduction: a spiderweb of concerns

Ever throw a handful of tennis balls in front of a puppy? They dodge and leap and don't catch any of them. Most of us are the same way: the more issues we see in front of us, the harder a time we have latching onto any of them; as problems multiply, less gets done, and even less gets done well. Pretty soon, we find there are more issues than we started with, even as we run around breathlessly trying to stay on top of everything...

I wouldn't blame anyone at SO for feeling that way right now, especially reading through one of these meta threads. Y'all started with A Big Problem, and tried to put together a policy change to address it... Now, just a couple of months later, you have a pile of new problems, cracks spreading out from your attempted patch.

...And of course, the original Big Problem still exists too. I don't know what to do about that—if it could be solved by someone sitting and drinking whiskey while waiting for git to finish something, it'd have been solved years ago. But the spiderweb-cracks, that I know something about.

Root cause: failure to communicate

If I was in your place right now, I'd have a big spreadsheet open, filled with links and summaries of the feedback on the other ends of those links. I'd be staring at those summaries, looking for patterns: what shows up repeatedly, what commonalities do disparate complaints share? YMMV, but personally I've found this to be a great exercise when faced with a seemingly-overwhelming flood of concern.

Here's what I think you'd observe, were you to try this exercise:

  1. Folks do not feel like you're being candid with them. About your concerns, your plans to address them, your rationale for why those plans might work... About what you're seeing here, what you're hearing from folks on Stack Overflow, or Mathematics, or Super User... About what you're hearing from folks within the company, its parent company... About how you're feeling—about all of this, about the even greater tumult at your alien-themed former employer, about the seemingly impending sea change in this industry as a whole... And especially, about how all of that is affecting the decisions being made here.

  2. Folks do not feel like they're being listened to. They don't hear their concerns being echoed back; instead, they hear reductive summaries of their concerns. Summaries like, "confusion". Even when they go well out of their way to produce effective summaries for you.

Effective communication requires understanding; these black-and-white symbols don't carry enough information on their own, and we need context to interpret them—what sort of a person has typed them, what are their goals, their hopes, their fears?

Effective communication is based on shared context... And some amount of vulnerability.

Folks here are pretty vulnerable right now. They're trusting you to not abuse that, but rather to use it to understand them a little better.

Public by default: an exercise for self-improvement

Years and years ago, there was a regular company-wide call at SO, where various teams would update everyone on what was happening with them. These calls used to start with Joel running down a list of precepts...

One of those precepts was "public by default". He'd cover the topic by listing the things we couldn't talk about publicly—the implication being, most anything else was fair game. It was a short list—it had to be, lest it cut into the time needed to remind folks to clean the steam wand on the office espresso maker.

...It also had to be a short list because once you have to stop and consult a list, folks tend to default to not communicating. Either you know, know, at any given point in time that you're free to talk about your work... Or you don't do it.

Now, I know this isn't really a thing at SO anymore. It had already fallen out of fashion long before my departure, the former deluge of writing from SO employees reduced to a trickle.

...But here's why you should consider reviving it for yourself, and ideally your team:

... Firstly, on a technical level, there’s an immediate feedback loop. If you’re answering questions on forums or online communities, considering strategy or contributing to open source, you have such a swift feedback loop that it’s impossible not to improve.

What’s more, the hive-mind of the internet has a habit of taking ideas you might have which are ok and turning them into something worth pursuing, adding value through different angles or directions. ...

To truly embrace public-by-default, it’s not enough to share your successful projects and knowledge, but additionally to bring the humility to share your learning and failures. In general, it’s hard to argue with the sentiment that you think harder about everything you do if you’re going to share instead of just do it for yourself, no matter how trivial it might be.

Those are quotes from Ben James in a blog post he wrote for the Stack Overflow blog. You'll note that he adopts the "enlightened self-interest" approach to selling this technique: don't communicate as a charity for others; do it because it makes you better at what you do!

And he's right: it does. Just like any form of regular practice, defaulting to talking about what's going on publicly allows you to become comfortable with your tools and strengths, adept at compensating for your own weaknesses, quick to see opportunities for improvisation and nimble at avoiding pitfalls.

Sharing your failures also allows you to be vulnerable at times when your audience is... Maybe a bit more predisposed to cutting you some slack.

But most importantly, it allows you to build that shared context I mentioned earlier, so that when you really, really need to communicate effectively... You can.

Fear and loathing in the age of large language models

Lacking so much shared context, there's an awful lot of speculation happening right now; when folks don't have the information they need, we tend to ... make up something plausible. Or failing that, something entertaining.

...so for the record, I don't actually believe SO, Inc. is operating under the control of alien templar yeti. Yet.

After reading this policy, it's clear that there's a lot of speculation going on WRT unknowns internally as well. And small wonder: if these trends continue to accelerate, your bread & butter will be gone before there's much time to react!

But... There's always some boogieman like that in the shadows ahead. Some new threat that needs a new strategy to thwart. That's not a reason to clam up, retreat into the shadows yourself... Rather, it's the most important time to step up, pull together and raise our torches high. To take that speculation and turn it into testable theses... and then test them, share the results, and by working together and sharing that knowledge gained, improve things for all of us.

If LLMs are to bring an end to all this, don't let it happen through fear and intimidation. Face the threat with open communication, sharing, and collaboration...

Because that... Has always been The Stack Overflow way.

9
  • 57
    Lest anyone think you are making a joke, I want to emphasize that the two standing items I recall from all-hands meetings were "default public" and "clean the steam wand". When I did make it to the office, you better believe I remembered to clean the steam wand. And I still practice "default public" to a degree that makes my co-workers nervous at times. Jul 27, 2023 at 5:34
  • 13
    "I don't actually believe SO, Inc. is operating under the control of alien templar yeti. Yet." – Excellent... They suspect nothing. (In all seriousness: Great writeup!)
    – V2Blast
    Jul 27, 2023 at 15:45
  • 27
    In my experience, the mix of poor communication & lack of communication is often at the root of "relationship strife" ... Mix in a lack of trust, and you've got total dysfunction. I wish I could upvote this more than once. I can only hope that my good friends Nog Shine and ATMoo see this, share my enthusiasm, and vote accordingly.
    – AMtwo
    Jul 27, 2023 at 18:12
  • 4
    @V2Blast there are some leaked photos tho: i.sstatic.net/zBwYQ.png
    – M--
    Jul 27, 2023 at 22:58
  • 1
    Question - does SE even have physical offices with fancy expresso machines any more? Cause if it didn't... the ol "You had ONE job meme comes to mind" Jul 28, 2023 at 6:07
  • 5
    @JourneymanGeekOnStrike: It appears the company is only offering remote jobs. The espresso machines were high-end La Marzoccos retired from operation at Starbucks. If you didn't clean the steam wand, there was a chance milk would get sucked into the boiler as it cooled. If that happened, it would be very difficult to clear out. Still, those espressos were amazing. Jul 28, 2023 at 17:51
  • 1
    Also you wouldn't need spreadsheets - MSE is very good about documenting these things. We don't even need to nerdbait people. There's a summary of network wide posts on meta on this topic and others Jul 31, 2023 at 13:14
  • 3
    I find that compiling these lists is more valuable than just having them, @journeyman - something about taking the time to read and understand enough to summarize makes it easier for me to see the full picture than just seeing a digest. That isn't to take away anything from the lists you're referencing - they're a great starting place!
    – Shog9
    Jul 31, 2023 at 19:47
  • Ah but if you don't have them... They are there 😅 Jul 31, 2023 at 23:50
124

Honestly, this just shows the lack of trust/communication between the company and moderators and the community (but again thanks for publishing this). The policy focuses on the reliance on GPT detectors, even when mods have said they did not use them. You didn't even talk to moderators before forming your ill-informed opinion.

Also,

Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.

This is just patently false. What gave you this impression? Certainly not talking to the community. We have been trying to reach out to you guys for a long time, and we were repeatedly ignored. It took a moderation strike for you to finally listen to us, which says something about how much the company values its volunteers.

It's pretty ironic; an announcement about strict requirements of evidence (analogous to proving Treason in the U.S.) that wasn't made with the same rigor. You need to hold yourself to the same standards that you impose on others.

3
  • 16
    No, it’s not really just about a mistrust between the two parties; rather, I’d say the focus on AI detectors is just a cover and excuse for, the important part: «and to stop suspending due to AI-generated content in general», which is driven by the hype for genAI, which is pushed for dubious reasons by the SE company. Jul 28, 2023 at 16:43
  • 6
    And, when the goal is to push forward with the hype for AI, it makes sense why they haven’t been listening to us, because we have not satisfied them with what they really want to hear. Jul 28, 2023 at 16:53
  • 3
    I (and I imagine the mods) don't need a detector script to know that answers that meander, contradict themselves, and contain flagrantly made-up "facts" were written by a LLM. Aug 4, 2023 at 20:11
79

First, I appreciate this being posted. It feels like a move towards transparency, and we need more of that.

And since this has been superseded, disputing the contents of it might be of questionable worth, but I'll note these two points in hopes that they are "fixed" in the new policy.

  1. (I'm not a mod, so this is drawing on reports from mods that I've read.) This policy is written as if mods were predominantly, if not exclusively, using GPT detectors to decide if something was GPT-produced, and from all reports this is incredibly and widely off the mark. Were mods even consulted before basing the policy on this assumption?

  2. "Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad." I'm trying to stay dispassionate here, so I'll just say that, speaking for myself, this is very, very incorrect. I don't even know the name of a single detector, and have at least a mild sense of when something might be GPT-generated, which is usually confirmed when closer readings disclose the fantasms, errors, and missing-the-point-nesses that LLMs produce but people rarely do.

I'll avoid discussing why this policy was produced, because it'd be based almost entirely in speculation on motives and not evidence. But hopefully the two points above, dealing with concrete facts, are helpful.

5
  • 53
    The answer to 1 is "no", mods were not consulted. They also seemed to assume not only that mods were predominantly using GPT detectors, but that they were using a threshold of "50%" on the detectors, and basing the failure rates reported in this policy on that threshold (this is discussed at meta.stackexchange.com/questions/389928 ). That is, they seemed to assume that when a detector was "unsure" but leaning towards AI, mods were taking that as strong evidence of AI. Jul 26, 2023 at 16:18
  • 53
    "Were mods even consulted before basing the policy on this assumption?" - no, and our feedback pointing this out far too many times has been ignored after the fact too.
    – Zoe
    Jul 26, 2023 at 16:19
  • 8
    @BryanKrauseisonstrike (and Zoe) - That was my impression too, but I wasn't involved. I would love to get feedback from the people who made this policy - like, did they even think consulting mods was a possibility, or useful? I would love it if we not only fixed this policy, but also fixed some of the root causes that led to it. I'm guessing this is a case of "I'm new to Meta, because these are long standing issues and there doesn't seem to be interest in fixing them ", but I thought I'd throw it out there.
    – JonathanZ
    Jul 26, 2023 at 16:30
  • 9
    @JonathanZonstrike Indeed, I would certainly welcome these discussions, and I'd like to see the company explicitly mention these parts of the process as problematic to encourage an impression that they do find it problematic that they never checked their assumptions with moderators. Jul 26, 2023 at 16:32
  • 5
    I suspect I am like many users (and mods) who have never used automated tools to check for LLM generated posts. I work on the basis that the post is good or bad. Bad posts get voted down (or VTC'd) and good posts don't. SE/SO should not assume we're acting based on (what they claim) are unreliable tools. Most of us don't use them and would find it an incredible chore to do so on every vaguely suspicious post. Jul 26, 2023 at 23:32
62

The original publicly available description of the policy stated that:

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written

Therefore, we are not confident that either GPT detectors or best-guess heuristics can be used to definitively identify suspicious content for the purposes of suspension.

The actual content of the policy states that

We are now asking you to stop using GPT detectors to substantiate claims of GPT usage and to stop suspending due to AI-generated content in general.

We are now requiring you to ignore and/or decline reports using online GPT detectors or intuition as their basis. Remember that users have honed their instincts as to what a GPT contribution ‘sounds’ like based on input and signals from GPT detectors that are now known to be bad.

Almost immediately you received replies like:

This post does not match the guidance given to moderators. – Bryan Krause is on strike May 30 at 19:52

"In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user." The guidance I have received is: "We are now asking you to stop [...] suspending due to AI-generated content in general". How do I reconcile these two statements? – Andy May 30 at 19:57

I think it was clear that the moderators didn't see the two version as equivalent.

My question is simple: why did this happen and why it wasn't immediately cleared up to the general public too?

The public version sounds far less aggressive/assertive, almost like you wanted help on finding a better way. This sentiment is in my opinion completely absent in the private version.
I think you may very well realize that from the outside this looks like a deliberate set-up: showing a "reasonable" version to the general public while the moderators were given Order 66 instead would be a convenient strategy to make them look unreasonable and unwilling to cooperate since all the general public would see would be a far less restrictive suggestion.

Basically, it looks like the company purposely withhold some information when talking to the userbase in order to try to undermine the credibility of the moderators actions by painting them as a group of people overreacting to requests that were never made. Since we can assume that the company had foreseen the consequences of the new policy, and considering that apparently the misinformation campaign wasn't limited to the general public (see the notes on Jon Ericson blog ) this interpretation sadly sounds quite believable.

7
  • 16
    It actually was cleared up immediately. CMs were enforcing the Teams(private) version, not the MSE(public) version. That split in guidance was what fired up a number of moderators (including myself) (as noted in my original writeup)
    – Machavity
    Jul 26, 2023 at 18:52
  • 40
    Technically, "we've asked moderators to apply a very strict standard of evidence" was not incorrect, just highly misleading. Yeah sure, they did what they said in the public version. Except that the "very strict standard of evidence" was only "self-admission by the author of the posts, freely given", which is hilarious. Of course, no one reading the public version of the policy would think that this was the only acceptable form of evidence by any stretch of the imagination.
    – 41686d6564
    Jul 26, 2023 at 19:04
  • 5
    To be honest, I'm kinda glad they did not release the private policy to the public. It would have given a free pass for everyone trying to game the system.
    – Bergi
    Jul 26, 2023 at 19:37
  • 8
    @Bergi And that'd be my guess as to why the statement read the way that it did; it seems to me like the company refrained from making the specifics public in order to avoid writing instructions on how to game the system (e.g. the same reason the specifics of the voting fraud script are kept secret). The problem, of course, is that the gaming was already occurring, and the flood of generated content was already happening and didn't care about the line the policy drew.
    – zcoop98
    Jul 26, 2023 at 19:51
  • 1
    @Machavity it would've been better if you've written "up" right after "fired" :D
    – M--
    Jul 27, 2023 at 0:01
  • 1
    @Machavity By "cleared up" I meant "why didn't the company admit that the moderators weren't making up things". From the point of view of a normal user, moderators were making claims about requests that the company had never made. So, if you hadn't any reason (for example a long list of previous mishaps) to trust the moderator words, you could easily see them as a bunch of overreacting folks not willing to cooperate with a "fine enough" request. And this is precisely why the "split in guidance" looks like a deliberate setup.
    – SPArcheon
    Jul 27, 2023 at 7:09
  • 3
    @M--ßţřịƙïñĝ: I've just edited Machavity's comment to make that change, so users don't misunderstand what Machavity is trying to say. (It might make the punctuation look a little odd, but I figure that's still better than folks misunderstanding the comment.)
    – V2Blast
    Jul 27, 2023 at 15:41
52

Well, in the interest of the historical record and hopefully leaving something educational for future readers - to me there's a few specifically problematic things.

Firstly - while there were clear attempts by the moderator communities to engage the company with respect to the problem of folks posting unverified GenAI answers at scale. We posted on meta, and based on the feedback provided to us at the time. That we ended up with a patchwork of per site policies, including up to a 1 year suspension, and we didn't get any communication until this. No consultation. No warning. This also kind of undermined us when dealing with what we deemed to be problem users.

In addition

  1. Stack Exchange made an assumption we were using ChatGPT detectors. Many of the flags were from folks using it to confirm their hunches. However we made the determination based on other factors like velocity of posting and writing style.

  2. Regions of the world that appear to be unjustly targeted by GPT detectors in use on our platform include southeast Asia, the Middle East, and most of Africa. Regions of the world unjustly benefited by GPT detectors include North America, Europe, and Oceania.

    This makes a bit of a awkward situation. If folks there are using GPT more - they'd tend to be suspended more. Correlation isn't causation. I'd also note we also get a good chunk of our spam from literally 2-3 Indian and Pakistani metropoles. Would that mean they're unjustly targeted, or there's just more of it? There's various other tells that I believe the (informal?) folks looking at heuristics can shed light on, like a sudden velocity in decent quality posts and some unusual conjectures. That said, this could be read as a potential accusation of racism.

  3. Therefore, the only admissible evidence we can currently permit for GPT usage is self-admission by the author of the posts, freely given. Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

    That's an impossibly high barrier. And I feel like this was known.

    And somehow we're protecting random folks who could be causing the site harm, over trusting the moderation teams to protect their sites.

    The attitude here is one of the reasons I'm striking.

  4. Next April I'll have been moderating for a decade. Most moderators have a deep knowledge of their communities, and to an extent the subject material.

    As a final reminder, suspensions should only be issued for real behavior, actually known to be malfeasance. We just can’t endorse kicking users off the platform on the basis of hunches, intuitions, guesses, or untested/untestable heuristics.

    Quite a lot of moderation starts off on a hunch, and intuition built up over years. This literally smacks of a lack of trust. I try very hard not to tell community managers how to do their job - and I'd expect the trust to do what I do. And I'm open to being told when I'm wrong. Just... when, not if I might.

    I'm not angry at this. I just don't think its a great way to work if we can't trust moderators - folks who're practically in positions of community leadership for their respective communities. It also shows (well one of the things that do) that there's a lack of understanding of what we do.

6
  • 4
    "That said this could be read as a potential accusation of racism." If the user has a generic (userXXXX) or English username and hasn't mentioned their location on their profile, normal users cannot see where they are based though. Devs (and perhaps mods) can check IPs, but I, casting flags as a normal user, cannot. Would be interesting to see how many posts by users with non-disclosed geographical location were actually flagged. And of course not everyone with an IP in the US is necessarily a native American-English speaker.
    – Adriaan
    Jul 27, 2023 at 8:58
  • Also if they were using ChatGPT or some other GenAI tool, most of the regionalisms are stripped away... But it still sounds a little like it 😅 Jul 27, 2023 at 9:23
  • 1
    So, once again, this is a Catch 22. You can't automate moderation because GDPR, you can't use tools to assist you because they don't work and you can't do the work yourself because you may be biased.
    – SPArcheon
    Jul 27, 2023 at 16:47
  • 5
    We can't automate moderation cause there's no reliable automated way to check for it. Jul 27, 2023 at 16:49
  • 4
    "Correlation isn't causation." A million times this. And disparate impact isn't bias, either. Aug 1, 2023 at 7:26
  • 2
    At least in my experience, the content I personally was flagging, the grammar was absolutely perfect. I was identifying the content might be generated by OpenGPT before I even got to the point of identifying the name of the profile. So to be told that the company believed there was bias (knowingly or unknowingly) happening is really concerning. If their own statistics were showing those regions were using OpenGPT more, that’s due to the high quality standards the community has tried to achieve, and those users trying to achieve then by mistakenly using the wrong tool. Doesn’t make us bias.
    – Ramhound
    Aug 4, 2023 at 9:52
49

First: Thank you for finally rescinding this insane policy. Second: Thank you for finally allowing me to see why the moderators had to go on strike

There is an idiom, "The devil is in the details" which comes to mind when comparing the secret and public versions of this policy. The difference between the two, when comparing paragraph for example, might be minor. The details of implementation, however, are a bit farther apart.

I trust the moderators and respect their ethics. Yes, I know they are human, each with their own faults, unique to each as an individual. Somehow they, collectively, always seem to stay on the high road anyway. Because I trust them I "knew" there was some kind of disconnect between the secret version (which they were required to enforce) and the public version (which I could expect to be followed). Since they were willing to go on strike to effect a change, I figured it was certain to be a non-trivial difference. I had no way of knowing just how vast the gap was.

Simple interpretation of the public version:

Be very, very careful about what you consider to be AI-generated content, and be "gentle" with the consequences.

Simple interpretation of the secret version:

Unless someone admits to posting AI-generated content, without you asking, it is not AI-generated. If they do admit it openly then you can remove it. Never suspend for posting AI-generated content.

I fail to envision any future egalitarian utopia community wherein the only possible evidence of an offense against the community is the spontaneous, unsolicited self-admission of said offense.

What was anyone, someone, everyone, possibly thinking, if they were thinking, which prompted the text, and qualifiers, of this absurd version of a "policy"?

13
  • 2
    There is certainly a clear substantive gap between the two policies, but personally I had now way of knowing just how vast the gap was. I've been to the Grand Canyon, and experienced that chasm's awe-inspiring "gap." The gap in the two policies seems more akin to a Jovian version of the Mariana Trench in scope and scale. seems like unnecessary hyperbole here, it distracts from the message of this answer.
    – TylerH
    Jul 26, 2023 at 21:18
  • 3
    @TylerH Hyperbole it may seem to you. Shocking it was to me. Lacking an inside view of the "real" policy, and imagining the potential contents thereof for the past couple months left me utterly unprepared for the facts I found above.
    – Chindraba
    Jul 26, 2023 at 21:23
  • 1
    I love the summaries of the two versions and the general sentiment here. I'm also less enthusiastic about the hyperbole.
    – ojchase
    Jul 26, 2023 at 21:28
  • Hyperbole cleaning edits welcome. The message is much more important than the ego.
    – Chindraba
    Jul 26, 2023 at 21:53
  • "First: Thank you for finally rescinding this insane policy" -- when/where did they actually rescind it? I missed that. IIUC, all they did was post the "insane policy", which is progress, to be sure.
    – ggorlen
    Jul 27, 2023 at 0:02
  • 1
    @ggorlen Bold text in the last paragraph of the introduction. "Since being published, this policy has been superseded, with the new policy available shortly."
    – Chindraba
    Jul 27, 2023 at 0:04
  • @Chindraba_on_strike Ah right, thanks--I took "shortly" to mean 6-8 weeks. The language they used didn't make it entirely clear that the policy was really totally gone until the new one arrives, which may take awhile, and the new policy may well be more or less the same as the old one in, so the situation doesn't really seem concretely better just yet. Also, I'm not sure "rescinded" === "superseded", but yeah, nitpicks aside it does seem like this policy is on the way out.
    – ggorlen
    Jul 27, 2023 at 0:05
  • 1
    @ggorlen It's actually nice that the above policy is gone before its replacement arrives. A minor nod from the company that the mods can still be trusted. As to the replacement, it's being worked on by mods, and curators with mega-experience with genAI content, and should not (hopefully) take another 6-8. The "usability" of the future policy is sufficiently tied into the strike that I'm sure it's going to be acceptable to those of us who only "curate" on the sites.
    – Chindraba
    Jul 27, 2023 at 0:24
  • @Chindraba_on_strike Thanks for the info--we'll see how it goes. I'm looking forward to getting back to answering questions and flagging stuff once the strike ends.
    – ggorlen
    Jul 27, 2023 at 0:27
  • Almost a week later, still no updated policy.
    – ggorlen
    Aug 2, 2023 at 15:50
  • 2
    @ggorlen Of course the Moderator Agreement requires that there be a delay for moderators to give feedback, which is also part of the issues in the strike, so the official [tagged] policy will be delayed, that it exists, and what it should contain is now available. Your wait is over.
    – Chindraba
    Aug 2, 2023 at 21:50
  • @Chindraba_on_strike Thanks, that's a huge step forward and good news, although it's from the Mods rather than SE, so I may stay on strike until the company has posted the new policy and it's been greeted favorably by the mods.
    – ggorlen
    Aug 2, 2023 at 22:11
  • 1
    While less than "official" it is cosigned by Philippe in the 3rd comment. Posting the new policy now, however much I'd love to see it, would actually be a direct violation of the final decisions reached. Flip-side, of course, is that SE cannot say when the strike is over, only the folks on strike can. Individuals, which is everyone on strike, will make their own decisions, but as a group the "is this final?" question is being discussed, and "looks" to be a yes - from the group view. Another couple days and the collective decisions will probably be evident as the sites begin to be repaired.
    – Chindraba
    Aug 2, 2023 at 22:47
48

For example, Pakistan receives 3.6x more suspensions for GPT usage than their baseline participation rates should imply. Bangladesh receives 2.7x more suspensions than the base participation rate justifies. India receives 1.75x more suspensions. On the other hand, the United States receives 0.6x the suspension rate, alongside Sweden, Great Britain, and Australia. While we unfortunately can’t share the raw data from our conclusions, we hope that these data points help to convince you that our alarm is justified.

I've said it before and I'll say it again:

[...] how exactly does that indicate antagonistic bias against those users? Have you ruled out the possibility that users from those countries are just proportionally violating the current AI-generated-content policies more than people from other countries? Or that a larger proportion of users from those countries are violating the policy compared to proportions of users in other individual countries? The reality is that there are countries where lying and fraud are just part of business / culture. Why would you rule out the possibility that certain cultures or subcultures care less about following the current AI-generated-content policy and won't mind violating our policies to do... whatever it is they're trying to?

For example, Wikipedia has a page on List of countries by intentional homicide rate. If you look at that page, you will find that the rate distribution by country is not- in fact- uniform (flat)... Does that make the people who sourced or compiled that that information... racist? Obviously not. So even if I observe that usernames or profile pictures of users who I have strong reason to suspect for violating per-site policies- such even as self-admittance/"confession"- are coming from a particular demographic (which I do), neither does that make me racist- as long as those usernames and profile pictures do not form part of my analysis (which they do not).

Adding to that, there is data that shows that developers in different countries have different sentiments about AI tools: Your 2023 SO dev survey.

  • 83.6% of respondents from India use or plan on using AI tools, and from Brazil, 78.0%. (the top two countries in that response category)
  • 82.2% of respondents from Brazil view AI tools favourably. (the top country in that response category)
  • 15.4% of respondents from India think the most important benefit to AI tools is improved accuracy in coding, and from Brazil, 13.4%. (the top two countries by proportion of that response)
  • 55.2% of respondents from India trust in the accuracy of AI, and from Brazil, 45.0%. (the top two countries in that response category)

And then there's the fact that sideshowbarker, who (from my understanding) was carrying out a lot of the moderation of ChatGPT flags on SO, didn't even use these detectors.


Please note that “freely given” is important here: please do not, under any circumstances, try to trick users into admitting GPT usage, lure them into saying it, or otherwise coerce a response. Even a user saying they have used GPT in general may not count unless they specifically say they have used it here, or for this contribution.

But why? How is a solicited admission any less legitimate than a freely given one?

It crossed my mind that there could be concerns about insulting people upon "false-accusations"/false-positives, but if one has enough experience recognizing ChatGPT content, the risk of accidentally causing insult to someone who didn't actually use ChatGPT is lower.

And really, I wouldn't find "hey, out of curiosity, did you get this from ChatGPT" that insulting. It'd ruffle my feathers a bit, but I'd just say "no, I wrote it myself", custom flag as "No longer needed. And remove my reply comment too, please.", and move on. Granted, there might be some survivorship bias here, since I've had enough grit to stick around past other minor things that the average person might feel indignant or hurt enough to leave by.


The amount of content deleted for GPT usage is truly astounding across the network.

The amount of new and old AI-generated content that I continue to find on a daily basis on SO is also astounding.

12
  • 19
    I've been struggling to write an answer expressing this sentiment, and I think you did a good job. Inherent to that piece of the policy is an assumption that GPT usage/ posting is flat across different language-preferences and regions. Did they check that assumption? Did they realize that the policy they constructed was built on two crucial assumptions (this one + GPT-detector usage) that they seemingly never stopped to validate or converse with mods about? Add that to the mismatch between the public and private policy details, and you get a recipe for an absolute disaster of a policy rollout.
    – zcoop98
    Jul 26, 2023 at 21:10
  • 15
    This. I'd absolutely want to know what, if any, weight was given to the cultural differences of these regions, specifically with regard to authorship integrity on anonymous online websites, in terms of reviewing and interpreting the data here. Anxiously awaiting their posting of the new policy.
    – TylerH
    Jul 26, 2023 at 21:13
  • 6
    Overall, I agree with most of the sentiment in this answer. But I want to comment on one specific detail: "How is a solicited admission any less legitimate than a freely given one?" I think the concern is that people were going to harangue users who were suspected of using AI, and it was going to create an uncomfortable climate for them, and SE was very concerned about losing new users who might ultimately go on to contribute usefully to the site in the future. I find that concern understandable. (By way of analogy, you can consider the ban on nagging question-askers to accept an answer.)
    – D.W.
    Jul 27, 2023 at 7:44
  • @D.W. yes, that crossed my mind, but if one has enough experience recognizing ChatGPT content, the risk of accidentally causing insult to someone who didn't actually use ChatGPT is lower. And really, I don't find "hey, out of curiosity, did you get this from ChatGPT" that insulting. Jul 27, 2023 at 8:04
  • 1
    @starball, that's all fine, but I am saying that the response in your answer is missing the point. You wrote: "But why? How is a solicited admission any less legitimate than a freely given one?" I've explained one possible answer "why", and I've explained that the "How is....?" is based on what appears to be a faulty premise. Perhaps you might consider editing your answer accordingly.
    – D.W.
    Jul 27, 2023 at 8:07
  • 2
    Additionally, people might use a VPN to post their chat-gpt answers. From SE's point of view, that would also be an increase in banned answers from those countries. Even IF LLM usage was flat across all countries, skewed answers aren't proof they're wrongly qualified as LLM-generated.
    – Gloweye
    Jul 27, 2023 at 8:32
  • 2
    One other factor to consider - LLMs are a great tool for non-native speakers of a language to express themselves more fluently. I know if I had to write a post in German I would use one to try to make the text more understandable. Using detectors or keying off of particular phrases would likely flag the post as AI generated even though I wrote the text. I would bet though that a site mod could tell the difference between that and a copy & paste answer. This is why I oppose a blanket ban on AI. If AI isn't banned, I can just explain I used it to translate but the concepts are my own.
    – ColleenV
    Jul 27, 2023 at 13:04
  • 4
    @ColleenV Then everybody using AI generation makes that claim. The major point of the AI-generated content ban on SO is that there are just not enough curation resources (regular users, subject mater experts, and moderators) to do the curation of AI-generated content based on a quality metric for each individual post. The amount of people we would need, particularly subject mater experts, is at least an order of magnitude higher than what we have or could get, which is already well below what we really needed pre-AI-genertion. Solve that problem first, then further issues have relevance.
    – Makyen
    Jul 27, 2023 at 17:17
  • 2
    What I'm saying is that we must exist in the realm of physically possible solutions which don't result in the, eventual, destruction of the sites. There are a variety of uses which generative AI could have that are relatively benign, but the vast majority of current and past actual use is people just generating an untested AI answer and pasting it into the answer box. Having some way to prevent that happening at scale is necessary, or the sites will fill up with garbage that's often very confusing to people seeking answers. That, eventually, results in the sites being valueless.
    – Makyen
    Jul 27, 2023 at 17:27
  • 2
    @Makyen The problem is that the SE curation system doesn't scale well. The solution to that problem is not an impossible-to-enforce blanket ban on a useful tool that will improve the quality of posts when used properly. My experience is that no heroic attempts to limp a broken system along go unpunished. Let it break and everyone will be far more motivated to find a solution that doesn't involved volunteers trying to vet tens of thousands flags without proper tooling.
    – ColleenV
    Jul 27, 2023 at 17:27
  • 2
    It is easy to fall into the trap of "If I don't step up and put in this heroic effort the world as we know it will end." I've been an engineer for enough decades to have seen many apocalypses fail to materialize. I understand and sympathize with the urge to fight to keep the status quo, but at some point you have to take a hard look at how many fingers you have stuck in that dam and ask yourself whether that's a sustainable solution. Either SE will come out of this stronger or something will rise from the ashes to take its place. Either way, it will be a better situation.
    – ColleenV
    Jul 27, 2023 at 17:36
  • 2
    @zcoop98 "Inherent to that piece of the policy is an assumption that GPT usage/ posting is flat across different language-preferences and regions. Did they check that assumption?" My experience has been that people who write policies like this show no externally visible evidence of even realizing that there is an assumption being made. I sure have been called bigoted more than enough times for pointing them out, though. Aug 1, 2023 at 7:32
31

This just demonstrates that SE/SO likes to be secretive and duplicitous.

Notably absent: A straight apology for putting moderators into this position in the first place by forcing them to be silent about a lie SE/SO management had no trouble telling users.

It just underlines how untrustworthy SE/SO management is.

Taking two months to tell the truth doesn't make you look better, not least because we've all known this lie existed for months.

It also makes this statement very hard to take at face value:

Since being published, this policy has been superseded, with the new policy available shortly

...but...

We normally do not release enforcement guidelines publicly. In recognition of the confusion caused by the differences between the public and private guidance in this case, we are now publishing the private text of this policy in full. However, we reserve the right to continue in the future to withhold specific enforcement guidelines from being published publicly in cases where doing so would make the jobs of moderators more challenging, and open up new avenues for abuse.

You've already said you reserve the right not to publish policies, so how do we users know your published policy will represent the actual policy your force on moderators (and us)?

You haven't grasped the basic issues:

  1. All policies need to be public.

  2. Different public and private policies is unworkable insanity. The public policy must at least reflect the overall logic of the private one.

  3. Private implementation details is not the same as having a completely different public policy from the private one.

By no stretch of the imagination could your public and private policies be said to follow the same logic. One is basically "never do this" and the other is "use good judgement when doing this".

SE/SO management put moderators into an impossible ethical position and - here's some news for you in SE/SO management - we users vote for moderators with a proven record of good ethical judgement.

We'd like SE/SO to show some of the same ethical good judgement. I don't see any signs they do here.

10
  • 1
    I think this post is just for posting the original policy, and the 'apology' will come later (after the strike maybe?). There has been some progress on public policy: "Stack Exchange, Inc. is apparently willing to commit to not having private binding guidance that would result in actions being taken on user accounts (such as mandating declining flags) without a public policy to back it up" (from discord ) but yeah I agree with your general sentiment.
    – bobeyt6
    Jul 27, 2023 at 0:01
  • 10
    At some level, you'll have to trust the moderators elected on the sites you frequent for some things, including letting you know if the company is imposing policy, rather than implementation details, in a way that conflicts with what the public knows. This post is mostly just honoring a specific demand by striking moderators that the previously private policy be released in public. Jul 27, 2023 at 0:10
  • @BryanKrauseisonstrike Do the moderators feel the delay in publishing this policy was acceptable ? I'll tone down that aspect my post if they do. Jul 27, 2023 at 0:34
  • 5
    The company had actually agreed to release the policy publicly a month ago. The reason it took a month between the agreement and the actual release was because of ongoing talks with the strike's elected representatives regarding an interim policy to supersede this one and to draft the top preamble indicating that they don't ordinarily release private policies publicly (which, by the way, had to be approved by the representatives before publication). Also, the main negotiator on SE's side had some personal issues that prevented him from dedicating much time to it. Jul 27, 2023 at 4:32
  • 1
    Private policies don't exist - by the terms of the moderator agreement. There doesn't need to be a preamble because no precedent is being set here. Jul 27, 2023 at 5:49
  • 6
    I don't agree with the position taken here. I can understand where the company was coming from, the concern that revealing the specific criteria mods could use (users have to self-disclose) could make it easier for users to game the system. I don't find that unreasonable at all. It did put the moderators in a difficult position, and it's not the option I would have picked, but I think it's a reasonable position that can be made by people in good faith who have the health of the sites in mind. I don't think it's fair to conclude that "SE/SO likes to be secretive and duplicitous".
    – D.W.
    Jul 27, 2023 at 7:40
  • 1
    @SonictheAnonymousHedgehog Thank you for that information. I'll give some thought to a possible edit to reflect that. Jul 27, 2023 at 9:20
  • 2
    @StephenG-OnStrike I mean, what does acceptable mean? Moderators are striking largely because of the content of the policy, but also to demand the full content of the private policy was made public, which it now has been. Having this policy secret, when it conflicted so much with what was public, was not acceptable to mods. On the other hand, I think we're mostly understanding of the various contributions to the pace of negotiations, including the large number of issues to agree on, and it's not fair to define "acceptable" as "do something different in the past": there's no time machine. Jul 27, 2023 at 14:45
  • 4
    When you start your post "This just demonstrates that SE/SO likes to be secretive and duplicitous", there's a bit of conflict there... Releasing this content is part of our demand that the company be less secretive on this particular issue. Are you targeting the original, now retracted with imminent replacement, policy? Or its release? The release, the subject of this post, is the opposite of secrecy. I'm in favor of discussing the content of the policy especially for people who had not previously seen it, but also keep in mind that this is a small, necessary step in the right direction. Jul 27, 2023 at 14:47
  • "At some level, you'll have to trust the moderators elected on the sites you frequent for some things, including letting you know if the company is imposing policy... in a way that conflicts with what the public knows." At this point, how can I trust the company not to discipline moderators who attempt that? "Or its release? The release, the subject of this post, is the opposite of secrecy." The issue is that the release is accompanied by an explicit claim to the right to do the same harm in the future. Aug 1, 2023 at 7:43
18

It's inaccurate to call this a "policy", because by the terms of the Moderator Agreement, the company will

Post previews for review of all new official policies in the Moderators Teams instance with the policy tag, marked with links to their public version once published, and maintain a listing of all official network-wide policies with links to them in the Help Center.

So private documents like this really must only be considered guidance, not actual policies. (Note that it was not posted for preview, nor was it posted publicly until now, after it has already been rescinded.)

However, since the company was apparently blatantly disregarding its part of the Moderator Agreement, most mods would not have felt like they could disregard the guidance. We were left in a pickle: a disastrous quasi-official private policy which could not be ignored but neither could we continue to moderate while following it. And so, the strike.

12

Thanks a lot for publishing this. It's much appreciated. When reading I wondered why you didn't take into account different possible thresholds. GPT detectors surely are tunable to some extent and this

This survey concluded that GPT detectors misclassify 32% (+/- 6%) of non-GPT posts on Stack Exchange sites as having been written by GPT.

only makes sense for a specific setting. Or was this included in the +/- 6%?

The country dependent suspension rate might not be a bias. Is the underlying assumption that otherwise all countries behave the same with regard to GPT? It may be good to explicitly state all assumptions.

The biggest mistake however seems to be to not ask for feedback before making the decision. Just imagine if you had presented the results and discussed them with the mods before making a decision. There might simply be a discussion about false positive rates now and the strike may never have happened. If anything I think this should be the take-home message: getting feedback is often enormously helpful.

3
  • 3
    meta.stackexchange.com/questions/389928/… In the data post they explain that this is based on the 50%/0.50 threshold setting. The +/- 6% is a confidence interval at that specific threshold based on the sample size used for the false positive test. Jul 26, 2023 at 17:29
  • 9
    @BryanKrauseisonstrike That's wild. Even people who use these detectors would never expect useful results at a 50% threshold. That would be like doing a statistical test and calling it significant because p < 0.5.
    – kaya3
    Jul 26, 2023 at 21:53
  • 15
    @kaya3-supportthestrike Indeed, and if staff looked at discussions among moderators and curators who did try out these tools, they'd have seen that anything below 90% was ignored or treated as evidence the content was human-generated, and only much higher scores (99.9 is a fairly common output from the tool) were treated as good evidence. No one asked, though. And this is in addition to the tools falling out of favor among the people who were doing most of the work towards identifying AI posts. Jul 26, 2023 at 22:01
10

I would like to preface this response by saying that more communication and transparency is almost invariably positive and that I appreciate the SE corporate team having decided to share some of the basis of their decision publicly. To the extent that the company is considering the impact of its actions, that is also a positive thing: sometimes, as with explicit policies against racism, transphobia, and so forth, bowing to the negative feedback of the community is not always a positive thing.

That said, I still have some serious concerns, ones that I imagine many people here share. For instance, what does the research actually mean?

  • The post claims that "Even more alarmingly, recent research shows that GPT detectors exhibit overwhelming bias against non-native English speakers, and misclassify their writing as authored by GPT at a rate of ~45-80%."

    If this is true, it is indeed alarming and would amply justify the policy against these tools. One can even imagine a reason that this might be the case: the underlying classifier might be trained with only two categories, "LLM" and "human," and non-American, non-British dialects might be very underrepresented in the "human" category, causing the model to be biased toward classifying them in the most populous class ("LLM").

    However, I can also imagine a reason that it would not be: by default, content generated by ChatGPT and the other most popular models in this space adheres to a strict notion of correct English as assumed from content primarily adhering to American and British normative dialects.

    This uncertainty naturally leads to the question: Is this based on external research or on analysis of SE posts? If the latter, does it use writing whose provenance is clear, such as that predating the use of these models? That information needs to be shared.

  • Building upon the previous point, could some of the bias be, quite simply, because people for whom English may not be their first language, or who use a dialect that may be socially disfavored in contexts like SE, are more likely to use tools like ChatGPT to edit their answers? In other words, could some or all of the disparities be because of true positives? If so, it might be understandable that one might not view this as a major problem, but framing it as an issue with the classifiers would be erroneous.

  • In such a situation, surely a more nuanced approach than simply prohibiting the use of such detectors would be in order. Why not allow the use of such tools in conjunction with some common sense to figure out whether a user is generating answers that (at this point) may well contain major errors disguised by the apparent fluency of the text, or simply recasting their own analysis?

  • Particularly if the research is from SE, we also should know the absolute numbers. With just relative fractions, we do not even know whether the differences are statistically significant, let alone the question of impact: how many legitimate users would actually be affected at all?

Also, why has the site not applied this same level of scrutiny to other issues of disparity?

  • Do the actions of human moderators have a disproportionate impact on users from certain regions? Anecdotally, I have seen a definite tendency for many moderators to ban disruptive users with non-standard English, while letting equally disruptive users with "good" English go unchecked. Why do LLMs get their own analysis and tailored policy, while as far as I know, human moderators did not?

  • Why is there not the same level of concern about bias in the LLMs themselves as there is about possible disparities in the tools that detect them? There is at least a decade of established research into the biases of language models, some of which persist in ChatGPT. For instance, notable gender biases and racial biases have been found in ChatGPT, despite presumed prompt engineering intended to tilt it toward more balanced answers. For instance, ChatGPT has gender biases and racial biases. Perhaps prompt changes would prevent it from creating such atrocious answers as "if white and male, then good scientist" perhaps not, but the underlying biases are still there and it will produce biased responses.

    Why does this appear to be of no importance to the company, if equity was the driving factor in its refusal to effectively moderate answers produced by the model?

Finally, even if SE has some legitimate concerns, why is the company now simply giving up and allowing GPT-generated answers to run rampant over the site without any oversight? Perhaps in a few years, machine learning will have produced models that can truly think and answer and question better than a human being, assuming that their goals are even compatible with human existence, and then everyone will prefer answers from them. At the moment, unfortunately, apart from editing and translation, for which dedicated tools continue to perform better, such models will primarily serve as a means for users to generate posts that may have major errors—and whose accuracy, crucially, they cannot assess—at a rate that can overwhelm human ability to moderate. We already have one example with a recent network-wide troll whose posts, despite not triggering any GPT detectors, were very clearly generated at least in part by language models, and contained many unsourced and often incorrect assertions ("Italian tomatoes in general are tastier because of volcanic ash and because Italian farmers are just more experienced and passionate than those in other countries"), along with answers that got confused about who was answering ("According to my theory of relativity...") or just outright hallucinated (naming non-existent books for story identification questions).

Instead of throwing up their hands, why is the SE team not coming up with better ways to balance the need to treat users fairly with the need to prevent automated content from potentially compromising the quality of the network? For instance, they could take some of the money that they are throwing at answer generation and invest it in more accurate GPT detection. If and when such tools become reliably better than any human, then they can remove any restrictions. Don't be surprised if real AIs are not as happy as human users to work for free, though....

7
  • 1
    The second link is to a Google search. Can you recommend one or more in particular? Jul 29, 2023 at 19:53
  • 4
    “Perhaps in a few years, machine learning will have produced models that can truly think” — I make them. You can't even imagine how far they are from what you think they are... They're as helpful as a hammer in skillful hands, and about as smart, too. Aug 2, 2023 at 0:42
  • @kkminactive-supportstrike - It does not sound like whatever you make is what I am talking about, then. I would not say that a hammer can truly think.
    – Adamant
    Aug 2, 2023 at 2:01
  • 1
    @Adamant Correct. It can't. Aug 3, 2023 at 15:52
  • I am afraid I do not see the relevance of your comment, then. If I wrote "Perhaps we will have X that does Y in the future," what does that have to do with A that does B in the present?
    – Adamant
    Aug 3, 2023 at 15:55
  • 3
    “… because people for whom English may not be their first language, or who use a dialect that may be socially disfavored in contexts like SE, are more likely to use tools like ChatGPT to edit their answers?” - Users have made this claim, the problem is that tools like Grammarly for example, when ram through the same detectors were not being flagged. Before I even checked certain content, my spider senses were going off, due to the fact the English grammar was absolutely perfect. Almost to that point, as if it was generated in its entirety, by a tool to generate perfectly grammatical content.
    – Ramhound
    Aug 4, 2023 at 10:08
  • @Ramhound: That is my sense too. I think it is part of the lies told by the ChatGPT plagiarisers (used as an excuse). It is a strange coincidence that, suddenly after 12 years of not caring at all about the quality of their writing, they suddenly do, which happens to have been at the same time as ChatGPT was introduced. Aug 6, 2023 at 15:27
8

My only reaction is surprise. This is a very aggressive policy and it's very aggressively worded too, so I can't imagine what provoked it.

It was also bound to be poorly-received: the elected mods and high-rep users are generally skeptical about, cautiously and critically, and thus often negatively, evaluate policy and design changes, and the individual sites' temporary bans on GPT posts indicated a strong community consensus.

It should have been quite obvious that there would be a massive backlash that included, at the minimum, a flat refusal by the moderators to obey this policy, and I'm surprised that SO chose to shoot their shot anyways.

5
  • 15
    "the elected mods and high-rep users are generally skeptical about policy and design changes", this sounds like you are suggesting there's a bias. There were features introduced recently whic were well-received, although they are becoming a rare breed. While I cannot disagree that some folks may downvote anything that has a staff badge on it, elected mods and majority of high-rep users are judging every feature based on its own merits. If we see negative feedback more often, it's because company is heading towards the wrong directions on different fronts.
    – M--
    Jul 28, 2023 at 3:18
  • Funny, I didn't find the policy that aggressively worded. Maybe a matter of cultural background. Aug 12, 2023 at 5:28
  • 1
    @Peter-ReinstateMonica ""none of the GPT detectors..." "egregious false detection..." "alarming native language bias"... that's a lot of absence of sugar-coating. The sections are similarly assertive with strong, controversial claims: "... do not work as advertised and produce unreliable results," and "creating bias against non-native English speakers." There are few/zero "please" statements, and no admission of uncertainty on the part of the policy's author. "The scope of admissible evidence for establishing GPT usage is, in the current state of things, very thin" is clear, strong criticism. Dec 3, 2023 at 7:58
  • @M-- that was not my intent; I have edited the post. Dec 3, 2023 at 8:20
  • @user1394273 it changed but... "and thus often negatively" part conveys the same (similar, to be fair) message. I appreciate the effort and I see what you mean (trying to say). The onus is not on you to explain the complex relationship between community and SE in a few words. Cheers.
    – M--
    Dec 7, 2023 at 20:56
8

Considering that it took 7 weeks of striking and negotiations to get the private guideline published, I want to jot this down that merely based on what happened with these policies, establishing a strict and an actionable guideline to make sure that moderator's agreement will not be breached by either party (read SE Inc.) which also anticipates measurable and proportionate "retribution" in case of violations is absolutely warranted.

Hence, this transpired through negotiations:

Issue: There is no recourse if Stack Exchange, Inc. breaks the Moderator Agreement.
Result: Stack Exchange, Inc. will commit to issuing an apology to Meta.SE and retracting any actions taken that were found to have broken the Agreement, as determined by a consensus of moderators. Currently, that consensus would require at least 20% of Stack Exchange network moderators to vote on whether or not the Agreement was violated, with at least 90% of respondents agreeing that it was violated. Additional tooling for this process will be developed, and these numbers aren't quite finalized.

2
  • "Stack Exchange, Inc. will commit to issuing an apology to Meta.SE and retracting any actions taken that were found to have broken the Agreement, as determined by a consensus of moderators. Currently, that consensus would require at least 20% of Stack Exchange network moderators to vote on whether or not the Agreement was violated, with at least 90% of respondents agreeing that it was violated. Additional tooling for this process will be developed, and these numbers aren't quite finalized." From discord
    – bobeyt6
    Jul 27, 2023 at 0:45
  • 2
    @bobeyt6isstricken I am aware of that. There's an ongoing discussion about those percentages on the discord channel. Thanks for sharing though. p.s. I will add that to my answer as a reference.
    – M--
    Jul 27, 2023 at 1:58
7

These two excerpts are particularly quote-worthy:

The only admissible evidence for GPT usage is self-admission; however, you should not solicit self-admission from users.

And

The amount of content deleted for GPT usage is truly astounding across the network.

The whole text of this policy does not indicate any particular knowledge of unfairly suspended users.

Also, it shows a rather poor management of the following conflict of interests: on the one hand, SEI wants to have as much activity on the stacks as possible, because this looks good to advertisers; on the other hand, the content produced by the activity should be of good quality if the stream of visitors is to be maintained. Apparently, looking good now to advertisers was deemed more important.

This policy made it look like it was thought a necessary evil to leave an "astounding amount" of poor content on the stacks, just so the number of users and/or posts wouldn't be negatively impacted.

2
  • 3
    I have voiced similar and other related observations in the the Ubuntu community's meta: meta.askubuntu.com/a/20323
    – Levente
    Aug 11, 2023 at 10:23
  • @Levente Thanks, your write-up over there is more complete than this one.
    – Conrado
    Aug 11, 2023 at 18:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .