138

A new Network policy regarding AI Generated content was recently announced by Stack Exchange.

As discussion is not currently permitted on such posts, please discuss the policy in the answers here so that we have a single place for discussion and feedback regarding the policy.

(also I asked Journeyman Geek and he said having a discussion post sounded good)

3

14 Answers 14

118

The guidance post fails to answer its own question.

The entirety of the question from the guidance Q&A is:

What is the network policy regarding AI Generated content?

Earlier this week, Stack Exchange released guidance to moderators on how to moderate AI Generated content. What does this guidance include?

Reading this, I would expect, you know, guidance for curating sites and moderating content. Unfortunately, Philippe fails to provide any guidance at all for non-moderators, you know, the users responsible for the overwhelming majority of moderation across the network.

Instead, we got a corporate grandstand about how moderators need to be more careful when suspending users for AI-related issues, something that should have been an email, not a dramatic network policy bomb.

And I, a regular user, am left with no further guidance about curating these posts, and the mods are left to deal with my unguided flags.

3
  • 18
    i mean, the policy post pretty effectively makes it clear that the the unguided flags should just be declined due to no evidence. despite what we can see with our own eyes. This makes raising such flags pretty much pointless.
    – Kevin B
    May 31, 2023 at 3:01
  • 1
    yeah I made the same observation here and here. not pleasing to look at. May 31, 2023 at 4:30
  • 5
    Obviously it's a difficult topic. Last year the network banned ChatGPT without having a good answer on how to actually enforce this ban. Of course there will be collateral damage. Now the discussion is how much damage one is willing to accept. This cannot be solved easily or simply by words. I would argue that we need more insight into the actual numbers. How many posts have been deleted? How many users complained about it? Can we estimate the false positive rate somehow (even that is probably quite difficult)? May 31, 2023 at 7:07
98

Why did the ban announcement on MSE not provide any specific data (vs. "trust us")?

This was somewhat mentioned (by me) here, but I'm creating a dedicated post about it

We recently performed a set of analyses on the current approach to AI-generated content moderation. The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high.

Could you post your analysis itself, and not just your conclusion? Bonus points if you can post the data you analyzed as well.

... we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts

That is... possibly true. But as you (should) know, a suspicion isn't a reason for a ban. At least, you explicitly state "We've reminded moderators that suspensions ... are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics.". Stating effectively "we did some research, trust us please" is effectively an unverified heuristic. Sorry.

A much better statement would include proof of that statement. Or data to support it. Or a link to a peer-reviewed paper that supports it. Not just "we also suspect". That isn't a high enough bar here.

Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.

Please publish this evidence. Or, if you can't (i.e., because it contains PII), then please state explicitly why you can't publish it. Side note: a user (who I'm not naming unless they do here) posted a detailed analysis of SE traffic post-ChatGPT. That is worth reading if you haven't, yet.

We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.

That's been known. Since they were released, it is not a secret that they have a somewhat high FP rate. This is why that was used as one data point that a user may have (ab)used ChatGPT, and therefore it was worth looking for additional evidence. While I haven't asked the entire mod team, I'm fairly confident that they weren't randomly suspending users based on that with no further evidence.

Could you instead provide data on the number of wrongfully-suspended users? Bonus: how are you determining if a user was wrongly suspended? Some more context would benefit that claim.

You make a large number of claims, yet provide ZERO sources/data points to support them. Please be more specific.

2
  • 6
    Just as a small remark: Even if it would be difficult to determine if a user was wrongfully suspended he/she still might been exactly that. The conclusion should not be that it doesn't happen if it cannot be proven. For example I could imagine that users may in general not protest in all cases where they would be eligible for a protest. If it's difficult to determine if a post was AI generated it's probably also difficult to determine if a post wasn't. For example, how would I convince anyone beyond doubt that this comment has not been generated by an AI? May 31, 2023 at 7:10
  • 4
    @Trilarion then we shouldn't do such a heavy handed approach. Guilty until proven innocent is bound to piss off many more people and with reason.
    – Braiam
    May 31, 2023 at 19:32
76

The new policy can be summarized as "You are allowed to moderate AI generated content, but you are not allowed to use anything to detect such content including your own brain."


AI generated content cannot be moderated using the regular approach used for low quality content because it does not fall under low quality categorization:

Temporary policy: ChatGPT is banned

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting.

If moderators are not allowed to moderate AI generated content based on it being AI generated, the only thing we have at our disposal is leaving them to the community to downvote and delete vote, which is not a feasible approach as the community is already having hard time moderating even blatantly obvious poor content due to the sheer number of such posts.


If all means of AI detection are excluded that means AI generated content will not be moderated and will cause permanent damage to the sites and their quality. This deterioration will also happen rather fast as it will be extremely easy to flood the sites with AI content.

I can’t fathom why such a policy was made, unless the company actually wants to allow AI generated content for some inexplicable reason; but, saying that would directly go against what the vast majority of the community wishes. In that light, preventing moderation can look like the safest approach as it allows them to to have the cake and eat it at the same time.

Not only is this extremely bad policy, but it is also not backed by any verifiable data that moderators were suspending people en masse for valid content (mistakes are always possible, but they are also easily correctable as they have always been).

Please, allow moderators to continue moderating AI generated content using all detection means necessary, and issuing suspensions as they were doing before this policy. This is the only way we can preserve the quality of the sites and future participation of experts willing to invest their time, to share their knowledge.

13
  • 3
    I wonder if the mods themselves have somehow gauged the effectiveness of the moderation of the AI content so far? Or are we all flying blind to some degree? May 31, 2023 at 16:11
  • 6
    @Trilarion Of course we have. We care about our sites and are constantly self-evaluating our actions and the actions of our fellow mods. We don't earn any sort of points for the most accounts suspended or most posts deleted or anything like that. All indications to moderators are that the vast majority of actions that mods take against AI-generated content have been correct to community standards, and we haven't been provided with a single example that suggests otherwise. May 31, 2023 at 16:53
  • 2
    @BryanKrause I see. So basically it seems that the perceived risk of misdetection differs greatly: the company thinks it's really big, the mods think it's really small. Both groups have somehow checked their data, but none is presenting any of that to the public. For us users it comes down to whom we trust more. I think it's clear how this competition of trust ends just wanted to make it as clear as possible. For the mods the question is what to do if the company cannot be convinced that AI generated content can be detected without much risk? Am I getting it approximately right? May 31, 2023 at 17:01
  • 13
    @Trilarion I think the biggest disconnect is in the assessment of different risks. The community consensus, shared by moderators, is that AI-generated content poses substantial risks to the quality of content here, especially due to the ease of producing such content relative to the ease of curating it. The company has alluded to internal evidence that overmoderation of this content is harming the site by discouraging participation. I have not seen any evidence that this is the case that does not have far simpler explanations. May 31, 2023 at 17:06
  • @BryanKrause Ok so let's hope that the disconnect can be mended. Because if not and that is not unlikely, something has to give. The irony is that I agree that ChatGPT is currently not healthy for SO but at the same time I'm using it almost daily for my work because it's useful to me. But then I know what I'm doing and can compensate for the shortcomings of ChatGPT. May 31, 2023 at 18:15
  • 5
    @Trilarion I think there are even some promising integrations of ChatGPT with SE/SO; Philippe posted recently a couple possible ways to go, which I think are all workable with some refinement. But intermingling actual content generated by AI with human-generated content is something I believe is strongly damaging, and makes it harder to use the sort of workflow I expect to develop, where people like you may use ChatGPT for some things but still need to come to SO when it breaks or they get stuck. I think ChatGPT is also not as useful for beginners who don't know how to spot its mistakes. May 31, 2023 at 18:18
  • Quora and other sites are allowing AI content. Quora even has a great big chatGPT box sitting above the human answers to a question
    – Richard
    Jun 2, 2023 at 7:13
  • 9
    @Richard Stack Exchange is not other sites. If we allow AI content (or should I say answers) which are commonly incorrect, we will be on fast track to completely destroy the sites in SE network. See: Is there a list of ChatGPT or other AI-related discussions and policies for our sites? Jun 2, 2023 at 7:34
  • 3
    All AI is doing here is making the scaling problems with the SE design impossible to ignore. The problem is that not enough people who can upvote recognize the quality/correctness issues and there aren't enough down-voters to make an impact. This happens with non-AI posts as well. Any tool that allows users to quickly post reasonable looking but wrong/plagiarized answers en mass would put the same stress on the system. They've needed to fix these issues for years, but there's no money in accuracy, only engagement.
    – ColleenV
    Jun 6, 2023 at 18:13
  • @ColleenV Agreed. However, even if those issues with wrong/poor posts would be fixed, AI would still be too huge to handle in any other way that does not include mods, just regular users. Jun 6, 2023 at 19:01
  • 2
    Well, we keep moving further and further away from volunteer mods being exception handlers. For too long SE has exploited the emotional investment many of the mods have in their communities to have them do stupid shit like handle thousands of AI post flags instead of spending company resources to try to resolve the underlying design issues that make brute force manual solutions necessary.
    – ColleenV
    Jun 6, 2023 at 21:51
  • 3
    @ColleenV Speaking as a normal user who's been here for years but mostly upvotes, my problems have already been asked, the questions that need answers are too obscure for me or low quality, and the result is my reputation changes on the scale of 20/month. I'm not going to be any assistance down-voting. The reputation hit already makes it not worth it for all but the most egregious of bad answers.
    – ojchase
    Jun 8, 2023 at 15:48
  • 1
    @ojchase I hear you. If it was an easy problem to solve, it would have been solved years ago. Part of the problem I think is that no-one wants to do more than incremental changes. I honestly think there are a lot of things than need to be completely redesigned, but there is too much invested in the status quo for that to happen. Stack Exchange's successor is probably already being worked on by someone. (I know there are a few contenders; I don't know which one will succeed. If I did I wouldn't say ;))
    – ColleenV
    Jun 8, 2023 at 17:04
50

So, so, many issues here that it's difficult to know where to start. And I'm not sure I could cover all my concerns in one answer in any event.

I've already addressed some points on the Mod Team and direct with Cesar in the TL (kudos to Cesar for showing up and doing his best to respond to a veritable deluge of angry mods BTW, you're a gent and a scholar) so I'm going to take the opportunity here to look at some things directly from the announcement posts.

We've reminded moderators that suspensions (and typically mod messages as well) are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics.

I strongly feel this mischaracterizes both what the mods were doing in handling GPT generated content by implying that we were going round suspending at the drop of a hat based on shallow guesses or blindly taking detectors as gospel and simultaneously implying that mods were somehow moderating this content without SE's awareness of what was being done and how it was being done. Neither of these are true, so characterizing this as trigger-happy mods who got carried away and are now being reined in feels very much like we're being thrown under the bus, and whether that's intentional or not (I'd lean not, the CM team has pretty much always come across as decent folks with a tough job who have our backs and I'd like to think that's true) doesn't really matter to the poor sods getting splattered by the bus, it hurts just the same.

But putting the potential for hurt feelings aside it also runs straight into a consistency problem in the very next paragraph:

As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.

Wait, so we can only act on "verifiable" malfeasance, AI detectors are unreliable, can't be trusted, heck, are even racist (apparently), and our own judgement and analysis can't be trusted either but for "poor quality" content we can use both of those things when moderating and they therefore count as evidence of "verifiable malfeasance", an AI-detector tells you nothing directly about the quality of the content, we all know that the current crop of GenAIs are liable to produce poor quality answers but the fact that it was produced by an AI is not proof in of itself that the quality is poor. So the "false positive" rate for a GenAI detector being used to determine if it's AI-generated is too high to justify acting on but surely the false positive rate for determining if it's too low quality for SE purposes (something it's not even designed to do, and doesn't even try to do) is going to be off the freaking charts, but we are to be encouraged to use them for that purpose instead of their actual purpose.

That's as clear as mud :(

Fortunately for me GPT content hasn't been too big a problem on TWP thus far - and when our users asked what the site's stance was we took the position that GPT-sourced content was already not permitted as it was covered by existing policies.

Un-referenced content from there was covered by plagiarism policies, any word-salad was low quality, as was just posting a referenced GPT answer with no additional added value from the poster, as per the Help Center:

Do not copy the complete text of sources; instead, use their words and ideas to support your own. In particular, answers comprised entirely of a quote (sourced or not) will often be deleted since they do not contain any original content.

Nothing in the current policy changes that as far as I'm concerned, it's just as unacceptable as it was before, it's just arbitrarily harder to actually moderate/curate for the mods and the community. Thanks for that. I sure do love it when a job I volunteer for gets made harder for no good reason!

3
  • 6
    I say that they opened themselves to a Catch 22 scenario that can get abused on both sides. It could be used to circumvent the problem (after all you are trying to remove low quality content, not AI generated content. Who cares about the source if the content is good?) and yet is blurry enough to allow for demonizing a new scapegoat unlucky mod should that be the case
    – SPArcheon
    May 31, 2023 at 13:18
  • 1
    I think the company has made kind of clear what they want: significantly less suspensions or deletions. How this is achieved exactly they leave partly in the hands of the mods. May 31, 2023 at 16:30
  • 1
    "simultaneously implying that mods were somehow moderating this content without SE's awareness of what was being done and how it was being done" well, how are posts being evaluated? If SE doesn't know how moderators do it, at least the community at large should be able to be privy of such process of evaluation.
    – Braiam
    May 31, 2023 at 19:35
49

The new Code of Conduct, active as of today, May 31, 2023, forbids inauthentic usage, including:

Plagiarizing or copying content from websites, books, or other online and offline tools without proper attribution in a manner that violates our referencing standards.

In my view, this includes GenAI posts, which are generally copied and posted without any attribution to the software tool that generated it. However, the new policy makes it impossible for moderators to moderate to this standard.

10
  • If the content was from another website, one would simply link to that site or even argue with fair use if the snippet was only small. Just for clarity: why would linking to ChatGPT not be enough (if ChatGPT wasn't banned)? Or is the issue simply that people cannot circumvent the ban without not giving attribution? May 31, 2023 at 17:39
  • 12
    @Trilarion There are multiple problems with GenAI content. One of them, the one addressed by this part of the CoC, is lack of attribution. The TOS of some of these programs already requires attribution when the generated content is used, but even without that, we require attribution whenever your writing is not your own. Whether acknowledging that ChatGPT generated content is sufficient or not remains somewhat unclear, especially because GenAI is currently incapable of attributing its own content. That's a problem if it effectively regurgitates training data in some contexts... May 31, 2023 at 18:24
  • 10
    ...It's also a problem with reliability of information, which is more important on some sites on the network that have specific requirements for referencing answers, including Medical Sciences, one of the sites I moderate. That issue seems less problematic under the current guidance, because answers that fail to reference sources are not allowed regardless of who wrote them. The specific issue with the new policy, though, is that even if a post is apparently AI-generated and does not give proper attribution, we are forbidden from acting on it. May 31, 2023 at 18:26
  • 9
    As I explained to one user: if you take some chat bot's output and paraphrase every sentence (as this user claimed to be doing) the result is still not your own work; presenting it as your own work is still plagiarism. That user apologized in their mod message reply. We hardly ever get genuine apologies in mod message replies.
    – rob
    May 31, 2023 at 22:32
  • 1
    @rob I've also rarely contacted users about AI content and yet saw multiple apologies. May 31, 2023 at 22:34
  • 14
    ChatGPT is quite happy to cite sources when it needs to; unfortunately the sources it cites usually either don't exist, or don't say what it claims they say. But checking that wastes even more time.
    – kaya3
    Jun 1, 2023 at 0:07
  • I would go far beyond saying " which are generally copied from tools without any attribution". Unless you can somehow prove that the attribution is correct, there is no way for you to know if any "attribution" presented by the generator is real or just made up based on the format of authentic attributions in the dataset.
    – SPArcheon
    Jun 1, 2023 at 7:38
  • I asked a chat tool (Baize) about Covid statistics in England, it claimed that "In June 2021, England had the highest number of deaths per capita of any country in the world, with over 15,000 deaths.", mentioning the UK gov as the data source. Does not match the number given here
    – SPArcheon
    Jun 1, 2023 at 7:39
  • 3
    @SPArcheon In this post I'm talking entirely about not attributing the content to a GenAI, not about attribution that GenAI hallucinates. This is of course not the only problem with AI answers, but it is a simple one that I do not currently see a mechanism by which we moderators can enforce a relatively basic aspect of the CoC. Jun 1, 2023 at 11:55
  • Sorry, I misinterpreted that as a tangential problem - AI generated content often does not mention the source (-> the data the model used for training) and/or hallucinates made up sources (claiming for example that some data comes from a source that does not exist)
    – SPArcheon
    Jun 1, 2023 at 12:05
49
+50

TL;DR, from Google Labs’s generative AI: The new policy was announced without moderator or community input, lacks data to support its claims, and goes against the wishes of the community. Moderators are no longer able to use their judgment to remove harmful content, and the policy is likely to increase the spread of misinformation.


The new network policy regarding AI generated content is problematic. It was rolled out in one of the worst possible ways, goes against the wishes of the community, and is not supported by any available data.

On May 29, 2023, the Community Management team posted an announcement on the private “Stack Moderators” team. They also pushed out a notification to all moderators directing them to read this announcement. To the best of my knowledge, this was the first time that any of the community elected moderators were made aware that anyone at the company had any concerns about how algorithmically-generated content was being moderated or that any research into detecting and handling such content was ongoing.

Simply by making such a policy announcement, the company did not follow the policy on making policies. There was no substantial preview or review period - the policy was declared to be created and moderators were only given a short “heads up” period prior to it being posted publicly. Questions and concerns raised were not adequately answered or addressed prior to the policy being made public.

In addition, the day that the announcement was posted for moderator “heads up” was a holiday in the United States, Canada, the UK, and other European countries. Given that it was announced on a holiday and was then posted a day later to the public, it is highly likely that not all moderators were able to see and react to the policy edict before it was made public. Some moderators may still be unaware.

The policy, as presented to moderators, had two main components:

  1. Stop using detectors when investigating reports of algorithmically-generated content.
  2. Stop suspending users only for posting algorithmically-generated content.

To support the policy, two claims were made:

  1. Detectors do not work as advertised and produce unreliable results.
  2. Detectors are biased against non-native English speakers.

Neither claim was supported by data made available to moderators or to the broader community when the policy was made public.

With respect to the first claim that detectors were unreliable, no information was provided about when the detectors were tested (i.e., what version of the detectors were tested), what communities the test data came from, or how many posts were sampled. In addition, there were no examples given for posts that appeared to be miscategorized.

Data to support the second claim that detectors are biased against non-native English speakers, would require some level of PII, so couldn’t be shared (even with moderators), but I would have expected aggregate data, such as the number of users from each country, the participation rate from each country, and the number of suspensions over time. It’s unclear what the baseline suspension and account deletion/destruction rates for various countries are and how suspensions related to algorithmically-generated content compare to other moderation activities.

The claims do raise questions. However, in the absence of clearly specified methodologies and making data available where possible, it is inappropriate to be dictating policies that go against the desires of the community. Many communities on the network have developed policies to address posting algorithmically-generated content, some with clarifications regarding how to attribute generated content and others with a prohibition on the use of generated content. On Software Engineering and Project Management, where I’m an elected moderator, these policies have overwhelmingly positive community support. Nevertheless, in cases where the community has zero tolerance for algorithmically-generated content, the ability to take the appropriate action that the community has decided to be appropriate has been eliminated by the new policies.

I believe that not taking strong, decisive action against algorithmically-generated content is the wrong thing to do. Visitors to the Stack Exchange network come here expecting high-quality answers that draw on human knowledge and experience, with valid citations and references where appropriate. When someone plagiarizes the output of a generator, that person is misrepresenting their own knowledge and experience. Considering that these generators do not have the concept of facts and truth, and only a subject matter expert (often not the asker) can verify the claims, it would be better to remove potentially harmful content. Since the data on Stack Exchange is public, I would expect that future algorithms are trained on the data. This compounds the harm by feeding incorrect information into search engines and future algorithms which would spread the harmful information. The harm of an erroneous suspension - which could be reduced or eliminated by a moderator - is far less serious than the potential harm of supporting the proliferation of incorrect information and training future models on algorithmic outputs.

Moderators are not prevented from acting, but we’re extremely hampered. We are no longer allowed to use our personal judgment or the only tools in the toolbox to curb content that the communities - the people who elected us and have offered their inputs into community policies - do not want to see. Although we can deal with certain classes of content, our choices are to either let some algorithmically-generated content be posted on the network or to go against this new policy.

I will also say that I have, to date, suspended 7 accounts for posting algorithmically-generated content. Of these accounts, 6 of them were newly-created accounts that appear to have been created just to post such content. None of these people reached out to challenge their suspension. I did not hear directly from the Community Management team that any of these suspensions were escalated to them or deemed inappropriate. It is unclear what may have happened if they were allowed to post convincing content that allowed them to increase their reputation and gain access to other functionality.

I do think that we should have discussions around both claims and reach a baseline policy for the network. But we should have the underlying data (to the extent possible) and have discussions on the research and validity of the claims before making policies. And then, when policies are established, they must follow the agreements between the company and the elected moderators to go through working groups, moderator review, and finally appropriate public discussion and announcement.

6
  • 19
    Wait.... WHAT??????? " the day that the announcement was posted for moderator “head’s up” was a holiday in the United States". Wasn't that exactly one of the "coincidences" that happened to Monica too???
    – SPArcheon
    May 31, 2023 at 15:42
  • 25
    @SPArcheon For Monica it was specifically a religious holiday that, based on her religious views, specifically forbid her doing the sort of work that participating here entails. So, that was really far worse, though also from what I know was done out of ignorance rather than malice. May 31, 2023 at 15:46
  • 3
    @ThomasOwens You might consider altering the double-negative in "I believe that not taking strong, decisive action against algorithmically-generated content is the wrong thing to do" to instead read that you believe taking strong action is the right thing to do, though it does change the meaning a bit and if you think it's more important to highlight the wrongness of this policy rather than the rightness of the policy you prefer then it's certainly worth keeping the way it is. May 31, 2023 at 15:48
  • 4
    @BryanKrause once is ignorance, twice is suspicious, trice is malice. Hopefully we don't get to act three.
    – SPArcheon
    May 31, 2023 at 15:48
  • I fully understand the demand of having public data on the risks of misdetecting AI generated content. A risk that seems motivated well since AI is trained to mimick human generated content. Detecting AI generated content seems to be a non-trivial task. However, I wonder what mods will do if the data is not forthcoming or not convincing enough? In the end it always comes down to deciding if under the current policies it's worth moderating or not. And the company should have some freedom in setting these policies, after all it's their site. May 31, 2023 at 16:55
  • 3
    @Trilarion: It's the company's site, but I'm now strongly considering looking for a better site that isn't run by people hostile to the community that uses it. I disliked a lot of their previous policies and actions in recent years (especially since Monicagate), but until now it seemed like the site could still be used by the community to build and curate a library of useful Q&As. Directly hampering those efforts is a new low. Jun 2, 2023 at 18:04
44

My thoughts, as someone who has flagged more than 1,200 (subsequently) deleted answers (and some comments, and even some questions) that I believe were GPT:

Note: I've also posted a separate answer on this question on my thoughts on what I feel a sensible AI policy on Stack Exchange would be. I feel it's a separate enough topic under this question that it needs to stand-alone.

  • I've seen high quality answers deleted, low quality answers deleted, and those that I don't have the subject matter expertise to judge on my own.

  • I've seen false positives from AI detectors (but many more false negatives), but it's rare that AI detectors would be the only source of confirmation that a question was generated by AI.

  • I've had flags declined, as the Moderator who reviewed the flag felt that there wasn't enough evidence for deletion (let alone suspension).

  • I have been, and continue to be, reluctant to post the number of different techniques that I personally use to detect AI, but the Mod team seems to have agreed with them in most cases. And yes, there are indications in the "writing style", obviously -- What else would there be? But analysis of writing style is also banned under the new policy.

  • Code blocks in an answer generate a much higher percentage of false positives. For that reason, we've recommended that anyone using an AI detector on a Stack Overflow answer first remove any code-blocks. I wonder if the Staff removed code-blocks when doing their data analysis.

  • I do not believe that attempting to moderate AI answers is unfair or biased against non-English speakers. In my recent review (just before the policy was posted), I scanned 300 answers and automatically ran them through a detector. The false positives that I saw were from, as far as I could tell, native English speakers, or at least those with a high command of the language.

  • I do not believe that the answers I flagged were simply using ChatGPT or AI for grammar assistance. I personally believe that this is a perfectly acceptable use of ChatGPT here, and I've tested ChatGPT grammar correction on some difficult-to-read posts here. The results do not appear to be GPT "generated" answers like we normally find, nor do they have any higher percentage of false positives as a result of ChatGPT grammar correction.

  • I'm obviously biased, as I've been so active in the flagging of these answers, but I believe we have had an extremely high rate of true positives in the community's curation of AI-generated answers. If the SE staff disagrees, I would encourage, nay challenge, them to undelete the thousands of posts that they feel have been wrongfully deleted. Hey, it's the least you can do after you feel that the posters of those answers have been wrongly suspended.

    After undeleting the answers, take a random sample of them and explain here on Meta why the Staff feels they are not AI.

2
  • 2
    Reading between the lines, I think the alleged bias against certain languages boils down to, "if you are unable to phrase yourself in idiomatic English and used ChatGPT to fix that, you could get suspended." But the company has never explained this in any detail, so - like with many of their allegations surrounding this issue - we are left to speculate wildly.
    – tripleee
    Jun 7, 2023 at 3:49
  • Of course, the simple workaround is to use some other machine translation system instead; and we still require the OP to be able to understand and respond to comments and answers in English. The whole issue seems rather niche, anyway, if my speculation is correct.
    – tripleee
    Jun 7, 2023 at 3:51
28

I apologize for any mistakes due to lacking the full picture that only moderators have since apparently (at least based on what I read from multiple sources) the company has shown two "different but not completelly unlike" pictures to the general public and the few who have access to the moderator private chatrooms. And if the general public already thinks this is horrible, I can only imagine what the mods got to see instead to be planning for a full fledged strike.

Alas... this is kinda pathetic. Basically, it shows that the company lacks a basic understanding of what the difference is between "computer assisted decision making" and "computer automated decision making".

Let me show you an example.


Art. 22 GDPR Automated individual decision-making, including profiling

The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.


Obviously, this is not the same as what we are discussing here, but it should make one thing clear: Automated decision-making has been identified as a risk for quite a while now.

...
......

But... I beg you this question. Who sane in their mind would equate the use of a tool to SUPPORT manual, user made moderators decisions with automated machine handled actions based on heuristic detection?

This seems very hypocritical and feels like part of an agenda to once again take the role of "heroes of light" in media (as a thing like the "welcome wagon" felt in the past - not motivated by care but by marketing advisory) or tries to follow the recent trends (AI generation being $$$ in the eyes of stakeholders?).

I will be very blunt and I apologize if this seems harsh. But I find it quite ironic that in all the years I have spent here the company never posted a policy against good old Smokey. I mean, it is heuristic detection of bad content so that actual humans can avoid looking for themselves and have a pre-processed list of things to manage. It is the same exact things and should be banned too.
Please, holy overlords, suspend all the corrupted users that have usage privileges on Smokey and ban its evil creator mastermind until the thermal death of the universe at the very least.

"And another thing...(cit)", another funny question that came to mind... how does our company actually plan to implement this shiny edict? How do you plan to know if I used a tool to find this poorly generated useless content or if I stumbled upon it by chance?
Because either the content is bad and should be removed either way (and being generated is only the reason it ended up bad) so you can't oppose users for asking to remove it or if it is brilliant, useful, and legally abiding (by mentioning sources for example) no one would have to bother with trying to remove it even without this waste of time promoting nonsensical policies.

2
  • 1
    For the record, and for context, Smoke Detector is open source, and thus it does not have an "evil creator mastermind" in the sense you seem to be alluding to. The users who were responsible for the original architecture are by and large no longer very active in the Charcoal community. For the terminally curious, the Github commit history at github.com/Charcoal-SE/SmokeDetector has the full details.
    – tripleee
    May 31, 2023 at 15:13
  • 13
    @tripleee that was meant to be satire. If suddenly the company bans usage of AI generated content detector as a support tool to facilitate moderation because they are unreliable because they are just heuristic/probabilistic tools, then they have to ban Smokey too since on principle it works in the same exact way. Basically I am arguing that if they were fine with users helping themself with Smokey for years, it feels quite weird to ban usage of similar tools to help find low quality ai generated content.
    – SPArcheon
    May 31, 2023 at 15:24
25

Please see my other answer here for some additional thoughts on AI moderation in general.

As I mentioned in my other answer, I have been extremely active in the community moderation of AI answers, with over 1,200 flagged and subsequently deleted.

That said, I don't feel that either the old policy (for most of the sites) nor the new policy is ideal. We've gone from one extreme to the other:

  • On Stack Overflow and many other sites, GPT/AI content was entirely banned, whether it was good content or bad. There are multiple reasons for this, of course, which are listed in the policies. That said, we are throwing out good answers as well, even those from users who used AI tools to supplement their own expertise.

    The problem isn't those "responsible" users, it's the vast majority of others.

    To state the obvious, SE works on the basis of reputation. And users posting AI content are often not experts in the questions they are answering. And as stated in the policy, too many community members aren't expert enough to recognize it, leading to upvotes, acceptances, and even bounties for bad answers. And worse, IMHO, the reputation is then awarded to the user for posting these bad answers.

    Sure, it's been possible for bad answers to gain rep long before AI, but now the user doesn't even have to pretend to know that they are talking about - ChatGPT (and others) will pretend for them.

  • Now we're swinging to the other extreme with the new SE policy -- It will be practically impossible to police this. There's simply no way that the mod staff can validate the quality of that many answers (and the number will certainly increase when the floodgates open after since it can't be policed by Moderators).

    Rep farmers rejoice!

There's a place for a middle ground

From nearly the beginning of this debate back in December, I've advocated that responsible use of AI/GPT should be allowed on Stack Sites. This would mean that:

  • If you feel you have sufficient expertise in the question being asked, you can use AI to assist in finding an answer. If you choose to do so, you should, to the best of your ability:

    • Cite any reliance on third-party sources such as ChatGPT for the answer - This is required under the SE plagiarism policy anyway, as you are relying on information from another source and must cite it.
    • Confirm that the answer is correct and works
    • Understand the answer that you are posting
    • Correct any issues that could reasonably be identified
    • Communicate the answer in your own words - This does not mean simply rewriting what ChatGPT answered, but using your own expertise and understanding to write the answer.
    • Reply to follow-up questions/comments about your answer
    • Warn of corner cases or potential issues if needed

The question is what to do with the rest. Based on my experience seeing incomplete answers, untested/unvalidated code, and just plain GPT hallucinations, I still believe that these should be aggressively removed. Suspension may not be the proper "first response", but the answer should be removed until and unless the user rewrites (and has the ability to) to the criteria above.

IMHO, we should continue the use of AI tools and other heuristics to combat the potential for bad answers from users who lack the needed expertise to help here and (in some cases) are just rep-farming, sometimes even with more nefarious purposes, such setting up future spam accounts that will have the rep needed to comment and edit other answers.

14
  • Suppose you follow the middle ground, what happens if you remove step 1? What value does chatgpt actually give you? May 31, 2023 at 20:37
  • 6
    @user1937198 Quite a bit, IMHO - ChatGPT can be extremely helpful in generating ideas on how to solve something, as long as you have the expertise to know when it's tossing out good suggestions and when it's just completely hallucinating. May 31, 2023 at 20:40
  • 1
    If it's just ideas, go back and find a source for the idea elsewhere afterwards, you need so for the understand phase. May 31, 2023 at 20:44
  • 3
    I don't think that even correct AI-generated content should be allowed. When you post an answer, there's an implicit contract with other users that you're writing from your own expertise and experience. Posting AI-generated content strikes me as fundamentally dishonest because the poster's not sharing their own knowledge or experience, and they're representing that something works that they have no direct knowledge of. Jun 1, 2023 at 2:01
  • 1
    @EJoshuaS-StandwithUkraine While I can understand that viewpoint, I think it really depends on the type of question/answer involved. I've answered many questions where I didn't know the answer, but simply offered suggestions based on research that I actually couldn't confirm personally, since I hadn't experienced the issue the OP was having. Take this one for example - I had the expertise to comb through the Github issue and find some possible solutions, but I couldn't confirm any. Jun 1, 2023 at 2:36
  • 4
    If I was using ChatGPT to research some possibilities, I'd at least have the expertise to know if the suggestion was reasonable or not, and avoid regurgitating GPT hallucinations, or even those I just knew wouldn't be useful. Jun 1, 2023 at 2:37
  • 7
    Can GPT be useful? Yes, when you use it responsibly and have sufficient subject matter expertise so you can properly evaluate its output. Are the majority of people attempting to use ChatGPT (etc) for SE answers actually doing that? Probably not, especially if they're just posting raw GPT output.
    – PM 2Ring
    Jun 1, 2023 at 14:13
  • GPT can be downright dangerous if you think you're using it responsibly but you don't understand what it's given you. One of the SO Python Room Owners was recently hired as a consultant for a software project in a field in which he has considerable expertise. The code base looks good on the surface, but it seems to be riddled with flaws.
    – PM 2Ring
    Jun 1, 2023 at 14:13
  • 1
    (cont) A few days ago he learned that most of that code was generated with the help of GPT-4, and none of the human "authors" actually understand the code that they've submitted. So his job of debugging this mess has turned into a nightmare. See chat.stackoverflow.com/transcript/message/56391171#56391171
    – PM 2Ring
    Jun 1, 2023 at 14:13
  • 3
    @PM2Ring Absolutely agreed - Responsible use is good, but the Mods should still be allowed to police irresponsible use, since it's potentially damaging - Pretty much the reason for my answer :-). Jun 1, 2023 at 16:25
  • 1
    I appreciate your point on rep farming in particular. Jun 7, 2023 at 2:30
  • 1
    I am not sure whether rewriting correct ChatGPT4 answers is a good investment of energy. For this question e.g. stats.stackexchange.com/questions/76925/… I literally copy and pasted the raw ChatGPT4 output because I immediately saw the answer was correct & could easily verify it. A rewritten version would probably have been worse and less detailed. I agree you need to have some domain expertise to get the most out of ChatGPT4. But for me it solves my problems faster now than SO.
    – user1377115
    Jun 13, 2023 at 18:45
  • @TomWenseleers You're right - It may not be all that useful, but perhaps for a slightly different reason. When I originally wrote this, I believed that "writing in one's own words" could help demonstrate that they had taken the time to understand the answer. Now, after observing so many GPT answers, I see that far too often users just "reword" the output, which I agree isn't useful in the least. Jun 13, 2023 at 18:49
  • Err, I see that I made that point in my answer originally (which I clearly didn't re-read). Perhaps a better phrase would be "add your own expertise". Jun 13, 2023 at 18:53
24

I am highly skeptical about the actual factual basis for this policy. SE seems very hand-wavy about their actual supporting evidence behind this.

For one thing, if SE can reliably determine whether a specific suspension was based on a false positive, they can also reliably determine which posts were generated by ChatGPT. On the other hand, if they can't reliably determine that, it calls the entire analysis into question.

Also, I'm not sure which "legitimate contributors" are being driven off. Most of the ChatGPT-generated content seems to be coming from low-rep accounts that have little to no other content (or who produce mostly low-quality or mediocre content). It's hard to imagine, for example, Jon Skeet using ChatGPT because he simply doesn't need to. Almost by definition, legitimate contributors have the expertise to write their own content.

2
16

I honestly don't see why this policy was bestowed upon us.

I had never heard of GPT detectors prior to this, but can easily imagine they are beneficial for moderators on large sites in the network.
But as with all the tools at our disposal to cut down low-quality content, it is merely another tool. Moderators usually have proven to be users with common sense, able to critically consider the larger picture. If content is thought to have been generated by AI, it and its poster will not immediately be deleted. (If they actually are, the moderators responsible for those actions will simply have to account for those actions and face whatever consequences, as has been agreed to in the moderator agreements, and we don't need to adopt a completely new policy.)
Unless concrete numbers are provided to illustrate the contrary, I don't believe that, demonstrably directly through the use of these detectors, a relatively larger number of wrongful suspensions has taken place.


I will leave commenting on the outrageous way this was handled for another thread.

3
  • 1
    Content that was generated by AI was immediately deleted. This was the policy introduced by SE at the end of last year.
    – Dharman
    May 31, 2023 at 15:49
  • 2
    @Dharman And how was that assessed?
    – Joachim
    May 31, 2023 at 18:37
  • 8
    @Joachim After you've seen enough GPT content (and according to one mod, SO was consistently getting over 800 a day at one point), you don't even need an automated tool to identify what is likely GPT, but it's nice to have the "confirmation" from a tool as well. But it's not just the automated tools that are now disallowed - Just about any analysis is off the table. Oh, and I think we're already at several hundred today on SO after this policy change. The difference is astounding. Jun 1, 2023 at 14:00
9

This policy stands to undermine a core value proposition of the SE/SO platform.

As I spelled out in this meta post at my "home" SE site (RPGSE):

Under the basic premise that the SE value proposition is a favorable signal to noise ratio, ChatGPT only provides noise.

While I am open to the possibility that some AI-generated content may be able to meet SE and SO standards of quality, I find this policy move to be anti-Human and pro-machine/algorithm in character.

What makes this platform so valuable is the People - flesh and blood - who volunteer their time and expertise to build a community based on the creation of high quality content. The reason that SE/SO is successful is due to the People who contribute.

Please step back from this stride taken in the wrong direction.
People matter.

2
  • 6
    But according to the CEO, "community is the future of AI" (whatever that means!)
    – kaya3
    Jun 6, 2023 at 17:43
  • 5
    @kaya3 A term I used to hear about utterances of that kind was gobbledygook, but I think that bit of slang has passed from usage. Jun 6, 2023 at 17:44
5

The policy and much of the discussion misses the point:

The goal is a site with high-quality1 questions and answers.

I trust moderators and most users to apply due diligence when suspending users for consistent bad-quality posts. To me it would not make a significant difference whether that bad quality was ChatGPT-generated or not. (And on the flip side, I would welcome any high-quality content, regardless of its authorship.)

Therefore, false positives are simply not an issue: The site is better off without them regardless.


1How ever "quality" is defined — I'm tempted to say "I know it when I see it" ;-).

3
  • 1
    This answer ignores the people who ensure that the posted content is of high quality.
    – JonathanZ
    Jun 6, 2023 at 13:12
  • 2
    @JonathanZonstrike How so? Mods and users should be able to moderate content with all available means and simply focus on quality, not on hard-to-detect traits orthogonal to quality (e.g. authorship). Jun 6, 2023 at 13:15
  • 2
    ChatGPT-ness is neither hard to detect nor orthogonal to quality. It also correlates highly with "wastes lots of time". And I'll apologize beforehand for not replying to any follow-ups - I forgot I'm respecting the strike!
    – JonathanZ
    Jun 6, 2023 at 13:21
-20

Banning AI will lead to a Blade Runner future

Which can be avoided by:

  1. AI answers should be allowed alongside, but not revealed unless the reader explicitly wishes to see them.
  2. Questions should specify their consequentiality, to allow selection and review of AI responses for higher criticality questions.

"Bad Actor" Reasoning:

  • If there are no reliable methods to detect AI-generated content, banning will do little to stop it, and will only prompt human operators to modify their AI to mimic humans.
  • Segregating AI and human content opens a legal avenue for such operators to participate.
  • This creates competition for better answers between AI entities.
  • It's better than nothing. It allows for a second-chance search when a question remains unanswered.
  • Allows for Stack Exchange to monitor AI content, to gauge when it reaches and surpasses the quality of human-generated content.

Response Quality Reasoning:

  • Keep Enemies Closer. It is undeniable that AI quality will eventually reach and exceed that of humans. It's better to keep it close and monitor it than to simply ban it (partially repeating the above point).
  • Account for Criticality. Not everyone is asking questions about life support machines or nuclear reactor programming at work. Some users inquire about ping command parameters, and while ping may flood a server given incorrect parameters, it's less severe than nuclear reactor programming. Specifying consequentiality of question creates necessary countering force from the question itself.
  • Wasabi and Truffle are examples of products that often don't contain the elements they claim in their names. Most users do not care where the answer comes from as long as it solves their problem. And with AI there may be no unanswered questions. But if AI answers are banned, those users would go to a Stack Exchange clone run by an evil AI and get into trouble.

Responsible segregation of content would create a harmonised society between humans and machines. Banning it would create a divide and lead to a Blade Runner society where your coworker may turn out to be an illegal replicant because they are banned.

4
  • 3
    an AI generated this didn't it? Nov 25, 2023 at 18:43
  • 8
    "Banning AI will lead to a Blade Runner future" - in what way? Can you elaborate? "Your coworker may turn out to be an illegal replicant because they are banned" doesn't make sense to me. There are plenty of ways to get suspended (not banned) from Stack Overflow that don't involve posting AI-generated answers, and a person posting AI answers is very different from a person actually being an robot in disguise, as in Blade Runner.
    – F1Krazy
    Nov 25, 2023 at 18:50
  • 1
    "Wasabi" is ambiguous/overloaded. What does it refer to? Something called "Wasabi" (a custom programming language) was part of the impetuous for Stack Overflow in 2006/2007/2008 ("Joel Spolsky has jumped the shark"). Nov 25, 2023 at 23:38
  • 9
    Re "It's better than nothing": No, not when an AI answer is totally wrong (or even blatantly wrong) in about 70% of the cases. Nov 25, 2023 at 23:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .