A new Network policy regarding AI Generated content was recently announced by Stack Exchange.
As discussion is not currently permitted on such posts, please discuss the policy in the answers here so that we have a single place for discussion and feedback regarding the policy.
(also I asked Journeyman Geek and he said having a discussion post sounded good)
The entirety of the question from the guidance Q&A is:
What is the network policy regarding AI Generated content?
Earlier this week, Stack Exchange released guidance to moderators on how to moderate AI Generated content. What does this guidance include?
Reading this, I would expect, you know, guidance for curating sites and moderating content. Unfortunately, Philippe fails to provide any guidance at all for non-moderators, you know, the users responsible for the overwhelming majority of moderation across the network.
Instead, we got a corporate grandstand about how moderators need to be more careful when suspending users for AI-related issues, something that should have been an email, not a dramatic network policy bomb.
And I, a regular user, am left with no further guidance about curating these posts, and the mods are left to deal with my unguided flags.
This was somewhat mentioned (by me) here, but I'm creating a dedicated post about it
We recently performed a set of analyses on the current approach to AI-generated content moderation. The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high.
Could you post your analysis itself, and not just your conclusion? Bonus points if you can post the data you analyzed as well.
... we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts
That is... possibly true. But as you (should) know, a suspicion isn't a reason for a ban. At least, you explicitly state "We've reminded moderators that suspensions ... are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics.". Stating effectively "we did some research, trust us please" is effectively an unverified heuristic. Sorry.
A much better statement would include proof of that statement. Or data to support it. Or a link to a peer-reviewed paper that supports it. Not just "we also suspect". That isn't a high enough bar here.
Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.
Please publish this evidence. Or, if you can't (i.e., because it contains PII), then please state explicitly why you can't publish it. Side note: a user (who I'm not naming unless they do here) posted a detailed analysis of SE traffic post-ChatGPT. That is worth reading if you haven't, yet.
We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.
That's been known. Since they were released, it is not a secret that they have a somewhat high FP rate. This is why that was used as one data point that a user may have (ab)used ChatGPT, and therefore it was worth looking for additional evidence. While I haven't asked the entire mod team, I'm fairly confident that they weren't randomly suspending users based on that with no further evidence.
Could you instead provide data on the number of wrongfully-suspended users? Bonus: how are you determining if a user was wrongly suspended? Some more context would benefit that claim.
You make a large number of claims, yet provide ZERO sources/data points to support them. Please be more specific.
The new policy can be summarized as "You are allowed to moderate AI generated content, but you are not allowed to use anything to detect such content including your own brain."
AI generated content cannot be moderated using the regular approach used for low quality content because it does not fall under low quality categorization:
Temporary policy: ChatGPT is banned
The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting.
If moderators are not allowed to moderate AI generated content based on it being AI generated, the only thing we have at our disposal is leaving them to the community to downvote and delete vote, which is not a feasible approach as the community is already having hard time moderating even blatantly obvious poor content due to the sheer number of such posts.
If all means of AI detection are excluded that means AI generated content will not be moderated and will cause permanent damage to the sites and their quality. This deterioration will also happen rather fast as it will be extremely easy to flood the sites with AI content.
I can’t fathom why such a policy was made, unless the company actually wants to allow AI generated content for some inexplicable reason; but, saying that would directly go against what the vast majority of the community wishes. In that light, preventing moderation can look like the safest approach as it allows them to to have the cake and eat it at the same time.
Not only is this extremely bad policy, but it is also not backed by any verifiable data that moderators were suspending people en masse for valid content (mistakes are always possible, but they are also easily correctable as they have always been).
Please, allow moderators to continue moderating AI generated content using all detection means necessary, and issuing suspensions as they were doing before this policy. This is the only way we can preserve the quality of the sites and future participation of experts willing to invest their time, to share their knowledge.
So, so, many issues here that it's difficult to know where to start. And I'm not sure I could cover all my concerns in one answer in any event.
I've already addressed some points on the Mod Team and direct with Cesar in the TL (kudos to Cesar for showing up and doing his best to respond to a veritable deluge of angry mods BTW, you're a gent and a scholar) so I'm going to take the opportunity here to look at some things directly from the announcement posts.
We've reminded moderators that suspensions (and typically mod messages as well) are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics.
I strongly feel this mischaracterizes both what the mods were doing in handling GPT generated content by implying that we were going round suspending at the drop of a hat based on shallow guesses or blindly taking detectors as gospel and simultaneously implying that mods were somehow moderating this content without SE's awareness of what was being done and how it was being done. Neither of these are true, so characterizing this as trigger-happy mods who got carried away and are now being reined in feels very much like we're being thrown under the bus, and whether that's intentional or not (I'd lean not, the CM team has pretty much always come across as decent folks with a tough job who have our backs and I'd like to think that's true) doesn't really matter to the poor sods getting splattered by the bus, it hurts just the same.
But putting the potential for hurt feelings aside it also runs straight into a consistency problem in the very next paragraph:
As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.
Wait, so we can only act on "verifiable" malfeasance, AI detectors are unreliable, can't be trusted, heck, are even racist (apparently), and our own judgement and analysis can't be trusted either but for "poor quality" content we can use both of those things when moderating and they therefore count as evidence of "verifiable malfeasance", an AI-detector tells you nothing directly about the quality of the content, we all know that the current crop of GenAIs are liable to produce poor quality answers but the fact that it was produced by an AI is not proof in of itself that the quality is poor. So the "false positive" rate for a GenAI detector being used to determine if it's AI-generated is too high to justify acting on but surely the false positive rate for determining if it's too low quality for SE purposes (something it's not even designed to do, and doesn't even try to do) is going to be off the freaking charts, but we are to be encouraged to use them for that purpose instead of their actual purpose.
That's as clear as mud :(
Fortunately for me GPT content hasn't been too big a problem on TWP thus far - and when our users asked what the site's stance was we took the position that GPT-sourced content was already not permitted as it was covered by existing policies.
Un-referenced content from there was covered by plagiarism policies, any word-salad was low quality, as was just posting a referenced GPT answer with no additional added value from the poster, as per the Help Center:
Do not copy the complete text of sources; instead, use their words and ideas to support your own. In particular, answers comprised entirely of a quote (sourced or not) will often be deleted since they do not contain any original content.
Nothing in the current policy changes that as far as I'm concerned, it's just as unacceptable as it was before, it's just arbitrarily harder to actually moderate/curate for the mods and the community. Thanks for that. I sure do love it when a job I volunteer for gets made harder for no good reason!
The new Code of Conduct, active as of today, May 31, 2023, forbids inauthentic usage, including:
Plagiarizing or copying content from websites, books, or other online and offline tools without proper attribution in a manner that violates our referencing standards.
In my view, this includes GenAI posts, which are generally copied and posted without any attribution to the software tool that generated it. However, the new policy makes it impossible for moderators to moderate to this standard.
TL;DR, from Google Labs’s generative AI: The new policy was announced without moderator or community input, lacks data to support its claims, and goes against the wishes of the community. Moderators are no longer able to use their judgment to remove harmful content, and the policy is likely to increase the spread of misinformation.
The new network policy regarding AI generated content is problematic. It was rolled out in one of the worst possible ways, goes against the wishes of the community, and is not supported by any available data.
On May 29, 2023, the Community Management team posted an announcement on the private “Stack Moderators” team. They also pushed out a notification to all moderators directing them to read this announcement. To the best of my knowledge, this was the first time that any of the community elected moderators were made aware that anyone at the company had any concerns about how algorithmically-generated content was being moderated or that any research into detecting and handling such content was ongoing.
Simply by making such a policy announcement, the company did not follow the policy on making policies. There was no substantial preview or review period - the policy was declared to be created and moderators were only given a short “heads up” period prior to it being posted publicly. Questions and concerns raised were not adequately answered or addressed prior to the policy being made public.
In addition, the day that the announcement was posted for moderator “heads up” was a holiday in the United States, Canada, the UK, and other European countries. Given that it was announced on a holiday and was then posted a day later to the public, it is highly likely that not all moderators were able to see and react to the policy edict before it was made public. Some moderators may still be unaware.
The policy, as presented to moderators, had two main components:
To support the policy, two claims were made:
Neither claim was supported by data made available to moderators or to the broader community when the policy was made public.
With respect to the first claim that detectors were unreliable, no information was provided about when the detectors were tested (i.e., what version of the detectors were tested), what communities the test data came from, or how many posts were sampled. In addition, there were no examples given for posts that appeared to be miscategorized.
Data to support the second claim that detectors are biased against non-native English speakers, would require some level of PII, so couldn’t be shared (even with moderators), but I would have expected aggregate data, such as the number of users from each country, the participation rate from each country, and the number of suspensions over time. It’s unclear what the baseline suspension and account deletion/destruction rates for various countries are and how suspensions related to algorithmically-generated content compare to other moderation activities.
The claims do raise questions. However, in the absence of clearly specified methodologies and making data available where possible, it is inappropriate to be dictating policies that go against the desires of the community. Many communities on the network have developed policies to address posting algorithmically-generated content, some with clarifications regarding how to attribute generated content and others with a prohibition on the use of generated content. On Software Engineering and Project Management, where I’m an elected moderator, these policies have overwhelmingly positive community support. Nevertheless, in cases where the community has zero tolerance for algorithmically-generated content, the ability to take the appropriate action that the community has decided to be appropriate has been eliminated by the new policies.
I believe that not taking strong, decisive action against algorithmically-generated content is the wrong thing to do. Visitors to the Stack Exchange network come here expecting high-quality answers that draw on human knowledge and experience, with valid citations and references where appropriate. When someone plagiarizes the output of a generator, that person is misrepresenting their own knowledge and experience. Considering that these generators do not have the concept of facts and truth, and only a subject matter expert (often not the asker) can verify the claims, it would be better to remove potentially harmful content. Since the data on Stack Exchange is public, I would expect that future algorithms are trained on the data. This compounds the harm by feeding incorrect information into search engines and future algorithms which would spread the harmful information. The harm of an erroneous suspension - which could be reduced or eliminated by a moderator - is far less serious than the potential harm of supporting the proliferation of incorrect information and training future models on algorithmic outputs.
Moderators are not prevented from acting, but we’re extremely hampered. We are no longer allowed to use our personal judgment or the only tools in the toolbox to curb content that the communities - the people who elected us and have offered their inputs into community policies - do not want to see. Although we can deal with certain classes of content, our choices are to either let some algorithmically-generated content be posted on the network or to go against this new policy.
I will also say that I have, to date, suspended 7 accounts for posting algorithmically-generated content. Of these accounts, 6 of them were newly-created accounts that appear to have been created just to post such content. None of these people reached out to challenge their suspension. I did not hear directly from the Community Management team that any of these suspensions were escalated to them or deemed inappropriate. It is unclear what may have happened if they were allowed to post convincing content that allowed them to increase their reputation and gain access to other functionality.
I do think that we should have discussions around both claims and reach a baseline policy for the network. But we should have the underlying data (to the extent possible) and have discussions on the research and validity of the claims before making policies. And then, when policies are established, they must follow the agreements between the company and the elected moderators to go through working groups, moderator review, and finally appropriate public discussion and announcement.
My thoughts, as someone who has flagged more than 1,200 (subsequently) deleted answers (and some comments, and even some questions) that I believe were GPT:
Note: I've also posted a separate answer on this question on my thoughts on what I feel a sensible AI policy on Stack Exchange would be. I feel it's a separate enough topic under this question that it needs to stand-alone.
I've seen high quality answers deleted, low quality answers deleted, and those that I don't have the subject matter expertise to judge on my own.
I've seen false positives from AI detectors (but many more false negatives), but it's rare that AI detectors would be the only source of confirmation that a question was generated by AI.
I've had flags declined, as the Moderator who reviewed the flag felt that there wasn't enough evidence for deletion (let alone suspension).
I have been, and continue to be, reluctant to post the number of different techniques that I personally use to detect AI, but the Mod team seems to have agreed with them in most cases. And yes, there are indications in the "writing style", obviously -- What else would there be? But analysis of writing style is also banned under the new policy.
Code blocks in an answer generate a much higher percentage of false positives. For that reason, we've recommended that anyone using an AI detector on a Stack Overflow answer first remove any code-blocks. I wonder if the Staff removed code-blocks when doing their data analysis.
I do not believe that attempting to moderate AI answers is unfair or biased against non-English speakers. In my recent review (just before the policy was posted), I scanned 300 answers and automatically ran them through a detector. The false positives that I saw were from, as far as I could tell, native English speakers, or at least those with a high command of the language.
I do not believe that the answers I flagged were simply using ChatGPT or AI for grammar assistance. I personally believe that this is a perfectly acceptable use of ChatGPT here, and I've tested ChatGPT grammar correction on some difficult-to-read posts here. The results do not appear to be GPT "generated" answers like we normally find, nor do they have any higher percentage of false positives as a result of ChatGPT grammar correction.
I'm obviously biased, as I've been so active in the flagging of these answers, but I believe we have had an extremely high rate of true positives in the community's curation of AI-generated answers. If the SE staff disagrees, I would encourage, nay challenge, them to undelete the thousands of posts that they feel have been wrongfully deleted. Hey, it's the least you can do after you feel that the posters of those answers have been wrongly suspended.
After undeleting the answers, take a random sample of them and explain here on Meta why the Staff feels they are not AI.
I apologize for any mistakes due to lacking the full picture that only moderators have since apparently (at least based on what I read from multiple sources) the company has shown two "different but not completelly unlike" pictures to the general public and the few who have access to the moderator private chatrooms. And if the general public already thinks this is horrible, I can only imagine what the mods got to see instead to be planning for a full fledged strike.
Alas... this is kinda pathetic. Basically, it shows that the company lacks a basic understanding of what the difference is between "computer assisted decision making" and "computer automated decision making".
Let me show you an example.
Art. 22 GDPR Automated individual decision-making, including profiling
The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
Obviously, this is not the same as what we are discussing here, but it should make one thing clear: Automated decision-making has been identified as a risk for quite a while now.
...
......
But... I beg you this question. Who sane in their mind would equate the use of a tool to SUPPORT manual, user made moderators decisions with automated machine handled actions based on heuristic detection?
This seems very hypocritical and feels like part of an agenda to once again take the role of "heroes of light" in media (as a thing like the "welcome wagon" felt in the past - not motivated by care but by marketing advisory) or tries to follow the recent trends (AI generation being $$$ in the eyes of stakeholders?).
I will be very blunt and I apologize if this seems harsh. But I find it quite ironic that in all the years I have spent here the company never posted a policy against good old Smokey. I mean, it is heuristic detection of bad content so that actual humans can avoid looking for themselves and have a pre-processed list of things to manage. It is the same exact things and should be banned too.
Please, holy overlords, suspend all the corrupted users that have usage privileges on Smokey and ban its evil creator mastermind until the thermal death of the universe at the very least.
"And another thing...(cit)", another funny question that came to mind... how does our company actually plan to implement this shiny edict? How do you plan to know if I used a tool to find this poorly generated useless content or if I stumbled upon it by chance?
Because either the content is bad and should be removed either way (and being generated is only the reason it ended up bad) so you can't oppose users for asking to remove it or if it is brilliant, useful, and legally abiding (by mentioning sources for example) no one would have to bother with trying to remove it even without this waste of time promoting nonsensical policies.
Please see my other answer here for some additional thoughts on AI moderation in general.
As I mentioned in my other answer, I have been extremely active in the community moderation of AI answers, with over 1,200 flagged and subsequently deleted.
That said, I don't feel that either the old policy (for most of the sites) nor the new policy is ideal. We've gone from one extreme to the other:
On Stack Overflow and many other sites, GPT/AI content was entirely banned, whether it was good content or bad. There are multiple reasons for this, of course, which are listed in the policies. That said, we are throwing out good answers as well, even those from users who used AI tools to supplement their own expertise.
The problem isn't those "responsible" users, it's the vast majority of others.
To state the obvious, SE works on the basis of reputation. And users posting AI content are often not experts in the questions they are answering. And as stated in the policy, too many community members aren't expert enough to recognize it, leading to upvotes, acceptances, and even bounties for bad answers. And worse, IMHO, the reputation is then awarded to the user for posting these bad answers.
Sure, it's been possible for bad answers to gain rep long before AI, but now the user doesn't even have to pretend to know that they are talking about - ChatGPT (and others) will pretend for them.
Now we're swinging to the other extreme with the new SE policy -- It will be practically impossible to police this. There's simply no way that the mod staff can validate the quality of that many answers (and the number will certainly increase when the floodgates open after since it can't be policed by Moderators).
Rep farmers rejoice!
From nearly the beginning of this debate back in December, I've advocated that responsible use of AI/GPT should be allowed on Stack Sites. This would mean that:
If you feel you have sufficient expertise in the question being asked, you can use AI to assist in finding an answer. If you choose to do so, you should, to the best of your ability:
The question is what to do with the rest. Based on my experience seeing incomplete answers, untested/unvalidated code, and just plain GPT hallucinations, I still believe that these should be aggressively removed. Suspension may not be the proper "first response", but the answer should be removed until and unless the user rewrites (and has the ability to) to the criteria above.
IMHO, we should continue the use of AI tools and other heuristics to combat the potential for bad answers from users who lack the needed expertise to help here and (in some cases) are just rep-farming, sometimes even with more nefarious purposes, such setting up future spam accounts that will have the rep needed to comment and edit other answers.
I am highly skeptical about the actual factual basis for this policy. SE seems very hand-wavy about their actual supporting evidence behind this.
For one thing, if SE can reliably determine whether a specific suspension was based on a false positive, they can also reliably determine which posts were generated by ChatGPT. On the other hand, if they can't reliably determine that, it calls the entire analysis into question.
Also, I'm not sure which "legitimate contributors" are being driven off. Most of the ChatGPT-generated content seems to be coming from low-rep accounts that have little to no other content (or who produce mostly low-quality or mediocre content). It's hard to imagine, for example, Jon Skeet using ChatGPT because he simply doesn't need to. Almost by definition, legitimate contributors have the expertise to write their own content.
I honestly don't see why this policy was bestowed upon us.
I had never heard of GPT detectors prior to this, but can easily imagine they are beneficial for moderators on large sites in the network.
But as with all the tools at our disposal to cut down low-quality content, it is merely another tool. Moderators usually have proven to be users with common sense, able to critically consider the larger picture. If content is thought to have been generated by AI, it and its poster will not immediately be deleted. (If they actually are, the moderators responsible for those actions will simply have to account for those actions and face whatever consequences, as has been agreed to in the moderator agreements, and we don't need to adopt a completely new policy.)
Unless concrete numbers are provided to illustrate the contrary, I don't believe that, demonstrably directly through the use of these detectors, a relatively larger number of wrongful suspensions has taken place.
I will leave commenting on the outrageous way this was handled for another thread.
This policy stands to undermine a core value proposition of the SE/SO platform.
As I spelled out in this meta post at my "home" SE site (RPGSE):
Under the basic premise that the SE value proposition is a favorable signal to noise ratio, ChatGPT only provides noise.
While I am open to the possibility that some AI-generated content may be able to meet SE and SO standards of quality, I find this policy move to be anti-Human and pro-machine/algorithm in character.
What makes this platform so valuable is the People - flesh and blood - who volunteer their time and expertise to build a community based on the creation of high quality content. The reason that SE/SO is successful is due to the People who contribute.
Please step back from this stride taken in the wrong direction.
People matter.
The policy and much of the discussion misses the point:
The goal is a site with high-quality1 questions and answers.
I trust moderators and most users to apply due diligence when suspending users for consistent bad-quality posts. To me it would not make a significant difference whether that bad quality was ChatGPT-generated or not. (And on the flip side, I would welcome any high-quality content, regardless of its authorship.)
Therefore, false positives are simply not an issue: The site is better off without them regardless.
1How ever "quality" is defined — I'm tempted to say "I know it when I see it" ;-).
Which can be avoided by:
"Bad Actor" Reasoning:
Response Quality Reasoning:
ping command parameters, and while ping may flood a server given incorrect parameters, it's less severe than nuclear reactor programming. Specifying consequentiality of question creates necessary countering force from the question itself.Responsible segregation of content would create a harmonised society between humans and machines. Banning it would create a divide and lead to a Blade Runner society where your coworker may turn out to be an illegal replicant because they are banned.
