Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned.
However, the reasons for the ban really apply to much or all of the network, and certainly for sites that are similar in nature to Stack Overflow.
I suggest (temporarily) banning ChatGPT network-wide.
With due consideration, we've decided no network-wide, general policy regarding banning ChatGPT, or other AI generated content, is necessary or helpful at this time. However, as detailed in answer to: "Is attribution required for machine-generated text when posting on Stack Exchange?", we do consider AI generated content to be "the work of others" and the requirements for referencing must be followed for all such content on the network.
I want to be clear: I am not in any way intending to downplay the significance of ChatGPT, nor the disruption it has caused to the platform over the last few weeks.
Instead, we're going to stand by the comment I left on this post on December 5th:
While we evaluate, we hope that folks on network sites feel comfortable establishing per-site policies responsive to their communities’ needs.
Each site on the network is going to be impacted by ChatGPT (and its future iterations) in different ways. Of all the sites on the network, Stack Overflow was hit by far the hardest. However, we are measuring its impact both on Stack Overflow and across the network -- and, the impact of ChatGPT is currently diminishing everywhere. Some sites will see more or less activity on a given day, but outside Stack Overflow, it appears to be leveling off to a very slow trickle. On Stack Overflow, its usage rate is still falling quickly.
Because sites are impacted to such different degrees by the usage of ChatGPT, we encourage sites to create these policies as they become an issue. A blanket policy does no good if affected communities are not simultaneously developing the methods they use to combat the material problems they face. Instead, it risks being actively unproductive, by setting an expectation that sites will purge this content without giving them targeted tools to do so.
Our work internally progresses on identifying these posts and making our systems more resilient to issues like this in the future. We recognize that this is a shot across the bow, and the problem isn't going to go away in the long term. But for now, it seems we've weathered this storm mostly intact. As always, we'll reevaluate this decision in the future, if the circumstances warrant it.
And, of course, if any site experiences a volume of GPT posts that are cumbersome to manage, or a site needs any other support managing an influx of unwanted content, we are always happy to help apply the tools we have at our disposal.
I'd advocate for a hard line on this.
Something that's utter random garbage is actually less harmful than almost correct garbage that needs an expert to work out.
If you're using the output of a machine learning or AI tool WITHOUT verification and/or disclosure, and we're going to have to waste time working this out, the user clearly isn't here with the right intentions. As such I'd be inclined to treat the user as severely as needed, and I'd be tempted to start with longer suspensions.
It's a good way to prevent folks who are clearly wasting people's time from wasting more of it.
Let me back this up and look at the broader picture here. We're about to enter a time that Star Trek once hinted at, but is now here and ready for your use: artificially generated content. The Internet has now reached maturity and search engines can run as much training data down your throat as you can handle. Quite literally we're watching the true next iteration of the Internet be born right now. Google can show you what already exists. AIs can generate almost anything your mind can dream up.
The problem there is ownership. The US helped the Internet in its infancy by making an environment where you can "Fair Use" just about everything.
[O]nline intermediaries that host or republish speech are protected against a range of laws that might otherwise be used to hold them legally responsible for what others say and do.
If you post a meme on SE that's using a copyrighted image, all SE has to do is take it down to avoid liability. With such a low bar to clear, it's allowed Fair Use to thrive. But... what do we do when these AIs start generating content that is wholly based on the works of others?
Allen's victory prompted lively discussions on Twitter, Reddit, and the Midjourney Discord server about the nature of art and what it means to be an artist. Some commenters think human artistry is doomed thanks to AI and that all artists are destined to be replaced by machines. Others think art will evolve and adapt with new technologies that come along, citing synthesizers in music. It's a hot debate that Wired covered in July.
And will these tools drown out actual users?
Established artist communities are at a tough crossroads because they fear non-AI artwork getting drowned out by an unlimited supply of AI-generated art, and yet the tools have also become notably popular among some of their members.
These are from September, involving art communities, but all ChatGPT is doing is basically a fancier search than Google can serve up. And this problem isn't going to go away because we're using machine learning everywhere. If an AI can't do it now, just wait.
For my fellow mods and I on Stack Overflow, the root problem boils down to two issues
Parrots are very smart birds and they can mimic sounds very well. But parrots cannot talk. They emulate the sounds they hear but they do not comprehend what they're saying.
ChatGPT is better than any chatbot we've seen. It writes in natural language, not the stilted text that typifies such systems. It generates what appears at first blush to be quality content. But we've noted that ChatGPT is doing what a lot of inexperienced users on Stack Overflow do: try to be the best-sounding parrot. Someone asked ChatGPT if it should be allowed to answer Stack Overflow questions and posted it in a now-deleted answer on our rule. I do have to admit it's amusing
I am writing to express my extreme disapproval of the idea of allowing ChatGPT answers on Stack Overflow. This would be an irresponsible move and would fundamentally undermine the integrity of the platform.
Polly want a cracker? (fascinating English.SE etymology lesson there)
ChatGPT is not a writer or programmer, it's just copying other smart-sounding sources that look highly relevant. It can (to its inventors' credit) write simple, passable code. But ChatGPT doesn't know what SQL injection is and another mod experimenting with it found it will merrily give you code suggestions using it. Why? For too long that was how a lot of people did it on the Internet, and you can still find that poor advice everywhere. Hence why a lot of folks on Stack Overflow will incessantly warn you about not doing that.
The various AI image generators can afford to be parrots. In fact, that's what the users want. "Paint me a picture of a cat riding a unicorn carrying a shotgun in the style of Vincent Van Gogh". That doesn't work so well when you need something like "How do I create a contact form on my WordPress blog?" That's not something I would trust ChatGPT to answer. It might give you workable code that someone will use to gain control of said blog.
I won't link the person who said this, but this is a real Twitter post
So I started a new stackoverflow account and I am plugging random questions without answers into https://chat.openai.com/chat and pasting the answer. So far, after 9 answers in 1.5 hours it has 1 accepted and 3 upvotes and a reputation of 62...
And
6 accepted answers, of the 26, 11 upvotes, 5 downvotes. I am not checking the answers in any way. I'll give it a rest for now and see how we do tomorrow...😄
I'm sure ChatGPT (or Google if you wanna go old school) can tell you where to find this user, but that second statement is downright scary. They're not checking the answers in any way. In other words they also are not a programmer. I've already gone over why that part is a problem, but there's another one here: This user admits they didn't write the code. From the Meta Stack Overflow FAQ on non-English content
Translating a question for a non-English speaker sets them and all participants up for a poor experience, due to the OP not being able to follow and respond to feedback from comments, understand answers, or get assistance from the Help Center.
The inverse of that is true here. A user comes in and asks a question. Someone posts an AI-generated answer without understanding anything about the question. What if the questioner posts a comment asking a clarifying question? What if the AI did something confusing in their answer? This user cannot interact in a meaningful way with said questioner. And, going back to the art site debacle, who owns it if it's been copied? I mean, we don't use Creative Commons for no reason. This butts up against plagiarism in a most uncomfortable way.
Answers generated by an AI should be considered as being written by the AI. That means you can quote them like any other source, but you must attribute them to the AI, just like any other source, and not use a bulk-copied AI answer quote as an answer. This way, we're avoiding the thorny issues of people running to the latest AI to get answers so they can copy-paste them as their own. We have plagiarism tools (current and forthcoming) in this wheelhouse so we don't need to reinvent any wheels.
If the AI gives a bad answer, we have votes for that.
Some of the other answers have focused on the difficulties with moderating AI-generated content for correctness, accuracy, usefulness, etc., and I don't meaningfully disagree with those answers. But I do think there is a big, big problem, especially for a website network like Stack Exchange, which is the near-impossibility of properly attributing credit for the words that the AI is producing.
The main problem is that the vast majority of AI-generating algorithms publicly available, including ChatGPT, do not properly credit the sources that were used in the training model that the AI used to tune its internal model. This means that any answers generated by the AI are, de facto, plagiarism.
Many of these AI algorithms/organizations will obviate around the plagiarism issues in ways that avoid obvious legal culpability: ChatGPT, for example, refers to this data as "human AI trainers". But it remains the case that any use of these algorithms will constitute plagiarism until/unless stringent rules are applied to these organizations that they A) directly attribute every text that was used to train the AI model, B) only obtained these texts from authors that gave explicit, documented consent to have their texts used in these models, and C) document these contributions and make that data publicly available.
That doesn't necessarily mean it will never be appropriate to reference an AI-generated text per the normal Stack Exchange rules about quoting sources/using quote blocks in an answer (I have some skepticism about how often that will be appropriate), but I do think that this means—pending major ethical overhauls in AI-generated content—it will never be appropriate for AI-generated text to manifest as the body of an answer posted on this network.
EDIT: I don't know why this answer in particular is attracting users who seem ignorant about the relationship between Neural-Network AIs and actual biological human cognition, but to address a few recurring comments I've been seeing:
Addendum: I am not saying that any/all AI Algorithms are not/will never be capable of original thought, but Neural Networks certainly are not.
Neural Networks get their name from a conceptual similarity between how biological neurons function and the abstract model of "Neurons" deployed in Neural Networks, which unfortunately has led to false equivalencies being made between the two, implying that human neurons and AI neurons are essentially equivalent or "the same, but varying in speed/power/etc.", and it needs to be understood that this is fundamentally untrue.
There's a lot of technical reasons why the comparison isn't particularly cogent, but to reduce it down to the simplest form possible: the biggest reason that Neural Networks cannot produce original thought is because they're not trying to. Neural Networks are designed with 'emulation' of existing data as an end-goal. To use ChatGPT as an example, its goal is not to create original ideas, its goal is to produce text that its model detects, with high consensus, is similar to the text that a human* has already produced. Stable Diffusion and other AI-Art-Generating algorithms operate on similar principles, attempting not to produce original paintings, but instead to produce an image that is similar to images that humans have already created.
* as defined by the model's training data, which I am assuming consists entirely of human-produced content but it should be acknowledged this may not be the case since the sloppiness with which sources are pulled into the model can in many cases pull in other AI-generated content, which would train the model not to talk like a human but instead talk like an AI
A really good case study for this "don't produce something original, produce something that resembles something that has already been created" function of neural networks is a test I performed, using Stable Diffusion, where I described a hypothetical D&D character ("Githyanki Woman in Red Trenchcoat") and had the algorithm try to generate an image. These were the produced results, which give us some important insights. The algorithm attempted to produce 4 images, and of those 4 images, two of them clearly modeled Cosplay photos (with some distressing facial distortion, which I'm guessing is the influence of the 'Githyanki' species modifier) and the other two generated... images of physical tabletop tokens, stand and everything.
It's not difficult to figure out what happened. The metadata of 'Githyanki' (a rare but playable humanoid species in some editions of Dungeons and Dragons) connected the algorithm to images that were created in Heroforge and other TTRPG token creators, along with cosplay photos of people modelling Critical Role characters (or their own original D&D characters). This is how the algorithm tried to establish 'similarity' between the images it created and the images it associates with the prompt provided.
But it's also very clear that none of the images produced are images that a human artist would create. In fact, the only way a human would produce any of those images themselves is if they had attempted to cheat the prompt—grabbing an image from a token generating website or a cosplay photoshoot and passing it off as their own work.
In other words, the AI did an exemplary job of replicating what a human committing blatant plagiarism might attempt to produce.
The reason I'm going into this long explanation and case study, aside from reinforcing my point about modern AI-generation algorithms only really being capable of committing mass-scale plagiarism, is to emphasize this point about these algorithms not being capable of original thought, and why arguments like "it's just like a human brain!" or "humans don't have to cite their training, why should AI have to?" are invalid. Some prompts are easier for the AI to replicate than others; certainly my example prompt broke the limits of what the AI was capable of replicating. But it's critical to understand that even when the AI is doing a much better job of reproducing what the prompt has asked it to generate, it's still doing the exact same thing as when it broke under the weight of my request: copying the data it found in its training data.
Now, the ethics of these algorithms can be solved: these algorithms could purge their databases, begin taking data only from artists who have explicitly consented to have their art/writing ingested, properly cite each work ingested, and make that data publicly available and easy to access, solving the widespread plagiarism I started out by addressing. I would even go a step further and argue that to be truly ethical the algorithm would also have to be able to cite the specific works whose influences composed the specific resulting output, but I'm guessing that might be technologically infeasible.
Important: something being "technologically infeasible" is not the same thing as saying "so we don't/shouldn't have to care about it". If AI-Art or AI-Writing can't be done ethically, I would argue it shouldn't be done at all, and whether or not running these algorithms ethically is a thing that can be done isn't in my purview.
But we need to dispose of this notion that these algorithms are, in any meaningful sense, engaging in original thought. They're not, and they're not truly replicating human thought, either at the macro level of human consciousness or at the micro level of individual brain neurons. Neural Networks are designed to emulate and reproduce existing works based on what the algorithm detects as being similar to those existing works, and they can be in some cases very uncanny replications, but they aren't original creations.
I fully support network-wide AI ban. A permanent one.
For now, the most prominent issue with AI answers is that they are mostly incorrect. Because they look like well-written elaborate answers they don't look like common low-quality answers that are otherwise poorly written or just contain code snippets (on Stack Overflow).
Because of that, they are harder to moderate - common flags like low quality and not an answer don't apply. Yes, incorrect answers could be downvoted, but that also requires some expertise in subject matter.
If the user posts plenty of such answers, they can also avoid an answer ban, by getting upvotes from other users deceived by the initial good impression.
Overall, incorrect AI generated answers are evidently harmful.
If we take a look beyond incorrect answers, it will become clearer why a permanent ban is necessary.
Let's say we allow users to post AI generated answers they will check for correctness (ignoring for the moment other potential issues mentioned in other answers).
If the user has the expertise to verify the correctness of an answer, then they also have the expertise to fully write one. Grammatical errors were never an obstacle when answer content was otherwise good, and there is plenty of users that edit such answers, so improving language is not the advantage in this scenario. Also, there are other tools that can correct language errors.
Because users that have the expertise, don't need AI, if we allow such answers to be posted, there would be extremely small amount of such answers, as verifying the AI generated content would probably take more time than writing the answer.
We are not going to lose any good content posted and verified by the experts in such cases.
This will probably never become true, but let's pretend that AI will generate if not always, then correct answers most of the time.
If this would be true, then people could just directly ask AI the question, instead of posting it on Stack Exchange. Similarly, currently people are supposed to read the documentation and use Google or other search engines for solutions before simply asking questions.
We don't need to allow AI generated answers, if AI can directly solve the problem.
Another aspect why we need to permanently ban AI, is that AI didn't magically appear. It is being trained on human generated content. Stack Exchange strives for quality, and we need to preserve human generated content to do that. If we allow AI answers, no matter how good they are, we will start losing experts willing to share their knowledge.
If you can post in a few seconds what AI gave you, why would you bother adding original content? With time, less and less people will be motivated to participate, and in the end, Stack Exchange will become a collection of AI generated posts nobody will read anyway because it would be much faster to ask AI directly.
And when you can effortlessly get virtual Internet points, they will lose value fast.
I asked this question directly to ChatGPT (the text of the OP's question):
Use of ChatGPT generated text for posts on Stack Overflow is temporarily banned. However, the reasons for the ban really apply to much or all of the network, and certainly for sites that are similar in nature to Stack Overflow.
I suggest (temporarily) banning ChatGPT network-wide.
It turns out that ChatGPT can give correct (or correct-looking) answers even for subjects of sites that are not at all like Stack Overflow. I asked an unanswered question from Judaism.SE and got a fairly good answer.
Its answer was:
This is a problem because it can mislead people who are looking for help, and it can also give an unfair advantage to ChatGPT users who are also answerers on Stack Overflow.
So, even ChatGPT itself agrees with the ban. Ban away!
Let's make this broader.
This would cover both ChatGPT and any AI art-based creations.
I don't think disallowing discussion about them in the sense of legal or ethical discussion should be excised entirely from the network; there are plenty of places in which experts could weigh in and help the larger Internet community navigate this whole thing.
But someone sharing their AI art here? Someone using a bot to answer a question? Get that crap off the network.
Related: I follow a ton of artists on Twitter (literally the only reason I haven't left the platform yet), and I can see that they're also worried and frustrated with the likelihood that AI art is both stealing their style and people are actively getting paid for it. While I believe that the future of AI could be bright, the fact that these models are trained on data without consent is something I morally object to, and have held an opinion on since it's come onto my radar.
I agree with this proposal. We are operating under a similar policy on the Politics Stack Exchange site. While moderating and removing answers posted using this tool, I've noticed that the ChatGPT tool is particularly good at generating content which seems eminently plausible to someone without knowledge of the topic, which in some cases has led to arguments developing in comment threads. In one case, the poster of the AI-generated content appeared to be feeding the critical comments back into ChatGPT and posting its replies.
Furthermore, if AI-generated answers were allowed, this would jeopardise the Stack Exchange reputation system, which is meant to be a decent measure of expertise and trust that the community places in a user. Users obtaining reputation through posting AI-generated content would not, in my opinion, have demonstrated that they are worthy of that trust or measure of expertise.
I'm surprised it hasn't been mentioned that Quora has a bot that asks questions that humans then get suckered into answering. It's awful. I've abandoned Quora (turned off all email notifications) in response, and I think most people who learn about this are similarly turned off.
We should explicitly ban bot-generated answers AND questions, across SE, forthwith.
Sadly, this got another major hit, probably the final blow: What is the network policy regarding AI Generated content?
In short, moderators are no longer allowed to suspend users when they suspect the posts are generated by ChatGPT, or other AI software, not even when using any of the existing detectors.
This can have only one result: ChatGPT answers are going to flood all the sites, and there's nothing to stop it anymore. This is a historical moment, the beginning of the end of Stack Exchange, which lost direction utterly and completely.
I like this, but I'm going to split hairs and take that stance that this shouldn't really be "new" policy, and that it should instead just be messaging that reinforces existing policy on this new kind of "source" for content (for lack of a better term).
Plagiarism is posting content as your own that is not your own, simply put. This is completely regardless of whether a given work's license (or AI's terms of use, in this case) allows or disallows that practice– passing off anything that is not yours as yours is plagiarism, and is not allowed.
Because our network includes many different sites that work in unique ways, I think each individual site should be free to come up with its own decision regarding whether it officially allows or bars ChatGPT as a "source" for answers. But no network site allows plagiarized content, which is what wholesale copies of AI-generated responses are.
If this messaging would help deter users, even in some minor way, from flooding sites with AI-generated answers posted as their own, or would give moderators across the network more confidence in dealing with such content, then that sounds like a completely worthwhile message to send.
I believe a permanent ban on MI generated content is necessary to preserve the quality of answers on Stack Exchange.
Perhaps SE should go further. The GPT neural net (NN) trained on data scraped from many sources including Stack Overflow and related sites. As others have noted this happened without explicit permission, attribution or compensation. I didn't consent for them to use everything I ever posted on the internet for this purpose. Perhaps SE should create an opt-in model so that human creators need to give explicit permission to use their contributions for NN training. The default legal position then becomes, "No. You can't use my creativity to train a proprietary, competing tool". Thoughts?
I'm not advocating for or against bans here -- I agree overall that there are valid reasons to ban ChatGPT answers, especially copy/paste/verbatim. But I do think some of the answers here are (and this may not be the best word), "overreacting" to certain aspects of the issue:
Position: All ChatGPT content should be banned.
Does this include grammar checks done by ChatGPT, verified by the editor, and modified as needed by the editor? I've found ChatGPT to be a good tool (when it's up) for quickly suggesting better grammar in several posts that needed improvements.
Example 1: Before and after. IIRC, I'm fairly certain I did change one word from the ChatGPT suggestion -- I felt "the issue persists" was better than "this issue persists".
Example 2: Before and after: I definitely made some small tweaks to the ChatGPT suggestion to make it slightly more concise, but I can't recall exactly what they were.
I do a lot of grammar/clarity edits (including, frequently, on my own mistakes). While I certainly could have cleaned up these two posts on my own, ChatGPT saved a lot of effort in these cases, and I believe the posts (and site) are better for it. In no way did ChatGPT suggest an answer here, or rely on any training other than proper English grammar.
Should this use be banned as well?
Counterpoint: ChatGPT can still make grammar mistakes that change the meaning of the post. If the editor doesn't carefully review the changes to assure consistent meaning, then there's a risk of a bad (and disallowed) edit.
Counterpoint-to-counterpoint: And yet, honest mistakes in editing happen even with human intervention (perhaps even more so).
I'm not saying I have the answer here, just raising the question of whether (or not) there are acceptable and positive uses of ChatGPT.
Position: Attribution issues are a nightmare, since ChatGPT can't tell us where it learned something.
I struggle with this one as well. Sure, this is a problem, but isn't it a problem for many posts by humans as well? Unless we came up with an answer purely by finding a website with the answer, we typically don't remember where we found the information originally.
Take the Stack Overflow answer to How to permanently export a variable in Linux?. At some point, the user who posted this had to learn how to do it themself, but it's unlikely they remember when or where. I certainly don't remember where I learned how to do that. Why would we expect an AI to be able to cite/attribute its source for where it learned something when most of us don't know ourselves?
The problem, of course, is when there is only a single-source of the information. Even so, if I read something in an article several weeks ago, and the topic comes up in a month in a Super User question, am I going to remember where I read it? I'll try to search the web to find it again so that I can attribute it, but most people wouldn't even go that far. But at that point, it has become part of personal my "memory" and "training", regardless of whether it is information from a single-source or available from multiple-sources.
Let me go a little deep in support of this question.
Firstly, Stack Exchange would be useless if ChatGPT was allowed network-wide. Remember that SE is a network of communities such that each person seeks to learn more from each other, not just the questioner. Even answerers get to learn by getting improvements from other users. This would be useless if AI could automatically answer all questions, with little error, and ChatGPT has seemed to achieve this goal.
I hear you saying "But we've got Google!". Well, for starters, most of my programming answers were basically searches into Google, but they mostly redirected to Stack Overflow and related webpages.
Secondly, AI is still not yet perfect. So although the range of error is small, now again here we're talking about lots of answers. That means, small translates to a (somewhat) big amount.
Thirdly, most of the issues concerning SO about AI pretty much applies to the entire network.
That's all I've got to say.
I disagree: we shouldn't blindly discriminate against AI. If ChatGPT provides a correct answer, then it shouldn't be deleted, provided it has the proper attributions. No need to waste human time if some AI can do it. However, if an SE user tends to post incorrect answers, then they should be banned.
Example of attribution that we could require users to add:
This answer was (partly) written by ChatGPT, but I have the expertise to confirm it is correct.
Notes:
Follow-ups:
