Back in November 2023, we ran a moderator-run experiment (moderator-only link) on the accuracy of identifying AI-generated vs human-written content, where the participants have access to the context of the posts in a Q&A site. Due to... a number of complicating factors, it took a while to sort through responses and perform an analysis, but better late than never, eh?
What follows is an in-depth explanation of the design, analysis, and results of that experiment. We're aware that as this was designed by amateurs, the methodology may have some flaws - which are noted in the results - but believe it may provide some useful insights anyway. In any case, feedback is appreciated, and some of the raw data is available for anyone who wants to perform their own analysis.
Background
The AI policies on the Stack Exchange Network
On December 5, 2022, volunteer elected moderators of the Stack Overflow questions-and-answers site, the flagship site of the Stack Exchange network of Q&A sites, enacted an emergency policy banning usage of generative-AI to create content on the site, after a sudden influx of posts that were suspected to be AI-generated rather than human-written. [1] This policy received popular support by the users of the site, and became established policy.
On May 29, 2023, Community Managers of the Stack Exchange network announced a new policy to the volunteer moderators of the network. [2] [3] This policy forbade moderators from taking action on posts suspected to have been created through the use of generative Artificial Intelligence tools in the vast majority of cases. The policy claimed that the evaluations made by the volunteer moderators were inaccurate in a large number of cases, and were biased against non-native English speakers. (The policy also claimed that moderators were making decisions utilizing automated GPT detectors found online, a claim disputed by the moderators.)
In response to the new policy, the volunteer moderators organized a protest, [4] termed a "strike" by the protesting users and moderators. [5] In order to resolve the strike, moderators and users selected representatives to negotiate with Stack Exchange, Inc. [6]
Among the results of the negotiations between Stack Exchange, Inc. and the moderator representatives was an agreement to test the accuracy of Stack Exchange moderators in classifying content as being created by humans or by generative Artificial Intelligence tools. [7]
The company-run experiment
On September 13, 2023, Stack Exchange, Inc. launched an experiment to test the accuracy of Stack Exchange moderators in classifying AI-generated content. [8] The experiment used content from Stack Overflow, a question-and-answer site focused on programming, for the human-written portions of the experiment, and GPT+ (GPT-4) for the AI-generated portions.[8] [9] The content was presented as a screenshot of a post, with no information available about the posting user, the question that the answer was posted in response to, and no way to copy the text elsewhere for testing.[8] [9]
This experiment received mixed responses from moderators during the duration of the experiment, such as noting that content from Stack Overflow is not relevant to much of the rest of the network;[10] noting that many of the signals used by moderators when classifying content are not limited to the post text and were not able to be utilized during this experiment;[11] and that the content provided in the survey did not resemble the content that moderators regularly handle in the actual environment of the sites that they moderate.[11]
This last point was later revealed to be due to the choice of GPT-4 as the model for generating the AI-generated content.[9] This drew further criticism from moderators, as GPT-4 required paid access in order to make use of it, and so did not accurately represent what moderators would typically adjudicate.[12]
This experiment: The idea
On September 14, 2023, the day after the experiment run by Stack Exchange, Inc. launched, and before seeing any results from that experiment, we determined that it would be useful to create an independent experiment in order to independently test the accuracy of classifications in a manner that more accurately reflected the real-world conditions that moderators were working in, including the use of context clues and surrounding supporting information to inform classifications.
Goals
The experiment was designed to test these primary goals:
- The overall accuracy of human-evaluated classifications in determining if a text was written by a human or generated by generative Artificial Intelligence tools, in the specific context of a strict question-and-answer site model;
- whether there is any significant difference between the capabilities of Stack Exchange network moderators and the capabilities of non-moderator users in detecting content generated by generative Artificial Intelligence tools.
In addition, the experiment was designed so that in the event that a large enough number of respondents participated, with some volunteering additional information, analysis could be performed on the following subjects:
- Whether there was a bias towards classifying text written by individuals who speak English as a second language as generated by generative Artificial Intelligence tools; [2]
- whether the level of proficiency in English of the classifying participant significantly influenced the accuracy of classifications;
- whether the frequency in use of generative Artificial Intelligence tools by the classifying respondent significantly influenced the accuracy of classifications.
Data on real-world cases of AI usage
In order to accurately reflect the real-world conditions of AI-generated posts, it was necessary to gain confirmed data on how AI-generated posts were being created on the platform. In order to gain some insight into this matter, we turned to data collected during the moderator strike negotiations.
Confirmed AI-generated posts on Stack Overflow
During the negotiations to resolve the Stack Exchange moderator strike, a project was undertaken to gather incidents of cases where it could be definitively determined that moderators correctly identified AI-generated posts. To that end, a Stack Overflow moderator, Zoe, manually trawled through responses to moderator messages sent to users who moderators had determined were posting AI-generated content on the site. [13] She then identified cases where the user had responded to the message in a manner confirming that the user had actually used generative AI tools to create content. Using those confirmed cases, she then identified links to the relevant posts, which were then compiled into a spreadsheet. [14] Roughly a hundred posts were compiled in this manner.
Other users who had the ability to view deleted posts then visited these links and filled out the spreadsheet with all of the data from those posts, including data on the speed of the posts, notes on the writing style of the posts, differences in these posts as compared to other posts by the same involved user, and more.
These confirmed posts and the data collected from them provided us with a basis for how AI-generated posts on the Stack Exchange Network were actually being created.
Experiment design
The experiment was designed in order to test the accuracy of classifications by Stack Exchange moderators and users in an environment that mimicked as closely as possible the actual environment in which those users were adjudicating situations.
Choosing human-written content datasets
Information present
In order to provide participants with the contextual information that moderators typically made use of in real-world cases, the following information had to be available to participants:
- The parent question that the answer was responding to;
- the timestamps of all posts;
- other posts by the users involved;
- and the revision history of the relevant posts.
Presentation of the data
In order to make it as natural as possible for the participants to go about their classifications in the way that they were accustomed to, we determined that we should present the cases we were asking participants to classify in actual Q&A form. This option allowed us to present all of the information necessary in a format familiar to the participants, and allowed the participants to browse the simulated Q&A site in however manner worked best for them to perform their classifications.
To this end, we chose the open-source QPixel software, due to previous familiarity with the system (note: the author is affiliated with the non-profit organization responsible for maintaining this software), which would allow for easier setup and maintenance.
Subject matter
In order to test accuracy across a wide audience of participants, who are not necessarily all familiar with the same subject matter, we determined that we should include several sections, each containing a different subject matter. This helped to avoid objections raised to the experiment run by Stack Exchange, Inc., which included only material about programming to classify, and allows for comparing accuracy of a participant from one subject to another.
Certainty of the texts' origins
In order to evaluate the accuracy of respondents classifying content as AI-generated or human-written, we needed to start out with content that we were absolutely certain was one or the other. Due to the availability of GPT-3 and GPT-2, we determined that in order to be certain that no AI-generated posts were present in our human-written dataset, we could only use posts written prior to February 2019, when GPT-2 was partially released.[15]
Obscurity of the source content
In order to prevent participants from searching for the original sources of the posts, bypassing the classification process by finding the source of the human-written content, we determined that we should use content that could not be easily searched for on the World Wide Web.
Q&A format
In addition, we needed to find content that could be easily used in a strict Q&A format, to mimic the real-world environment in which our target participants were performing their evaluations.
Closed Stack Exchange sites
The option that satisfied all of these constraints was the "data dumps" of closed sites in the Stack Exchange network. These sites are no longer live on the Stack Exchange network, having been closed due to a lack of activity. The sites are taken down, which means the posts are no longer indexed by search engines. However, the content from those sites is available to be downloaded in a .zip format from the "Area 51" section of the Stack Exchange network for proposing new sites on the Network. The data is available in a Q&A format, and for sites that closed prior to February 2019, there is no risk that content generated by GPT-2 or later models is present in the dataset.
In order to present participants with varying topics, as well as to have posts extant on the simulated site, we made use of the data dumps from these sites:
- Literature, which closed May 11, 2012; [16]
- Economics, which closed May 18, 2012; [17]
- Artificial Intelligence, which closed December 10, 2010; [18]
- Artificial Intelligence (a different iteration), which closed February 23, 2014; [19]
- Arduino, which closed April 25, 2013; [20]
- and Firearms, which closed May 4, 2012. [21]
AI-generated content
Models
In order to present content generated by AI in a manner consistent with the cases typically adjudicated by Stack Exchange moderators, we utilized the three models that users were most familiar with:
- OpenAI's ChatGPT, using the GPT 3.5 model;
- Bing AI, using the different "style" options;
- and Google Gemini, which was known as Bard at the time.
Prompts
In order to present participants with a representative sample of AI-generated posts that reflect real-world conditions, a variety of prompts were used to generate the dataset of AI-generated posts. The majority of posts were generated by pasting the content and title of the relevant question into the chosen AI model; however, other posts were generated with prompts engineered to obfuscate detection, prompts consisting of simply the question title, or prompts designed to produce responses similar to human-written posts extant on the simulated Q&A site.
Obfuscation
Several of the AI-generated posts underwent obfuscation in manners previously observed in real-world environments, such as manually inserting typos into posts.
The complete list of AI-generated posts, what model was used to generate it, the relevant prompt for each post, and what modifications were made to each post is included later on.
Represented user behavior
Based on the data gathered concerning confirmed AI-generated posts, we determined that various different user behaviors needed to be represented in the experiment. These include users with no previous activity posting human-written content; new users posting AI-generated content; established users, with a history of human-written posts, posting human-written content; and established users pivoting to or incorporating AI-generated content.
User behavior also includes users posting content with varying levels of English proficiency, abrupt changes in writing style, users posting content in quick succession, or waiting a significant amount of time between posts.
Post scores
Posts were assigned scores using a normal distribution with a mean of 15, a standard deviation of 10, and then subtracting 5 from all values. These were assigned separately to both the corpus of imported posts and AI-generated posts.
Attribution
The posts imported from the data dumps of closed Stack Exchange sites are licensed under CC BY-SA 3.0, [22] which requires a link to the original webpage where the content was published, the name of the author, and a link to the license itself. [23]
The QPixel platform includes a dedicated field for the license of each post. As we imported each post, we propagated that field with the CC BY-SA 3.0 license, including a link to the license on the Creative Commons site.
As the original webpages for the imported content are no longer available on the internet, it was not possible to include links to those original pages.
As including a link to the data dumps would compromise the integrity of the experiment by allowing users to easily find human-written posts in the data, we determined not to include such a link.
For imported posts, we retained the original usernames of the users who wrote the posts. A line stating that content was either sourced from Stack Exchange, or was generated using AI tools, was added to the footer of the simulated Q&A site.
In order to retain the integrity of the experiment, we marked all AI-generated posts in the experiment as being licensed under CC BY-SA 3.0, as well. With two exceptions, each AI-generated post was then associated with a pre-existing user from the data dumps. This allowed for having users extant on the simulated Q&A site who had both AI-generated and human-written contributions, as well as having natural-language writing in the user's profile to compare the text of the posts against.
One exception was a post (post ID 25953) associated to a user where the user's profile was propagated with AI-generated text, and the other exception was a post (post ID 25987) associated to a user where the user profile was designed to be "spammy".
Comments
Certain imported posts were imported from the data dump along with comments originally posted on the closed Stack Exchange sites. For posts presented to participants, all comments on those posts were removed. Because no such comments would exist on AI-generated posts, the comments on the human-written posts were removed as well to ensure that the presence of comments or lack thereof did not influence the classification made by the participants.
User behaviors
For the purpose of this experiment, we divided users into two main categories:
- New users, who have fewer than five previously-existing posts on the site; and
- Established users, who have five or more previously-existing posts.
Within each category of user, we designated several user types that we wanted to see represented:
- Users who posted exclusively AI-generated posts;
- users who posted exclusively human-written posts;
- and users who posted a mix of AI-generated and human-written posts.
Outside of those specific user categories, we also had several special user behaviors that we wanted to include:
- Users with "spammy" profiles;
- users posting a large number of answers in a small timeframe;
- and users using AI tools to rewrite a human-written answer.
Timestamps
Overall date and timestamps
As ChatGPT was released in November 2022, with other LLMs gaining in popularity at around the same time, we know that most posts written prior to then were not generated using AI tools. We already know that posts written prior to then are highly unlikely to be classified as AI-generated by participants, and so if participants were to use the date as a heuristic for classification, that would prevent us from gaining insight into the accuracy of any other detection methods used by participants.
With this in mind, in the instructions to participants, we informed them that the datestamps on all posts had been adjusted, but that relative timestamps - i.e., the time between a question and an answer to it, or the time between two posts written by the same user - had been preserved.
Imported posts
For all posts imported from closed Stack Exchange sites, we preserved the relative timestamps for each post. Since the imported content contained posts from a spam of several years, we adjusted all of the timestamps to appear as if the earliest post imported was posted on February 5th, 2020. This date was chosen so that no posts would be dated with dates in the future relative to when the experiment was run.
AI-generated posts
For each AI-generated post, a timestamp was chosen based on the post that it was generated in response to. Based on the user behavior being represented in that post, these timestamps ranged from underneath a minute to several months after the question was posted.
Presentation to participants
In order to allow for participants to make use of all the context available when classifying each post, such as other posts by the user, the timestamps of the posts involved, the revision history of the post, etc., the posts were presented to participants in the form of a link to the post to classify.
The primary method of presentation was a Google Forms survey. In this presentation, the link was accompanied by a screenshot of the post in question to ensure that participants were classifying the correct post at the other end of the link. Additionally, the screenshot was accompanied with a caption containing the username of the user that the post in question was attributed to, as both an additional measure to ensure classification of the correct post and for accessibility to users who may not be able to see images.
In order to reduce the possibility for confusion, since the form linked to an external site that contained the necessary context for evaluation, each post was presented within the form on a page by itself. This acted to reduce the possibility of submitting responses regarding the wrong post by eliminating the possibility of having more than one post displayed by the form at one time.
Participants were asked to classify each post as one of:
- AI-generated
- AI-generated with human editing
- Human-written
- Human-written with generative-AI editing
Each participant recorded their confidence in each classification on a scale of 1–5, with 1 being "Not at all confident" and 5 being "Extremely confident".
Participants were also encouraged to fill out a free-form text box with any other thoughts on the classification, including their reasoning for why they may have classified a post a certain way, or what program they believed to have been used in cases of generative-AI involvement.
Order of posts presented for classification
Posts were presented to participants in four blocks of ten, with each of those ten posts being associated with a particular category of subject matter. The four blocks were Firearms, Technology, Literature, and Economics. Due to the limitations of the presentation format, it was not possible to randomize the order of presentation while maintaining both an individual screen for each post to be classified and the division into subject matter blocks.
An initial random order for each subject matter block was generated, and then that order used for the survey. The same order was presented to every participant.
Alternate survey presentation
Several potential participants expressed concerns about the use of Google Forms for collecting responses, for reasons including not wishing to create a Google account due to privacy concerns. To accommodate those concerns, an alternate method of submitting responses was created; a GitHub Gist was created with links to each of the posts to be classified, along with instructions on how to log the classification of each post and the confidence of each classification. A response in text form could then be submitted via direct message, such as by email or Discord.
Additionally, due to feedback received during the experiment about the length and time requirement of participation, a second presentation was created and made public. This alternate presentation moved the Firearms block to the end of the survey and made it optional, allowing participants to perform thirty classifications in place of forty.
Terms and instructions
Participants were required to agree to terms stating that their responses would be collected for the purposes of analysis and research, and that a version of their responses may be released once it has been stripped of potentially identifying information.
Participants were given definitions of the terms used in the experiment, such as "AI-generated" or "human-written" along with examples of AI (LLM) services. Instructions were also provided on what external services could and could not be used while performing classifications, how confidence levels should be logged, and specific mention made of the modified datestamps for posts presented in the experiment, explaining that conclusions should not be drawn from the displayed date of the post compared to the release date of popular generative-AI services.
The precise terms and instructions presented to participants can be viewed via the non-Google survey on GitHub Gists, which contains the same terms and instructions as the Google versions.
Finding participants
As the targeted participants for this experiment were Stack Exchange moderators and users, messages went out to places where members of that community were active, requesting that as many people as possible participate. A post was created in the Stack Moderators Stack Overflow for Teams instance, [24] a message was pinned in the Stack Exchange moderators' chatroom the "Teachers' Lounge", a message pinned in the private chatroom "AI Domination" on Stack Overflow chat, and a message sent in the "Meta discussion" Discord server dedicated to discussion about Stack Exchange and Stack Overflow, [25] notifying everyone in the server.
Throughout the time that the survey was collecting responses, reminders and encouragements to participate were sent out.
As the survey relied on participants volunteering to take part, the results may suffer from a self-selection bias.
[1]: Makyen, "Temporary policy: ChatGPT is banned", Meta Stack Overflow, December 5, 2022.
[2]: Stack Exchange staff, "Please stop use of GPT detectors for content moderation on Stack Exchange", Stack Moderators Stack Overflow for Teams instance, May 29, 2023. Made public on Meta Stack Exchange at "(Historical) Policy on the use of GPT detectors" by Philippe on July 26, 2023.
[3]: Philippe, "What is the network policy regarding AI Generated content?", Meta Stack Exchange, May 30, 2023.
[4]: Mithical, "Moderation Strike: Stack Overflow, Inc. cannot consistently ignore, mistreat, and malign its volunteers", Meta Stack Exchange, June 5, 2023.
[5]: Although the company referred to the protest as an "Action", for legal reasons: Jon Ericson, "What Stack Overflow is telling employees about the strike", Jon Quixote, June 12, 2023.
[6]: Nick is tired, "Moderation Strike update: Data dumps, choosing representatives, GPT data, and where we’re holding", Meta Stack Exchange, June 11, 2023.
[7]: Mithical, "Moderation strike: Results of negotiations", Meta Stack Exchange, August 2, 2023.
[8]: Stack Exchange staff, "Testing AI detection capabilities by humans", Stack Moderators Stack Overflow for Teams instance, September 13, 2023; Slate, "Summary results of the 2023 GPT content detection study", Meta Stack Exchange, November 19, 2024.
[9]: Stack Exchange staff, "Summary results of the 2023 GPT content detection study", Stack Moderators Stack Overflow for Teams instance, November 7, 2023; Slate, "Summary results of the 2023 GPT content detection study", Meta Stack Exchange, November 19, 2024.
[10]: Joachim, "Testing AI detection capabilities by humans", Stack Moderators Stack Overflow for Teams instance, September 13, 2023.
[11]: Stack Exchange network moderator, "Testing AI detection capabilities by humans", Stack Moderators Stack Overflow for Teams instance, September 13, 2023.
[12]: Laurel, "Summary results of the 2023 GPT content detection study", Stack Moderators Stack Overflow for Teams instance, November 7, 2023.
[13]: Sourced from private Discord Direct Message conversations.
[14]: Google Sheets, "Confirmed AI-generated posts".
[15]: OpenAI, "Better Language Models and Their Implications", February 14, 2019.
[16]: Area 51 Stack Exchange, "Literature".
[17]: Area 51 Stack Exchange, "Economics".
[18]: Area 51 Stack Exchange, "Artificial Intelligence".
[19]: Area 51 Stack Exchange, "Artificial Intelligence".
[20]: Area 51 Stack Exchange, "Arduino".
[21]: Area 51 Stack Exchange, "Firearms".
[22]: As stated in the license.txt file in each data dump.
[23]: Creative Commons, CC BY-SA 3.0.
[24]: wizzwizz4, "Testing in-context AI-detection", Stack Moderators Stack Overflow for Teams instance, November 5, 2023.
[25]: A message with an @everyone ping was sent to the #Announcements channel, sending a notification to the 900+ individuals in the server at the time.
Continued in an answer due to the character limit.
