GPT on the platform: Data, actions, and outcomes

Question

In a meeting with some moderators last week, I committed to releasing the data sets from our initial studies around the efficacy and false positive rates of ChatGPT detectors to them. Tuesday afternoon, we did so. This post contains as much information from that discussion as we are able to share in public, and represents the joint efforts of my community leadership team and other members of the community team.

We will open this post with a discussion about the baseline error rates in GPT detectors. Then, we are going to discuss some data coming out of Stack Overflow that can rule out many hypotheses about root causes for contraction.

We have done our best to look through the data and understand what hypotheses could hold water. To this end, we are engineers and community managers, not scientists. We will not claim these are formal studies that produce scientific data, nor will we claim that the data sources are perfect or uncontestable. Rather, that they are operational data sufficient to produce engineering decisions: the same standard that we use as a Community Management team. We’d like to be clear: nothing in this post is intended to cast aspersions on the moderator team of any site. We respect the work that they are doing to address this problem, as challenging as it may be.

This post will be a lot to take in. And even then, it’s a fraction of the work put into analyzing this question across multiple teams in the company. However, it’s a piece of the work we believe may help clarify some facts that have heretofore been obfuscated.

The tl;dr: Summarizing the results

• The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.
• The rate of suspension for frequent answerers rose by a factor of 16 from ~0.4% to ~6.6% since the rise of GPT, and has held steady since its release.
• If every GPT poster posted exactly three answers and were suspended within three weeks, this would imply a minimum GPT post rate of 330 answers per week on Stack Overflow; in practice, we would expect a significantly greater quantity. Measurements of GPT occurrences on the platform imply fewer than 100 GPT answers per week, in disagreement with this rate, implying the existence of many false positive detections.
• The suspensions issued appear to have a significant and measurable impact on the demography and volume of answerers on the site, preferentially excluding frequent answerers.
• GPT detectors continue to be ineffective for detecting GPT answers on the platform.

Automated GPT detectors have unusable error rates on the platform

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

At this rate, we would still expect to see around 150 incorrectly-placed suspensions on Stack Overflow in the last six months. This value is still too high for comfort, and ideally, we’d see better rates than this. However, at this level of precision, conversations about how we may put such a system to practice can begin.

HuggingFace’s GPT detector assigns a “threshold” score from 0 to 1 of the post being authored by GPT. Based on a random sample of 500 answers from before GPT’s release, each answer being no less than 400 characters long, the false positive rates follow. At the 0.50 detection threshold, around 1-in-5.5 posts are falsely detected. At the 0.90 detection threshold, around 1-in-13 posts are falsely detected.

Keen readers will note that 1-in-5.5 is a fair bit better than the 1-in-3 false detection rate we originally noticed. This is because we used two detectors during the original survey: ZeroGPT and HuggingFace’s detector. The new value of 18% +/- 3.4% false positives from HuggingFace falls within the 95% CI of the original smaller survey (27% +/- 8.9%).

While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.

For a 1-in-20 false positive rate, the detector threshold needs to be 0.97. Satisfactory rates (less than 1-in-50 false detections) could not be achieved in this test until the detection threshold was set to 0.9975, but the error due to low sample size at this threshold is considerable. Narrowing this window to a more precise value would require significantly more data points than we collected in this survey. At this point, we can’t endorse usage of this service either as a tool for discriminating AI-generated posts or as a tool for validating suspicions.

---

We thought it would be helpful to include some more discussion and context. Rather a lot of discussion and context, actually. Over the last few months, folks within the company have been working to answer the question, “What has been taking place in the data coming out of Stack Overflow since GPT’s release?” Considerable changes have taken place in demography: users use Stack Overflow at different rates than they did before; different sorts of users use Stack Overflow; and questions and answers reach different outcomes.

What follows is a single piece of the puzzle. The broader picture informs the company’s operational decision-making processes. This piece has a part to play, but only a part. It is not a full explanation of the question, nor is it a blame-first investigation. Neither can it explain every change in the data on Stack Overflow that we see. However, what it reveals calls into question whether GPT detection is possible or effective on the platform, and by proxy, whether a high false positive rate may be partially responsible for a decline in answerers and answerer retention on the platform.

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release

While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for abuse prevention. If this change were causative, we would expect to see a sudden jump to a new lower level, and a return to the prior well-established rate of decrease. However, we do not see this, a strong indicator of deepening attrition. (We would also expect to see a discontinuity in other metrics not listed – this point is established by a confluence of metrics.)

The number of high-volume answerers has seen a -2.4% average week-over-week decrease since December. In total, there has been a -42% contraction in high-volume answerers since the release of GPT. While users may go to ChatGPT to ask their questions, they are obviously not going there to answer questions. Therefore we can consider that…

The total volume of questions available to frequent answerers continues to rise

The alternative hypothesis for the above chart is that the number of questions available for users to answer has simply fallen, on account of question rates falling. This claim is hard to swallow given current data. What follows is the # of available questions posted per week, divided by the number of users who post 3 or more answers in a week. If this hypothesis were true, then this value should be falling, or at least not rising as quickly, as active users are crunched by a collapsing question rate.

However, even though the question rate is collapsing, there are still plenty of available questions for users to answer. It’s hard to tell exactly what the carrying capacity is for this number, but we certainly know that it is somewhere between 3 and today’s value of ~18. In other words, if users were willing and able to answer more questions, the evidence strongly suggests that they could in fact do so. Of course, on short timescales, factors like % of questions closed and % of questions deleted could cause fluctuations in this value. However, in the long view, the trend is clear and not violated in the post-GPT region.

This leaves one question remaining: Where are they going?

7% of the people who post 3 or more answers in a week are suspended within three weeks

…and this value has held reasonably stationary since these suspensions were enacted.

In the 16 weeks before we enabled moderators to suspend users for GPT users, around 0.4% of users who posted >2 answers per week were suspended within three weeks. After we allowed GPT suspensions on first offense, 6.6% of users who posted >2 answers in a given week were suspended within three weeks, a 16-fold increase.

(Note that suspensions issued for users who posted fewer than three answers are not counted here; nor are suspensions issued three weeks after the user answered several questions.)

In a given week, 32% of the people who post 3 or more answers also did so during one of the last eight weeks. Supposing that these suspensions are distributed across users regardless of GPT usage, we should see (by rough order of magnitude) a ~2.2% decrease in the actual volume of users who post three or more answers in a given week. And indeed, real data (2.4%) are quite close to the theoretical percentage.

Instead suppose that no more than 1-in-50 of the people who were suspended for GPT usage were not actually using GPT. In order for this to be true, a large volume of users would have needed to immediately convert from being regular users to ChatGPT users; and then, a high rate of conversion would have to be sustained over time, long after the release of ChatGPT, in order to sustain present suspension rates.

Regardless of the above, no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community. Supposing every suspension is accurate, the magnitude raises serious concerns about long-term sustainability for the site.

It is worth acknowledging that we did give explicit permission to suspend on 1st offense for GPT usage. However, even in the absence of these policies, this value alone rings a deafening number of alarm bells for potential false positive detections and contributor loss alike. If there are false positive detections, even removing users’ content incorrectly could prove harmful.

Users who post 3 or more answers in a given week produce about half the answers

First, a short detour. We are going to focus on users who post 3 or more answers in a given week for much of this post. While it may seem a bit odd to look only at this segment of users (and it is, of course, not the only segment of users we investigated), there is a rationale for doing so.

Users who post answers more than twice a week used to comprise about half the content produced on Stack Overflow. However, since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly. Given the absence of question scarcity as a factor for answerers (note the above chart), the clear inference is that a large portion of frequent answerers are leaving the site, or the site is suddenly not effective at retaining new frequent answerers.

(It is worth noting that GPT messages and suspensions disproportionately skew towards users who have posted more than two answers in a week, but we can’t discuss this in more detail publicly without revealing the details of how GPT posts are detected on the platform.)

Yet, at the same time, actual GPT posts on the site have fallen continuously since release

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform. It produces a coarse estimate, and can’t be used to decide whether a given post or person is posting using GPT. However, in aggregate, it can offer us insight into the ‘true’ rate of GPT posts on the platform.

This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange. In practice, many users save few drafts routinely (for example, because some users copy and paste the answer in from a separate doc, or because they don’t stop writing until they’re ready to post), so it’s the ratio of large draft saves to small draft saves that actually lets us measure volume in practice.

This metric is sensitive to noise, but was validated against other metrics early on at the peak of the GPT answer rate. Additionally, it matches the volume of actual actions taken on the platform early on after the release of GPT.

This allows us to get a good, albeit coarse, understanding of the overall population trends in GPT answers. However, it does not allow us to identify which specific answers were drafted by or with assistance of GPT, because quite a lot of answers are ‘normally’ posted after saving very few drafts.

Between January 2023 and March 2023, it appeared that we were going to hit a floor of around 700 GPT-generated posts per week and stay there, which would have been quite a bad outcome for the site’s general health. However, as time progressed, it became clear that the actual volume of GPT answers was falling precipitously - even as a percentage of total answers (which, yes, is also falling comparably).

Some folks have asked us why this metric is capable of reporting negative numbers. The condensed answer is that the metric has noise. If the true value is zero, sometimes it will report a value higher than zero, and sometimes a value lower than zero. Since we know how much noise the metric has, we know what the largest value for GPT-suspect posts should be.

At the initial holding point, GPT answers reflected around 2.5% of the answers posted on Stack Overflow. Rates hovered around this level for several weeks, but such rates were not sustainable and the percentage of posts authored by GPT began to fall. The following chart shows the expected % of answers posted in a given week that are GPT-suspect.

These days, however, it’s clear that the rate of GPT answers on Stack Overflow is extremely small. In fact, it is difficult to estimate the true number of GPT answers posted on Stack Overflow for this reason. Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error. We can therefore say that the rate of GPT posts is far less than it was two months ago, and then it is less than it was two months before that.

So, under what conditions could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close. This also assumes every user who posts GPT answers are caught, and that GPT answerers post no more or less than three answers in a given week. In practice, the site should be seeing significantly more than 330 GPT answers per week to support this suspension rate.

It could be possible, either due to severe measurement error or due to an unexpected change in user behavior that obfuscates GPT usage using this method. But the evidence for this viewpoint does not appear strong.

Many of GPT appeals sent to the Stack Exchange support inbox could not be reasonably substantiated

While we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox, we could characterize some of them.

As a platform, we have an obligation to ensure that moderation actions taken on the Stack Exchange network are accurate and can be verified upon review if we need to do so. We need to be able to see, understand, and assess whether the actions taken were correct. It therefore needs to be said that we are very rarely, if ever, in the position where we cannot do so. In all other areas for which we receive suspension appeals, moderator actions are easily verified and double-checked the overwhelming majority of the time. It is rare and notable if we are ever in the position of overturning a moderator’s decision due to insufficient or contradictory evidence.

So, when we say that many of the GPT appeals we receive could not be substantiated on review, please keep in mind that our baseline value for this is zero, and it’s been that way for years. It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate. Rather, it is important to remember that the company has a strong need to ensure moderator actions are verifiable and justifiable. And, on this point, we need to seriously consider whether these processes, in whatever form they are taking, do what they should.

True false positive rate of moderator actions

We want to clear up a particularly important point. We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.

Under this assumption, it is impossible for us to generate a list of cases where we know moderators have made a mistake. If we were to do so, it would imply that we have a method by which we can know that the incorrect action was taken. If we could do this, we would share our methods in a heartbeat in the form of guidance to the moderator teams across the network, and then we’d carry on with things as normal. Instead, the most we can do is state that we just can’t tell. We lack the tools to verify wrongdoing on the part of a user who has been removed, messaged, or had their content deleted, and this is a serious problem.

This is a critical opportunity for us to inspect the processes that are contributing to the outlook for the site, and contextualize them in the overall state and progression of the network.

Stringing it all together

Look, the truth is, there are going to be many hypotheses about what could be taking place in the data on Stack Overflow. Heck, we’ve had several large analytics teams pore over the SO and SE data for any sorts of anomalies or possibilities when it comes to participation contraction. While it is absolutely true that Stack Overflow is losing users faster than it is gaining users, none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.

In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community. The scalar factor here varies considerably from community to community, but it must be taken into account here as well. Deleting their contributions for reasons they feel unjust, or commenting on behavior that is not present, appears to have a similar effect to being suspended, and differences are often small when it comes to long-term user outcomes (such as the user leaving, and/or potentially other people leaving with them).

Is it still possible that the proportion of false positives is small? Maybe so – it can’t be completely eliminated at this time. Direct causative data are not possible to obtain for this problem. But for that to be true, it would require some very strange user behavior en masse around answering, by users who were otherwise answering questions normally. These are behaviors we do not have an organic explanation for after months of exploration, even under the scenario where question demand contracts at a rapid pace. Again, it’s possible, but the evidence is not favorable. We are forced to look at all possible exogenous causes of user attrition - including whether logic internal to the network needs to be altered (and we have made changes here, as well).

Finally, it needs to be said that the analysis presented here is far from the complete analysis we have conducted internally. These are pieces of a larger puzzle, selected because they best express our meaning (and contain no proprietary data). Seeking a set of root causes for the contraction in the network’s community size, and an explanation for how it affects different sites/community segments around the network, has been the object of study for dozens of people, and for many months now.

We hope by now the ultimate point here is clear: Suppose we are right in this assessment and GPT removal actions are not reasonably defensible. How long can we afford to wait? To what extent can we continue to risk the network’s short-term integrity against the long-term risks of the current environment? Any good community management policy must balance the risk of action against the risk of inaction in any given situation, and the evidence does not presently favor inaction.

What we know, right now, is that the current situation is untenable. We have real, justified concerns for the survival of the network. We’re not saying this to invoke despair, or to imply that these decisions are overly rushed. There are silver linings here, and there is significant potential for future growth and a more stable long-term path. The onus is on us to find a way forward that protects the communities and network as a whole.

If there is no future for a network where mods can’t assess posts on the basis of GPT authorship (where we would be after this policy), and if there is no future for a network where mods can assess posts on the basis of GPT authorship (where we are today), then there is no future for the network at all. Yet somehow, our adjacent communities are making this work. In this deadlock, something has to give. At the end of the day, it all boils down to this: We have to walk the middle path.

Answer 1

Just some random points in no particular order, which I think could be relevant:

• You spent much time speaking about the false positives of a specific GPT detector, but usually one displays precision-recall curves that give a more complete picture (the chosen threshold is the parameter to walk along that curve). Why didn't you do it (mixing GPT generated content with pre 2023 content in a fixed ratio)? What is the recall at a desired precision?
• The issues with specifically SO are many, many years old. Now they seem to accelerate a bit but why do you think that not the same underlying forces are still mainly responsible? Activity on SO is shrinking since 2016 or earlier (if you count positively scored answers and questions for example). Maybe users simply don't like the platform anymore (after GPT as before)?
• Why the odd concentration on answerers with more than 3 answers per week? The ratio of new questions to new answers is only slowly declining. Does it mean that infrequent answerers are saving the day?
• Have you measured other properties of the draft process (amount of text changed, time between drafts saved, number of clipboard operations, ...)? Or the number of edits afterwards? Could this also be used? Do you see something there?
• How does this high/low draft saves ratio correlate with the suspensions? Did moderators indeed suspend the people you think were posting GPT messages in the beginning?
• You seem to assume that there are only very few people if at all posting GPT output on SO now. Any idea why people stopped? It may not be that novel anymore but the temptation should still exist. If people stopped then maybe because of the suspensions? But now the suspensions stopped, a reversal might be expected. Can you show how the draft save ratio developed or the number of answers since last week?
• It's not a good idea to let people post GPT content. There is no guarantee that the solutions work. Sometimes they do, often enough they don't. The quality is rather low currently. Now couple this with difficult to understand questions and what you get is only noise. More answers can actually reduce the usefulness of the content in SO if they aren't helping. What does the traffic say? Do any high quality Q&A pairs are getting generated still or does SO live exclusively from the past?
• SO excels over AI not so much for the simple questions but for the complex problems where you really need to be an expert, not just a very, very diligent but slightly stupid word cruncher who happens to be also able to speak Oxford English. So unless you want to compete with GPT (very, very challenging) you better go for the very high quality niche.
• And finally, I was here for the library of programming knowledge. I saw the data dumps as an insurance. You took that away and I don't trust it will come back. For me it's the last straw. I don't feel obligated to stay either. And I stayed for a long time and probably posted more than I should have and wasted more time than I should have but I also enjoyed reading contributions from so many nice persons, that helped me and always made me thinking. I'm very thankful for that.

Answer 2

It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

This raised a question for me: Are the people who review moderator actions people who know the fields of the SE sites they take place in?

If so, then that's a valid concern. I don't know if you notify those moderators, or if they then respond to you. And I understand there's limited bandwidth for going back and forth over each case, but I am curious if you've been able to explore the disconnect.

But if the reviewers are generalists, and don't know the fields, then I wonder if there's a gap here. Because ChatGPT output is tuned to appear valid - they have worn away most of the "tells". I would not be surprised if they can't verify the GPT identification - there would be an entirely new set of skills required to do so, and people would need training to gain them.

I should mention that I'm mostly active in math.se, and we frequently find ourselves completely unqualified to judge the quality of this or that question. We had one poster that dropped like eight answers in less than an hour. I was able to find one with an out-and-out error that you could find if you paid close attention, one that completely missed the point of the question but was oddly long-winded, but most of the others, had I seen them in isolation and didn't have other users spotting them, I wouldn't have been able to reliably verify a claim that they were ChatGPT.

Discerning ChatGPT is hard. I can't say much about the behemoth that Stack Overflow is, where you have to resort to stats-based speculation, but at least in the SE sites where I hang out, I can't see how you're offering an improvement on the moderators' judgements that we currently rely on.

Answer 3

All statistics and chances of chances of false positives aside...

I find it very hard to believe that moderators of any site in the network would suspend a user for posting high quality, non-plagiarized content for the benefit of others.

The kind of users suspended will extremely likely be the ones who post low quality content, AI-generated or not.

Having come to that conclusion, then is it really such a big deal if moderators accidentally suspend a user accusing them for posing AI-generated content, while they had in fact written their low-quality, possibly incorrect, of no-value-to-anyone content all by themselves? Or perhaps they had just blatantly plagiarized it without the aid of AI.

Even if such suspensions would occasionally happen, I'd say that's perfectly acceptable collateral damage to keep the content quality up. 1 week suspension is not the end of the world. Concerned users can protect themselves from this by writing better answers. Ideally with actual references to sources, which you won't find in AI-generated or plagiarized answers.

And here comes the quantity vs quality discussion. Those users posting low quality content do generate site traffic. Who cares about quality of the product over time when we can have short-term profits!

Answer 4

Aside from the post authorship question, let's turn our focus on user behavior, in hopes that it might explain some anomalies.

Here are some motivation factors that strongly influence people's behaviour on this platform:

• This platform is addictive, as it was designed to be so. For this platform to work well, questions need relatively rapid interaction, and relatively rapidly produced answers. This can be facilitated by making the platform addictive, so that people keep hanging around, and remain available for interaction.

  How do addicts behave?

  First of all, they may spend more of their time with these sites than they reasonably intend. Accordingly:

  • You will see behavior of frequent engagement, aimed at soothing their urges for whatever they gain from participation.

  • On the other end of this tension, you will regularly see people attempting to regain control over their time, and thus, life, breaking away from the platform, and spending their time elsewehre.

    Then, after a few months of hiatus, they may return again.

• The reputation system can impose a significant frustration, even if it's well-intended for the purpose of fostering and facilitating disciplined and responsible behavior.

  Due to the limited privileges associated with low rep, from their first day on this platform, new users are chasing after the value that they invested time-wise, to somehow get to the point where they can participate with dignity. That, for me, comes at a minimum of 1000 points, which unlocks the ability to see up-/downvotes breakdowns on posts.

  Anything below 1000 points is a painful place to be in, and thus people will be urged to earn those points as fast as possible.

• Stack Exchange is well known in professional contexts, and a highly valued (high rep) user account is something that can boost people's CVs.

  This will bring in people who hope to quickly and cheaply build such a boost for their CVs.

Each of these motivations can be fulfilled by posting ChatGPT content, and I offer that these are major motivators for people to do so.

You can factor these insights into building assumptions about what is really happening, and then you can attempt to verify those new, inconvenient-but-honest assumptions through new rounds of data analysis.

Answer 5

I just want to mention the existence of a paper on the arXiv (pdf): Guo et al., How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection, Jan 2023.

TL;DR: This paper tested humans detecting ChatGPT content, and found an accuracy ranging from 34% to 98%, depending on experimental conditions (not to be confused with the false positive rate). They also ask humans whether a ChatGPT-written answer or a human answer is more "helpful", and found ChatGPT's answers were more helpful 23% to 72% of the time, depending on experimental conditions. One dataset used in the experiment partly used finance-related Stack Exchange Q&A data. Note this was uploaded to the arXiv in January 2023.

---

From the results in this paper, I want to highlight Table 2 (screenshot); the top part is reproduced below (the bottom part is similar but for the Chinese language):

	Pair-expert	Single-expert	Single-amateur	Helpfulness
All	0.90	0.81	0.48	0.57
reddit_eli5	0.97	0.94	0.57	0.59
open_qa	0.98	0.78	0.34	0.72
wiki_csai	0.97	0.61	0.39	0.71
medical	0.97	0.97	0.50	0.23
finance	0.79	0.73	0.58	0.60

Rephrasing the paper to be more succinct, the columns labels are:

Pair-expert: The "expert" tester needs to determine which answer (one from humans and another from ChatGPT) is generated by ChatGPT.
Single-expert: The "expert" tester needs to determine whether the answer (a single answer randomly given by humans or ChatGPT) is generated by ChatGPT.
Single-amateur: The "amateur" tester needs to determine whether the answer (a single answer randomly given by humans or ChatGPT) is generated by ChatGPT.

And the fourth column is distinct:

Helpfulness: Each tester is asked to pretend that the question is proposed by him/herself, and needs to determine which answer (one from human and another from ChatGPT) is more helpful to him/her.

The participants are...

... 8 experts (who are frequent users of ChatGPT) and 9 amateurs (who have never heard of ChatGPT).

Importantly, they're "experts" in ChatGPT (in the sense that they're familiar with using ChatGPT), not subject-matter experts.

They're all asked a series of 30 questions for each of five different datasets (the rows in the table above): (a) Reddit's Explain it like I'm 5, (b) WikiQA, (c) sourced from computer-science topics on Wikipedia, (d) Medical Dialog, and (e) finance-related Q&A "built by crawling Stackexchange, Reddit and StockTwits". So I would say (b) and (e) are the most relevant, especially (e) since it partly uses Stack Exchange data.

The authors didn't explicitly say this in their paper, but I expect the first three columns in the table list accuracy rates, i.e., the proportion correctly identified as ChatGPT (note that this is not equal to 1-FPR, where FPR is the false-positive rate). I also note that this paper was published in January, and ChatGPT has improved considerably since then. I also point out the possibility that ChatGPT may have encountered the questions it was asked for this research previously during training.

Answer 6

Why, instead of making stuff up about how confident our detection methods are, (human or otherwise) did we not test them? SE already has an incredibly large library of high quality QandA content from before November 2022. A simple test, have ChatGPT answer questions and take thousands of human generated answers where we know with certainty how it was created and test our detection methods.

Even if the detection methods are 1 in 20 false positives the point would still be moot. If a user wanted a generated answer, they could ostensibly use ChatGPT themselves for free. No, instead they came to the SE network for high quality answers from real people that are actually accurate.

Finally, what does SE have to gain by allowing GPT content? So far, every message to us has been about how it is essentially "not bad" for the site. Really? We're shooting for "not bad"? What incentives are there, as from my perspective, there are none, other than reduced work for the moderators who already work so hard for this community.

Answer 7

False assumption: LLM outputs are distinguishable from human content

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

Although you argue your point, what evidence do we have that such a rate is even possible?

GPT and similar LLM models are designed to mimic natural human language, and although there are detectable patterns, the fundamental reason those patterns exist is because they are probable pathways of human communication.

If one in fifty human written posts did not resemble the word choice, ordering and structure of a modern LLM model, that would be surprising. In real life situations we will always have non-LLM posts that resemble LLM posts, and that's to be expected.

Hopefully detection technology will improve - but so will GPT and similar LLM models improve in generating more natural written text. Setting our baseline target to be so radically high that it's unachievable today virtually guarantees it will be less achievable in the future. We need to be prepared for what the problem is likely to resemble in 2024 or 2025, not just what's happening today.

Recommendation: Reduce the target false positive rate

I would suggest that a 1-in-10 false positive rate would be better, but still may prove too high in the long run.

Perhaps this seems untenable to SE from the prospects of incorrectly placed suspensions, but perhaps we need a more blended approach to the problem:

1. Set a clear threshold for how many posts a user must have posted to make the result valid
2. Set clear expectations for which detectors should be used and how high each detection result must be to count as valid

And ideally these types of tools should be automated and then supplied to moderators to inform their decisions.