GPT on the platform: Data, actions, and outcomes

Question

In a meeting with some moderators last week, I committed to releasing the data sets from our initial studies around the efficacy and false positive rates of ChatGPT detectors to them. Tuesday afternoon, we did so. This post contains as much information from that discussion as we are able to share in public, and represents the joint efforts of my community leadership team and other members of the community team.

We will open this post with a discussion about the baseline error rates in GPT detectors. Then, we are going to discuss some data coming out of Stack Overflow that can rule out many hypotheses about root causes for contraction.

We have done our best to look through the data and understand what hypotheses could hold water. To this end, we are engineers and community managers, not scientists. We will not claim these are formal studies that produce scientific data, nor will we claim that the data sources are perfect or uncontestable. Rather, that they are operational data sufficient to produce engineering decisions: the same standard that we use as a Community Management team. We’d like to be clear: nothing in this post is intended to cast aspersions on the moderator team of any site. We respect the work that they are doing to address this problem, as challenging as it may be.

This post will be a lot to take in. And even then, it’s a fraction of the work put into analyzing this question across multiple teams in the company. However, it’s a piece of the work we believe may help clarify some facts that have heretofore been obfuscated.

The tl;dr: Summarizing the results

• The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.
• The rate of suspension for frequent answerers rose by a factor of 16 from ~0.4% to ~6.6% since the rise of GPT, and has held steady since its release.
• If every GPT poster posted exactly three answers and were suspended within three weeks, this would imply a minimum GPT post rate of 330 answers per week on Stack Overflow; in practice, we would expect a significantly greater quantity. Measurements of GPT occurrences on the platform imply fewer than 100 GPT answers per week, in disagreement with this rate, implying the existence of many false positive detections.
• The suspensions issued appear to have a significant and measurable impact on the demography and volume of answerers on the site, preferentially excluding frequent answerers.
• GPT detectors continue to be ineffective for detecting GPT answers on the platform.

Automated GPT detectors have unusable error rates on the platform

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

At this rate, we would still expect to see around 150 incorrectly-placed suspensions on Stack Overflow in the last six months. This value is still too high for comfort, and ideally, we’d see better rates than this. However, at this level of precision, conversations about how we may put such a system to practice can begin.

HuggingFace’s GPT detector assigns a “threshold” score from 0 to 1 of the post being authored by GPT. Based on a random sample of 500 answers from before GPT’s release, each answer being no less than 400 characters long, the false positive rates follow. At the 0.50 detection threshold, around 1-in-5.5 posts are falsely detected. At the 0.90 detection threshold, around 1-in-13 posts are falsely detected.

Keen readers will note that 1-in-5.5 is a fair bit better than the 1-in-3 false detection rate we originally noticed. This is because we used two detectors during the original survey: ZeroGPT and HuggingFace’s detector. The new value of 18% +/- 3.4% false positives from HuggingFace falls within the 95% CI of the original smaller survey (27% +/- 8.9%).

While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.

For a 1-in-20 false positive rate, the detector threshold needs to be 0.97. Satisfactory rates (less than 1-in-50 false detections) could not be achieved in this test until the detection threshold was set to 0.9975, but the error due to low sample size at this threshold is considerable. Narrowing this window to a more precise value would require significantly more data points than we collected in this survey. At this point, we can’t endorse usage of this service either as a tool for discriminating AI-generated posts or as a tool for validating suspicions.

---

We thought it would be helpful to include some more discussion and context. Rather a lot of discussion and context, actually. Over the last few months, folks within the company have been working to answer the question, “What has been taking place in the data coming out of Stack Overflow since GPT’s release?” Considerable changes have taken place in demography: users use Stack Overflow at different rates than they did before; different sorts of users use Stack Overflow; and questions and answers reach different outcomes.

What follows is a single piece of the puzzle. The broader picture informs the company’s operational decision-making processes. This piece has a part to play, but only a part. It is not a full explanation of the question, nor is it a blame-first investigation. Neither can it explain every change in the data on Stack Overflow that we see. However, what it reveals calls into question whether GPT detection is possible or effective on the platform, and by proxy, whether a high false positive rate may be partially responsible for a decline in answerers and answerer retention on the platform.

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release

While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for abuse prevention. If this change were causative, we would expect to see a sudden jump to a new lower level, and a return to the prior well-established rate of decrease. However, we do not see this, a strong indicator of deepening attrition. (We would also expect to see a discontinuity in other metrics not listed – this point is established by a confluence of metrics.)

The number of high-volume answerers has seen a -2.4% average week-over-week decrease since December. In total, there has been a -42% contraction in high-volume answerers since the release of GPT. While users may go to ChatGPT to ask their questions, they are obviously not going there to answer questions. Therefore we can consider that…

The total volume of questions available to frequent answerers continues to rise

The alternative hypothesis for the above chart is that the number of questions available for users to answer has simply fallen, on account of question rates falling. This claim is hard to swallow given current data. What follows is the # of available questions posted per week, divided by the number of users who post 3 or more answers in a week. If this hypothesis were true, then this value should be falling, or at least not rising as quickly, as active users are crunched by a collapsing question rate.

However, even though the question rate is collapsing, there are still plenty of available questions for users to answer. It’s hard to tell exactly what the carrying capacity is for this number, but we certainly know that it is somewhere between 3 and today’s value of ~18. In other words, if users were willing and able to answer more questions, the evidence strongly suggests that they could in fact do so. Of course, on short timescales, factors like % of questions closed and % of questions deleted could cause fluctuations in this value. However, in the long view, the trend is clear and not violated in the post-GPT region.

This leaves one question remaining: Where are they going?

7% of the people who post 3 or more answers in a week are suspended within three weeks

…and this value has held reasonably stationary since these suspensions were enacted.

In the 16 weeks before we enabled moderators to suspend users for GPT users, around 0.4% of users who posted >2 answers per week were suspended within three weeks. After we allowed GPT suspensions on first offense, 6.6% of users who posted >2 answers in a given week were suspended within three weeks, a 16-fold increase.

(Note that suspensions issued for users who posted fewer than three answers are not counted here; nor are suspensions issued three weeks after the user answered several questions.)

In a given week, 32% of the people who post 3 or more answers also did so during one of the last eight weeks. Supposing that these suspensions are distributed across users regardless of GPT usage, we should see (by rough order of magnitude) a ~2.2% decrease in the actual volume of users who post three or more answers in a given week. And indeed, real data (2.4%) are quite close to the theoretical percentage.

Instead suppose that no more than 1-in-50 of the people who were suspended for GPT usage were not actually using GPT. In order for this to be true, a large volume of users would have needed to immediately convert from being regular users to ChatGPT users; and then, a high rate of conversion would have to be sustained over time, long after the release of ChatGPT, in order to sustain present suspension rates.

Regardless of the above, no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community. Supposing every suspension is accurate, the magnitude raises serious concerns about long-term sustainability for the site.

It is worth acknowledging that we did give explicit permission to suspend on 1st offense for GPT usage. However, even in the absence of these policies, this value alone rings a deafening number of alarm bells for potential false positive detections and contributor loss alike. If there are false positive detections, even removing users’ content incorrectly could prove harmful.

Users who post 3 or more answers in a given week produce about half the answers

First, a short detour. We are going to focus on users who post 3 or more answers in a given week for much of this post. While it may seem a bit odd to look only at this segment of users (and it is, of course, not the only segment of users we investigated), there is a rationale for doing so.

Users who post answers more than twice a week used to comprise about half the content produced on Stack Overflow. However, since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly. Given the absence of question scarcity as a factor for answerers (note the above chart), the clear inference is that a large portion of frequent answerers are leaving the site, or the site is suddenly not effective at retaining new frequent answerers.

(It is worth noting that GPT messages and suspensions disproportionately skew towards users who have posted more than two answers in a week, but we can’t discuss this in more detail publicly without revealing the details of how GPT posts are detected on the platform.)

Yet, at the same time, actual GPT posts on the site have fallen continuously since release

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform. It produces a coarse estimate, and can’t be used to decide whether a given post or person is posting using GPT. However, in aggregate, it can offer us insight into the ‘true’ rate of GPT posts on the platform.

This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange. In practice, many users save few drafts routinely (for example, because some users copy and paste the answer in from a separate doc, or because they don’t stop writing until they’re ready to post), so it’s the ratio of large draft saves to small draft saves that actually lets us measure volume in practice.

This metric is sensitive to noise, but was validated against other metrics early on at the peak of the GPT answer rate. Additionally, it matches the volume of actual actions taken on the platform early on after the release of GPT.

This allows us to get a good, albeit coarse, understanding of the overall population trends in GPT answers. However, it does not allow us to identify which specific answers were drafted by or with assistance of GPT, because quite a lot of answers are ‘normally’ posted after saving very few drafts.

Between January 2023 and March 2023, it appeared that we were going to hit a floor of around 700 GPT-generated posts per week and stay there, which would have been quite a bad outcome for the site’s general health. However, as time progressed, it became clear that the actual volume of GPT answers was falling precipitously - even as a percentage of total answers (which, yes, is also falling comparably).

Some folks have asked us why this metric is capable of reporting negative numbers. The condensed answer is that the metric has noise. If the true value is zero, sometimes it will report a value higher than zero, and sometimes a value lower than zero. Since we know how much noise the metric has, we know what the largest value for GPT-suspect posts should be.

At the initial holding point, GPT answers reflected around 2.5% of the answers posted on Stack Overflow. Rates hovered around this level for several weeks, but such rates were not sustainable and the percentage of posts authored by GPT began to fall. The following chart shows the expected % of answers posted in a given week that are GPT-suspect.

These days, however, it’s clear that the rate of GPT answers on Stack Overflow is extremely small. In fact, it is difficult to estimate the true number of GPT answers posted on Stack Overflow for this reason. Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error. We can therefore say that the rate of GPT posts is far less than it was two months ago, and then it is less than it was two months before that.

So, under what conditions could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close. This also assumes every user who posts GPT answers are caught, and that GPT answerers post no more or less than three answers in a given week. In practice, the site should be seeing significantly more than 330 GPT answers per week to support this suspension rate.

It could be possible, either due to severe measurement error or due to an unexpected change in user behavior that obfuscates GPT usage using this method. But the evidence for this viewpoint does not appear strong.

Many of GPT appeals sent to the Stack Exchange support inbox could not be reasonably substantiated

While we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox, we could characterize some of them.

As a platform, we have an obligation to ensure that moderation actions taken on the Stack Exchange network are accurate and can be verified upon review if we need to do so. We need to be able to see, understand, and assess whether the actions taken were correct. It therefore needs to be said that we are very rarely, if ever, in the position where we cannot do so. In all other areas for which we receive suspension appeals, moderator actions are easily verified and double-checked the overwhelming majority of the time. It is rare and notable if we are ever in the position of overturning a moderator’s decision due to insufficient or contradictory evidence.

So, when we say that many of the GPT appeals we receive could not be substantiated on review, please keep in mind that our baseline value for this is zero, and it’s been that way for years. It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate. Rather, it is important to remember that the company has a strong need to ensure moderator actions are verifiable and justifiable. And, on this point, we need to seriously consider whether these processes, in whatever form they are taking, do what they should.

True false positive rate of moderator actions

We want to clear up a particularly important point. We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.

Under this assumption, it is impossible for us to generate a list of cases where we know moderators have made a mistake. If we were to do so, it would imply that we have a method by which we can know that the incorrect action was taken. If we could do this, we would share our methods in a heartbeat in the form of guidance to the moderator teams across the network, and then we’d carry on with things as normal. Instead, the most we can do is state that we just can’t tell. We lack the tools to verify wrongdoing on the part of a user who has been removed, messaged, or had their content deleted, and this is a serious problem.

This is a critical opportunity for us to inspect the processes that are contributing to the outlook for the site, and contextualize them in the overall state and progression of the network.

Stringing it all together

Look, the truth is, there are going to be many hypotheses about what could be taking place in the data on Stack Overflow. Heck, we’ve had several large analytics teams pore over the SO and SE data for any sorts of anomalies or possibilities when it comes to participation contraction. While it is absolutely true that Stack Overflow is losing users faster than it is gaining users, none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.

In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community. The scalar factor here varies considerably from community to community, but it must be taken into account here as well. Deleting their contributions for reasons they feel unjust, or commenting on behavior that is not present, appears to have a similar effect to being suspended, and differences are often small when it comes to long-term user outcomes (such as the user leaving, and/or potentially other people leaving with them).

Is it still possible that the proportion of false positives is small? Maybe so – it can’t be completely eliminated at this time. Direct causative data are not possible to obtain for this problem. But for that to be true, it would require some very strange user behavior en masse around answering, by users who were otherwise answering questions normally. These are behaviors we do not have an organic explanation for after months of exploration, even under the scenario where question demand contracts at a rapid pace. Again, it’s possible, but the evidence is not favorable. We are forced to look at all possible exogenous causes of user attrition - including whether logic internal to the network needs to be altered (and we have made changes here, as well).

Finally, it needs to be said that the analysis presented here is far from the complete analysis we have conducted internally. These are pieces of a larger puzzle, selected because they best express our meaning (and contain no proprietary data). Seeking a set of root causes for the contraction in the network’s community size, and an explanation for how it affects different sites/community segments around the network, has been the object of study for dozens of people, and for many months now.

We hope by now the ultimate point here is clear: Suppose we are right in this assessment and GPT removal actions are not reasonably defensible. How long can we afford to wait? To what extent can we continue to risk the network’s short-term integrity against the long-term risks of the current environment? Any good community management policy must balance the risk of action against the risk of inaction in any given situation, and the evidence does not presently favor inaction.

What we know, right now, is that the current situation is untenable. We have real, justified concerns for the survival of the network. We’re not saying this to invoke despair, or to imply that these decisions are overly rushed. There are silver linings here, and there is significant potential for future growth and a more stable long-term path. The onus is on us to find a way forward that protects the communities and network as a whole.

If there is no future for a network where mods can’t assess posts on the basis of GPT authorship (where we would be after this policy), and if there is no future for a network where mods can assess posts on the basis of GPT authorship (where we are today), then there is no future for the network at all. Yet somehow, our adjacent communities are making this work. In this deadlock, something has to give. At the end of the day, it all boils down to this: We have to walk the middle path.

Answer 1

What makes you so confident that your methods of identifying GPT posts are more accurate than how moderators have been identifying them?

So far, all of the data I've seen hinges on the assumption that your methods of detecting GPT posts - i.e. the "gold standard" - are inherently a better method of detection than the heuristics that moderators have been using, which involve evaluating the writing style, the frequency of posts, and the context of the user, among other methods. However, I have not seen any proof of this assumption.

Moderators have offered to go through blind testing, with known-good and known-bad data being used in order to see if mods can actually identify GPT posts. Moderators have asked you to identify cases where suspensions were issued in error.

We have yet to see any examples of cases where GPT suspensions were issued in error, with a very small number of cases being mentioned as "unverifiable appeals", and Stack has declined to test the moderators' abilities in this manner.

There has been no coordination or communication with the people issuing these suspensions and deleting content before this policy change was pushed through in the worst way possible. Nobody reached out to mention concerns of too many false-positives; nobody asked moderators to cut back on suspending for first-offences; and nobody discussed changing policy before the change was announced.
Moderators were given no opportunity to explain that the detectors are known to be inaccurate and are not relied on in moderation, or to compare identification methodology.

These are some severe gaps in the data and how it was gathered, and I cannot trust in its accuracy without some sort of proof being presented.

Answer 2

Thank you for posting the methodology! That's a step forward.

Unfortunately, there's a critical flaw in your methodology: you assume that

In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange.

If, over time, the rate at which GPT posters save drafts changed, your analysis breaks down. And since the number of drafts is the only argument you're giving, this brings the conclusion into doubt.

This would be a reasonable assumption for isolated posters who are not going to invest any effort in bypassing detection. But if a significant fraction of GPT posters are organized, it is not a reasonable assumption: you can expect them to evolve to counter your countermeasures.

I know from experience that you sometimes get a captcha if you paste an answer too quickly after loading a page. I'd expect automated posters to want to bypass that captcha, either by using an automated captcha solver or by arranging not to post too quickly. I have no a priori knowledge of which.

---

So I googled “how to post on stackoverflow with gpt”, and looked for relevant results dating from after the initial rush in December 2022. The first relevant result starting from February 2023, on page 2 of the results, is a Chrome extension, released on 28 February 2023. Going by the demo video, this extension posts the answer a few characters at a time, at the rate a human would type. Someone using this extension would generate a similar number of drafts as a human.

The existence of such a tool invalidates your metric. And since it's your only metric, it invalidates your conclusion.

I am much more inclined to trust moderators to remove bad content than statistics that do good math on irrelevant data.

Answer 3

Appeals of moderator actions that cannot be validated need to be shown to the moderator team

When taking action against a user, we need to have strong evidence that we are correct. That evidence should be, whenever possible, documented in a form that allows a second person to double check that the action was correct.

When reviewing appeals, the standard should not be whether a moderator action can be proven wrong. We hold ourselves to a higher standard to that. If the reviewer can't find sufficient evidence to demonstrate the action to be correct, then we should be contacted to supply any additional details and the results of our original investigation. If the reviewer then feels the action hasn't been demonstrated to be correct, then the action should be reversed.

I can only find 3 instances when the Stack Overflow moderator team was contacted by Community Managers about appeals related to ChatGPT/AI suspensions. Of those:

• One poster admitted that they had used ChatGPT, but wanted some answers undeleted that they had written themselves and posted among the GPT answers. (To be clear, those posts were recognized as non-GPT but still deleted for another reason. The Community Manager had been informed in detail about that, and did not object.)
• One set of posts was affirmatively confirmed to be ChatGPT to the satisfaction of the CM.
• One set of posts was not specifically confirmed to be AI-generated, but a CM agreed that it was unlikely that this user was writing their answers specifically for each question they answered. That suggests that there were, at the very least, quality issues (the handling moderator noted that there was a lot of dupe answering).

However, your post strongly implies that there are many appeals that we have not seen. More specific numbers were discussed internally, leading us to believe that we have not seen the vast majority of appeals that could not be validated. We need to know if we're making mistakes so that we can figure out how it happened and prevent it from happening again. And if we're not making mistakes, then perhaps we need to document our work better. Either way, we'll have more information on possible root causes for the problem you're trying to solve.

Please share with the moderator team these and any future appeals that cannot be validated (regardless of the reason for the action).

Answer 4

^{_{Notes about my following SEDE queries: Upon new SEDE refresh, you may have to run the cross-site queries a few times for them to finally complete). Ignore the datapoints for the current month (cropped out of the screenshots). That data is not yet complete. Ex. Things like roomba have "lag".}}

---

The alternative hypothesis for the above chart is that the number of questions available for users to answer has simply fallen, on account of question rates falling. This claim is hard to swallow given current data.

The total volume of questions available to frequent answerers continues to rise

You're kidding me, right?

Your assertion that the number of available questions is rising is completely contrary to reality and using a warped ^_(IMO) view of the data. Here's a network-wide query of new contributions per month since 2018.

Here's a site-specific version of the query if you're interested.

Just in case it's not obvious enough, question influx rate is dropping. A lot. And so is answer influx rate.

Even if frequent-answerers are leaving as you claim (more on that later), all else being constant, that has no effect on the number of available answers to other answerers. If there are N new questions per day, and one answerer leaves, there are still N new questions per day. Each of those N questions are still available to each answerer. There will likely be more unanswered questions per day, but that's not your argument, and even if it were, that's a pretty vapid statement. And all else is not constant: Question influx rate is declining rapidly.

1. Did you forget that in your new Code of Conduct, you state:

   Broadly speaking, we do not allow and may remove misleading information that:

   • Misrepresents, denies, or makes unsubstantiated claims about historical or newsworthy events or the people involved in them

2. Why aren't we instead talking about the problem of loss of askers and questions? There are no answers without questions. Losing questions is logically a more root problem. Why not give attention to the more root problem?

---

The way I see it, answer rates are dropping as a result of question rates dropping - not because of the wrongful suspensions that you are trying so hard to believe are happening.

I have two reasons to believe that answers are dropping because of questions dropping:

1. Some pretty simple logic: Again, there can be no answers if there are no questions. Answers are to questions, and answerability is largely dependent on question characteristics.

2. That's certainly what the consistent proportion of new answers to new questions per month suggests to me.

Though there was a (not at all surprising) uptick in new deleted answers per new (deleted or not) question last December with the ChatGPT release and ban policy on SO, the proportion of new (non-deleted) answers to new questions (deleted or not) is hovering fairly constantly around 0.74 and has not experienced any dramatic change in the several years before, and since the release of ChatGPT. It has been fluctuating smoothly with a gradual decline from an average of ~0.8 in 2020. In that sense, from where I stand, I think you are missing the forest for the trees. Here's a network-wide query of average new answers per new question per month since 2018

Notice how the trend for non-deleted answers is smooth from Nov 2022 to Dec 2022 to Jan 2023. Here's a site-specific version of the query if you're interested.

The smoothness of the monthly new non-deleted answers in proportion to new questions and the noticeable jump in deleted ones leads me to believe that - largely speaking ^{_{(I did see some high-rep users violate the policy)}} - we didn't significantly lose our consistently-answer-contributing userbase, and instead, just rapidly gained a bunch of gold reputation prospectors trying to get rich quick and easy with the fun new toy without any regard for possible damage done to the ecosystem, and we were able (thanks to mods and flaggers, and CMs who cooperated with mods in blessing the ban policy) to keep the current and long-term damage under control in protection of the Stack Exchange network's reputation as a trusted source of information.

Let me add to that my own experience: My rate of answering on SO has been in decline recently, and it's not because I'm spending less time or effort looking for things to answer. There was a period earlier this year when I was averaging ~10 answers per day. Now it's more like ~3. It's largely because I've been getting fewer questions in my tags. A few months ago, I used to wake up to over a page of new questions, and now I often wake up to less than a page.

I'm very much inclined to believe that you're pointing your fingers at the wrong place and the wrong people.

---

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release

In your obsession with answerers, you are missing that according to your methodology and logic, askers seem to be "leaving" even faster. Continuing on your weird focus on answers instead of questions, here's a SEDE query I made of SO graphing number of contributors who asked three or more questions or wrote three or more answers in a week since October 2011 (basically extending your graph to actually show the historical trend and put question-askers in the picture as well):

Here's a SO query for the range from 2016-01-01 to 2022-11-30, and here's a SO query for the range from 2022-12-01 to today (2023-06-12 at the time of this writing). Here's the Google Sheet where I got the fit lines.

	2016-01-01 to 2022-11-30	2022-12-01 to 2023-06-12	change factor
answerers	-1.143154562	-4.850514565	4.243096014
askers	-0.3719258563	-1.875632304	5.043027454

I.e. using your approach and logic, your "dedicated asker-base" has been "leaving" 1.188525416 times faster than your "dedicated answerer-base". How did you not notice that? Were you too busy trying to find a scapegoat and look good to whoever you need to impress that isn't us? You're making much ado about a smaller problem and ignoring a bigger one, and destroying your relationship with the more dedicated part of your userbase in the process...

I see no data to support that that problem is any fault of the moderators or curators in enacting ChatGPT policies. If anything, from what I've been reading on Reddit, people are going to ChatGPT to ask their questions instead of Stack Overflow.

---

In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

You've skipped an important step (and it's why I've been putting "leaving" in double-quotes). What does writing fewer answers have to do with leaving the site? Again, there are fewer questions. I'm still here. Again, I'm writing fewer answers largely because there are fewer questions.

---

If you're concerned about traffic dropping with the rise of ChatGPT, I just can't understand why the first thing you'd think to do is basically allow ChatGPT answers instead of looking at things like improving your platform and making it more usable, improving user experience / guidance / onboarding, fixing bugs, and looking at highly-requested features.

When I read peoples' comments on Reddit about why they're leaving, it's often a case of What about the community is "toxic" to new users?. See Mithical's answer to that question- it's why I bring up improving user guidance and onboarding. In fact, you have a project to work on that: Introducing new user onboarding project. What happened to that project?

Like - c'mon. Stick to your guns. Stack Exchange succeeded because its whole approach to being a Q&A platform was valuable, and I'm convinced that it can and will continue being valuable without allowing LLM-generated answers.

---

7% of the people who post 3 or more answers in a week are suspended within three weeks

I don't see 7% as significant compared to the rough order of a ~25% decline in overall post contributions comparing November 2022 to May 2023. Again, I'm so confused why we're all sitting here talking about this mouse when there's a whole elephant in the same room.

---

no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community.

:/ Just 6 months ago you were quite supportive of SO's ChatGPT policy, for reasons of community and platform health.

You wrote:

https://stackoverflow.com/help/answering-limit

We slow down new user contributions in order to ensure the integrity of the site and that users take the time they need to craft a good answer.

https://stackoverflow.com/help/gpt-policy

Stack Overflow is a community built upon trust. The community trusts that users are submitting answers that reflect what they actually know to be accurate and that they and their peers have the knowledge and skill set to verify and validate those answers. The system relies on users to verify and validate contributions by other users with the tools we offer, including responsible use of upvotes and downvotes. Currently, contributions generated by GPT most often do not meet these standards and therefore are not contributing to a trustworthy environment. This trust is broken when users copy and paste information into answers without validating that the answer provided by GPT is correct, ensuring that the sources used in the answer are properly cited (a service GPT does not provide), and verifying that the answer provided by GPT clearly and concisely answers the question asked.

[...] In order for Stack Overflow to maintain a strong standard as a reliable source for correct and verified information, such answers must be edited or replaced. However, because GPT is good enough to convince users of the site that the answer holds merit, signals the community typically use to determine the legitimacy of their peers’ contributions frequently fail to detect severe issues with GPT-generated answers. As a result, information that is objectively wrong makes its way onto the site. In its current state, GPT risks breaking readers’ trust that our site provides answers written by subject-matter experts.

---

Supposing every suspension is accurate, the magnitude raises serious concerns about long-term sustainability for the site

In the quote just above, you were concerned about the long-term sustainability of the site in support of suspensions for ChatGPT-generated content:

Moderators are empowered (at their discretion) to issue immediate suspensions of up to 30 days to users who are copying and pasting GPT content onto the site, with or without prior notice or warning.

It's so confusing and frustrating for me to watch you suddenly do a 180 to this direction, contradicting yourself. From my point of view, it's just inexplicable. I'm not quite angry, but I'm certainly flabbergasted.

---

However, since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly. Given the absence of question scarcity as a factor for answerers (note the above chart), the clear inference is that a large portion of frequent answerers are leaving the site, or the site is suddenly not effective at retaining new frequent answerers.

Again, the data does not support this. The rate of incoming questions is dramatically dropping, and with it, the rate of answers to those questions. And that inference of yours is not clear to me. Again, what does fewer answers have to do with people leaving? Until I see actual data that directly supports that conclusion, I'll be here pressing X.

---

In your draft approach, I'd like to see data focusing on the sizes (number of characters) of initial drafts instead of the number of drafts.

Like - why have you ruled out that people are just getting craftier about evading detection, and pasting into the answer input, and then doing editing there to make the content look less like it was ChatGPT/LLM-generated? Because that's totally what I'd expect to be happening over time - especially with suspensions being handed out. And that's very same reason I'm doubtful of your following

These days, however, it’s clear that the rate of GPT answers on Stack Overflow is extremely small.

Stray question: Do the drafts you're looking at include content that gets posted quickly enough that there is no intermediate draft?

---

Some folks have asked us why this metric is capable of reporting negative numbers. The condensed answer is that the metric has noise. If the true value is zero, sometimes it will report a value higher than zero, and sometimes a value lower than zero. Since we know how much noise the metric has, we know what the largest value for GPT-suspect posts should be.

Can you clarify and expand on this?

• In the graph, what's the light blue line, and the dark blue line?
• How/why does the metric have noise?
• What does "true value" mean? Are you saying you've found a 100% accurate way to detect ChatGPT answers? (obviously not, but I can't understand what else "true value" would mean. Are you referring to your own technique? Because as I've already explained, I think it has flaws (unless I've misunderstood your explanation of it).)

---

While we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox, we could characterize some of them.

... what do you mean "Could not recover"? You mean you lost them? If so,... how? status-completed: this was clarified here.

---

We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.

Have you talked with NotTheDr01ds and sideshowbarker? Ex. see NotTheDr01ds' post here, and sideshowbarker's post here. I'm sure they'd beg to differ. Also, this statement seems to lose the nuance of true positive and false positive rate. I think humans can have an excellent true-positive rate with some basic heuristics while erring on "the side of caution".

Under this assumption, it is impossible for us to generate a list of cases where we know moderators have made a mistake. If we were to do so, it would imply that we have a method by which we can know that the incorrect action was taken.

Again, have you talked with mods like sideshowbarker and users like NotTheDr01ds? I'm sure they could give you some practical, useful, concrete heuristics.

---

none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.

Again, I think you're missing something obvious - namely that questions are incoming at a lower rate as people go to ChatGPT instead of Stack Exchange to ask questions. I almost see the relation you're staring at to be a spurious one. I don't get why you're ignoring the relation between decreased rate of incoming questions, and decrease rate of answers. As I've shown, answer influx rates has remained consistently proportional to question influx rate for much more than 5 years.

---

Suppose we are right in this assessment and GPT removal actions are not reasonably defensible. How long can we afford to wait? To what extent can we continue to risk the network’s short-term integrity against the long-term risks of the current environment? Any good community management policy must balance the risk of action against the risk of inaction in any given situation, and the evidence does not presently favor inaction.

Again, I find it funny and sad how consistent that is with the GPT policy Help Center page that was written if you just replace "GPT removal actions" with "GPT addition actions".

---

if there is no future for a network where mods can assess posts on the basis of GPT authorship (where we are today)

... why? I don't follow the reasoning. Again, as I've said, Stack Exchange succeeded because its approach provided real value, and I don't see how that value is gone with the rise of ChatGPT. I plan to write up a new Q&A specifically about this, or reuse ones where I've already written answers, such as here and here.

Answer 5

I came off of read-only mode for this.

My initial take on this is that the analysis tries to affirm that user retention is valued higher than user quality, but the data points are still reasonable enough to ask questions.

The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.

How are you determining if a post was made with ChatGPT at all? What's the selection criterion for that? I mean, even today I've managed to find one just sitting on the front page that very clearly has a ChatGPT response.

If you're saying that you have a way to reliably detect ChatGPT responses, but the mods don't, that's prime disconnect number one. Otherwise all of this data is kinda pointless, isn't it?

The rate of suspension for frequent answerers rose by a factor of 16 from ~0.4% to ~6.6% since the rise of GPT, and has held steady since its release.

I wonder if this a side-effect.

You only tie HuggingFace to this as an example but there are other known detectors out there which simply aren't mentioned.

There's also this sentiment - the standing orders were for diamond mods to issue suspensions to users if they used ChatGPT. Some did so quite flagrantly during the winter, and even wanted to promote it as a new kind of workflow.

But perhaps, just perhaps, more people are eager to use ChatGPT and want to gain notoriety as experts.

I'll never be able to speak to the accuracy of detection methods, which is...always going to be a moving target. I don't think we can confidently identify all of them myself, but it does feel like the consequence of wanting high quality content on the site comes at the cost of users being suspended more.

But I suppose I'd need to challenge y'all on this. Yes, the rise is high. But is it justified? Is there more to this story than "wait, people aren't posting as much ChatGPT stuff, but they're getting banished a lot more often."

One additional note - the bans aren't permanent, and I don't think they're even that long. It's the case that someone did a bad thing and gets to go away for a week or so. It's up to them if they come back or not and want to be a part of the community... From a vanilla moderation perspective, that's not functionally different than someone posting a whole bunch of low-quality questions or answers and getting rate-limited, or someone deciding to inflame tensions in Chat and getting a time-out for a few days. The action's on them, not us.

If every GPT poster posted exactly three answers and were suspended within three weeks, this would imply a minimum GPT post rate of 330 answers per week on Stack Overflow; in practice, we would expect a significantly greater quantity. Measurements of GPT occurrences on the platform imply fewer than 100 GPT answers per week, in disagreement with this rate, implying the existence of many false positive detections.

Again, I want to know how you're detecting ChatGPT posts to begin with such that you got this number.

The suspensions issued appear to have a significant and measurable impact on the demography and volume of answerers on the site, preferentially excluding frequent answerers.

So the impact of this statements reads to me as, "people who post often are getting suspended more often and that's bad". This only begs the question of what quality their posts were to begin with. As in, is this demographic of people who frequently answer adding any measurable value to the site, even before all of this ChatGPT nonsense?

GPT detectors continue to be ineffective for detecting GPT answers on the platform.

Sounds like y'all got a good one. Mind sharing?

---

I suppose my ire in all of this is really centered around the timing and optics, too. ChatGPT hit the network like a runaway freight train. It's still a problem if we want to pride ourselves on having expert answered content on the site.

Then, we have the company pushing more and more AI on the community as if we're OK with going along with it at all twists and turns. And sure, there's some skeptical optimism, but nowhere near what the press the CEO puts out is implying.

So instead of engaging with this data first, you took the really heavy-handed approach of telling our diamond moderators that they can't do what they need to do to get rid of AI-generated content, which - wait for it - goes against the notion of having expert-answered content.

The story I tell myself from that is that you value people being here more than the quality of the content. And if that story is true, that doesn't work for me, someone who likes the quality that Stack Overflow outputs. That implies that I, an SME in Java, Python and other technologies, have no place here, or nothing to contribute to the company as an unequal but not insignificant contributor.

I hope my story is just a story. But given the posturing taken from this whole strike-thing and even the tone of this post, I'm not so sure.

(No, don't try to go on a diatribe saying you're committed to quality. If you were, the people who keep posting ChatGPT stuff would be unceremoniously banned from the site and we wouldn't care about it.)

I'll give you some credit though - you did at least bring us the data we've wanted. It's just...a day late and a dollar short, and hell you've already made decisions on this data which could be flawed from an outsider's perspective.

Answer 6

Copying my response from Teams, with some small edits to remove things that you chose not to reveal publicly for some reason:

I am skeptical about several points in your analysis. But most troubling at all is this bit:

So, when I say there are [small number that was removed from public post, for some reason] appeals that we cannot identify as correct, please keep in mind that our baseline value for this is zero, and it’s been that way for years. It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

Moderators are human, and we make mistakes and have disagreements. This rate should have never been zero- if you are actually giving appeals a fair shake, you should at least be finding yourself contacting the moderators in question for clarification from time to time.

When I tell someone I've just suspended to use the "Contact Us" button for appeal, it's with the belief that appeals are given a thorough evaluation. Have I just been telling them to spit into the wind all this time?

So, now you have [same small number again] suspension appeals you aren't sure about. So talk to the moderators in question! Maybe there are hallucinations that it takes a subject expert to identify. Maybe the moderators in question are using a heuristic you are unfamiliar with. Maybe some of them are actually wrong.

You'll never be able to find out any of this if you don't talk about it with the moderators. Please work with us to develop heuristics that work and identify ones that don't, rather than just giving up and forbidding us from moderating at all.

Side note:

mods can’t assess posts on the basis of GPT authorship (where we would be after this policy)

Not sure you meant to post this publicly, but I'm glad you're finally admitting publicly what was said privately- that mods aren't allowed to moderate ChatGPT posts at all.

Answer 7

What percentage of users who have been suspended (for any amount of time) for GPT use were active answerers as you described them immediately prior to ChatGPT's introduction?

---

I wasn't a very heavy flagger of GPT posts, but of the 4 users I flagged... 3 users were previously not active users and the 4th was previously and currently active. The 4th had suddenly started posting well written answers with much quicker turnaround than normal (4 in an hour) that greatly deviated from their normal "Here's some code"-like answers. My interpretation, having seen GPT-generated text many times elsewhere (not just here on SO) was that all of these cases were GPT-generated and they were all subsequently acted on by moderators.

Given I found all of these cases within minutes of opening the bounty tab, I find it hard to believe that these are as rare as your draft-based data seems to suggest. Is it not possible that GPT users simply altered the way they were posting answers? If I open the bounty tab today will I be able to quickly find a few more? (yep, on my first click, new user, 6 GPT answers on bountied questions. No detector necessary.)

---

The sudden drop is certainly troubling, but I question whether or not giving up is better than continuing to fight against under-verified content flooding the network. Neither alone will win back the users who are leaving the platform in droves. Should we sacrifice quality despite the fact that it won't bring back the users who are leaving, or should we begin to address why they are leaving?

Answer 8

What about everything else?

We've known for a long time that detectors (HuggingFace is the one I'm most familiar with, but others as well), have an extremely high rate of false-positives. Having data that backs that up is great. More information and data is good*.

Some moderators have stated that they have handled many thousands of ChatGPT flags without ever using a detector. - what is the rationale behind the overall ban, vs just banning Detectors specifically, and allowing other metrics, such as writing style, certain phrases, repeated answers, etc. from being used?

You also stated (in the policy announcement on MSE) that "we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied [...]". Aside from the fact that a suspicion is a terrible reason for a blanket ban, do you have data to support that claim? If so, share that data.

If that data is hard/impossible to get, if suspensions shouldn't be based on hunches (which they are not), why aren't you following your own belief and making a policy based on a hunch?

You also wrote "[...] we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard [...] excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are [...] successfully identifying AI-generated posts when they are written. [...]".

Why are you not sharing the moderator requirements publicly?

It is terrible to have policies about suspending users that are secret. You mention what the standard excludes. Please share the standard itself.

---

*Note to SE: I do have some concerns regarding methodology for HuggingFace specifically (there's a clarification I'd like to request). Unfortunately, I'm not sure if that can be publicly shared, so if you'd like, you or a CM can ping me in a private chat room.

Answer 9

The "actual GPT posts have fallen" graph is the lynchpin of your entire analysis. Without that graph, you've proven exactly nothing (aside from "engagement is down and the CM team has difficulty verifying GPT bans" - which are both worrying, but neither is a moderation problem). Unfortunately, as several other answers have pointed out, your "number of drafts" metric is completely arbitrary and not supported by any evidence (that you have included in this post).

Anyone who doesn't want their GPT answer deleted can easily just decide to edit a word here, a word there, and end up with just as many drafts as a legitimate user would (on average) have. Nothing in your analysis even attempts to engage with the possibility that most of the attackers are now doing that sort of editing. Worse, this is by far the most obvious countermeasure that an attacker could try in response to moderation, and so it is entirely reasonable to assume that many attackers started doing it at around the same time, even without coordination. If we allow for attackers intentionally coordinating with one another, then more elaborate explanations become possible (though whether you consider that plausible as an explanation is perhaps a different question).

The sensible course of action would have been to talk about this with the moderators. Show them your unverifiable suspensions, and ask them to help substantiate them. Talk to each other, and develop some sort of rubric or standard for evaluating GPT flags. It doesn't have to be public, and it doesn't have to be perfect, but it really ought to exist. You should not just point to statistics and claim that you know better than the moderators who are actually doing the work.

Releasing the data was a good start, but real progress is going to require collaboration with the community. The overall tone of your post is still very much in the "we're going to talk and you're going to listen" voice, and it really isn't helping your case.

Answer 10

OK, finally there is something to actually discuss. This discussion should have happened before the policy was decided, and the policy should have been decided with community-elected moderators' input rather than imposed by fiat ─ and the real policy remains secret ─ so let's not celebrate too much.

But now at least we can have a discussion about the factual basis for the policy, while not forgetting that the strike is about the imposition of the policy against the community's wishes and without our feedback, the differing public and private versions of the policy, and the slander against moderators, not about disputes over what this data implies.

---

At the 0.50 detection threshold, around 1-in-5.5 posts are falsely detected. At the 0.90 detection threshold, around 1-in-13 posts are falsely detected.

OK. How many moderators are trusting this tool at thresholds of 50% or 90%? I think pretty much everyone knows that these tools are completely useless at such low thresholds.

While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.

"Theoretically" is a strange word to use here. It is possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, i.e. thresholds above 97%. The data you have presented shows that this is empirically true, not just theoretically.

Whether or not a threshold of 97+% means the tool misses a lot of true positives is irrelevant. It makes no sense to forbid the use of a tool just because it misses a lot of true positives. Lateral flow tests for COVID-19 can miss 20-80% of true positive cases; that just means we can't (and don't) rely solely on LFTs. It doesn't mean we should ban LFTs.

Also, moderators have said clearly that they do not rely exclusively on this tool, or any GPT detection tool. We should expect that the false positive rate for moderator decisions, which are based on multiple kinds of evidence, should be significantly lower than the false positive rate for just one kind of evidence.

At this point, we can’t endorse usage of this service either as a tool for discriminating AI-generated posts or as a tool for validating suspicions.

Nobody is asking you to endorse it. What we're asking is for you to let us choose how our own communities are moderated.

---

Over the last few months, folks within the company have been working to answer the question, “What has been taking place in the data coming out of Stack Overflow since GPT’s release?”

(Emphasis mine.) Stack Overflow is the largest Stack Exchange site, but that means it is not representative of other Stack Exchange sites. Even if this data really does show a need for an extremely permissive policy which de facto allows users to plagiarise AI answers, it at most shows that need for Stack Overflow, not all Stack Exchange sites.

In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

This graph looks pretty noisy, so I assume there are some wide error bars around that "quadrupled" figure. For the sake of argument let's say it's roughly correct, though. So why are the more active users leaving?

Are there just fewer questions? ─ Yes, there are. You attempt to rule this out as a factor, because the "number of questions per frequent answerer" has gone up, not down. But this is not surprising, and doesn't imply anything.

Imagine there are 100 questions per day, 50 nerds who answer one question per day, and 10 hypernerds who answer five each. Now imagine the number of questions goes down to 60 per day. The hypernerds are more affected by the question shortage, so suppose they're more likely to leave; say 30% of the nerds and 50% of the hypernerds leave. There are now 35 nerds and 5 hypernerds, and all 60 questions per day are still getting answered. So the number of questions per hypernerd has gone up from 10 to 12, but there aren't more questions available to be answered. So this number rising simply doesn't imply that there are enough "available" questions to retain more of the hypernerds.

The only assumption I've made is that hypernerds are more likely to become uninterested if there aren't enough questions for them, whereas regular nerds aren't there only to answer questions so a question shortage is less likely to motivate them to leave. Seems pretty plausible to me, and it's consistent with the trends in your data (on questions per hypernerd, and proportion of answers written by hypernerds), so we can't rule it out using this data.

Additionally, there is a flaw in your framing: the idea that what matters is the ratio of questions per answerer, rather than the actual number of questions. The ratio probably matters much less than you give weight to it, because each question can be answered by more than one person, so an answered question is still often "available" to be answered again. In practice, many hypernerds are interested in some specific tag, and they may be one of a small number of people answering questions on that tag, or even the only one. What matters to these users is the total number of answerable questions being asked in that tag. And if they leave the site, the questions on that tag don't become "available" to other users, because the other users aren't experts on that topic and aren't interested in those questions.

Has the quality of the questions gone down? ─ Perhaps the hypernerds are more motivated by having interesting questions to answer, whereas the regular nerds just answer a question every now and then if they happen to notice it. Unfortunately your data says nothing about question quality, but anecdotally this is the reason I have stopped writing so many answers on Stack Overflow. I could speculate on a few reasons why question quality might have fallen in the advent of ChatGPT ─ perhaps people who know how to write a good question are more likely to get a satisfactory response from ChatGPT themselves, and therefore don't need to ask Stack Overflow ─ but anyway a drop in question quality can't be blamed on the AI moderation policy.

---

After we allowed GPT suspensions on first offense, 6.6% of users who posted >2 answers in a given week were suspended within three weeks [...] no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community.

This is a non-sequitur, because you include AI plagiarists among the group of people "trying to actively participate in" Stack Overflow, but plagiarising answers from an AI is not what it means to participate in Stack Overflow. If all of those users who get suspended are indeed AI plagiarists, then the correct percentage of suspensions among people who are "trying to actively participate" in the community is 0%, not 7%. So your argument here is wholly uncompelling.

These suspensions only negatively affect the community if they are false positives. You have given some data on the false positive rate of AI tools, but not about the false positives for moderator decisions to suspend users (which, again, are made on the basis of multiple kinds of evidence).

Additionally, this metric ─ users who posted >2 answers in a given week, and were suspended within three weeks ─ seems suspiciously precise. Why is >2 answers the cutoff? Why is 1 week the period in which those answers were written, and why is 3 weeks the period in which they were suspended? It smells of cherry-picking to me. How robust is this finding to changes in the metric?

Instead suppose that no more than 1-in-50 of the people who were suspended for GPT usage were not actually using GPT. In order for this to be true, a large volume of users would have needed to immediately convert from being regular users to ChatGPT users;

No, that does not follow. It can be true if there is a steady flow of new users who use ChatGPT, or if users suspended for ChatGPT use return to the site after their 7-day suspension and then post more ChatGPT answers, or if not everyone who uses ChatGPT gets caught immediately. All three are very plausible.

this value alone rings a deafening number of alarm bells for potential false positive detections and contributor loss alike.

This 7% figure implies nothing about the rate of false positives, because it uses the wrong denominator. Many users who don't use ChatGPT and don't get suspended for suspected ChatGPT use, are not in the ">2 answers per week" category, but the number of those users obviously matters for the false positive rate.

Likewise, this figure doesn't imply anything about loss of legitimate contributors unless we already accept the doubtful claim about false positives.

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform [...] In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange.

This metric doesn't seem fit for the present purpose. Firstly, it's an absolute number, whereas we already know that the absolute numbers of questions and answers have been falling, so automatically the absolute number of ChatGPT answers is expected to fall alongside that. To support your argument about false positive rates, we need the proportion of answers written by ChatGPT, not the total number.

Additionally, it assumes that the behaviour of ChatGPT plagiarists has not changed over time. But this is an unreasonable assumption, because as Stack Overflow's policy on ChatGPT answers became more widely known ─ and people's 7-day suspensions for posting ChatGPT answers expired ─ we should expect that the AI plagiarists' behaviour changed to try to avoid getting caught. More sophisticated plagiarists will change a few words, delete "fluff" sentences which don't contribute to answering the question, introduce intentional spelling or grammar mistakes, or so on; these are smaller edits, so they will tend to make answers look "less ChatGPT-like" in the ratio of small to big edits.

So this metric going down does not really indicate that ChatGPT use has gone down.

This metric is sensitive to noise, but was validated against other metrics early on at the peak of the GPT answer rate.

The fact that it was validated early does not mean it remains valid, since AI plagiarists' behaviour should be expected to change over time.

The following chart shows the expected % of answers posted in a given week that are GPT-suspect.

There is no "following chart" ─ perhaps you intended to include a chart here but it got lost while editing?

Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. [...] could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close.

The estimate of 70-100 per week is probably a significant underestimate for the previously-stated reasons. It's quite plausible that 330 per week is the correct number. It's also plausible that the correct number is much higher, and moderators aren't catching all of them.

Even so, if you were right that there are only 70 to 100 AI answers being added to Stack Overflow per week, that would still be a bad thing and it would still be necessary to moderate them. If left on the site, those answers will accumulate over time, getting upvotes from people who don't know better (because they resemble high-quality answers), and Stack Exchange, Inc. has clearly recognised the harm that would be caused by leaving these answers up on the site.

So even if all of your analysis was correct ─ and to be clear, it isn't ─ but even if it was, it still wouldn't logically support the new policy of de facto allowing AI plagiarism.

---

It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

You haven't said why you were unable to verify these moderator actions. Is it because moderators have not provided sufficient information about how they have made their judgements, or is it because you don't agree with those judgements?

If it's the former, that would be a basis for you to tell moderators that they need to provide more information when they suspend users for suspected ChatGPT plagiarism; if it's the latter, then that would again be something to discuss with moderators. Either way, it doesn't support the new policy which practically forbids almost all such suspensions.

Instead, the most we can do is state that we just can’t tell. We lack the tools to verify wrongdoing on the part of a user who has been removed, messaged, or had their content deleted, and this is a serious problem.

You may not be able to tell, but I remain entirely unconvinced that the moderators issuing these suspensions can't tell. The data you've presented here doesn't support that conclusion.

Is it still possible that the proportion of false positives is small? Maybe so – it can’t be completely eliminated at this time. [... But] it would require some very strange user behavior en masse around answering, by users who were otherwise answering questions normally. These are behaviors we do not have an organic explanation for after months of exploration

Perhaps if you had discussed this with the moderators, they might have been able to offer some explanations which you failed to consider.

It looks really bad for you to impose a policy by fiat, keep the justification for the policy secret for a week, and then admit that the basis for the policy is that you couldn't think of any other explanations for this data ─ when you never involved the people with direct experience of the issue in your attempt to understand that data. I'm sorry, but that is not behaviour I associate with a good-faith effort to reach the truth.

What we know, right now, is that the current situation is untenable. We have real, justified concerns for the survival of the network.

It's a shame that in this sentence, the "current situation" you're referring to is the rate of users leaving Stack Overflow, rather than the behaviour of Stack Exchange, Inc. which has caused this strike action.

It's our community, so we are at least as concerned as you about threats to the community's continuing viability. Unfortunately, the behaviour of Stack Exchange, Inc. is currently the greatest threat to our community's continued existence, and while publishing this data and your analysis is welcome, it neither demonstrates an understanding of why we are on strike, nor does it address any of the strikers' demands.

Answer 11

We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.

What "evidence" do you have for that? Have you actually tested this with one of the Stack Overflow Mods who have reviewed thousands of GPT posts?

What if you:

• Take two of the Stack Overflow moderators who have handled thousands of AI flags.
• Take 100 random, pre-2020 questions from Stack Overflow that have existing answers
• Paste the markdown of the question into ChatGPT
• Take the unedited ChatGPT results and ask the Mods to identify the ChatGPT answer (or whether they believe it is inconclusive).
• Then tell us how many GPT answers were identified correctly and how many were "wrongly suspended" ;-)

Have you done anything close to this to support your "evidence-backed assumption"?

That's certainly the "easy" version of the test, since most GPT answers do (IMHO) have edits at this point. But if you don't believe it's possible for a human to identify a GPT answer at all, let's start with this easy test.

If you want to go a little more "difficult":

• Take two of the Stack Overflow moderators who have handled thousands of AI flags.
• Take 100 random, pre-2020 questions from Stack Overflow that have existing answers
• Take a random 50 of those
• Paste the markdown of the question into ChatGPT
• Add the unedited ChatGPT results to the pool of answers for those 50 questions
• Place all 100 original questions before the Mods and ask them to identify whether there's a GPT answer (and which one), no GPT answer, or inconclusive.
• Then tell us how many GPT answers were identified correctly and how many were "wrongly suspended" ;-)

I'll put my money where my mouth is and happily take the challenge as well. That way you can even determine the rate at which a non-Moderator is potentially flagging false-positives.

I'd be interested in the results if it was run twice, once with the subjects using automated detectors to verify their suspicions, and once with no automated-detectors allowed.

Answer 12

My summary of your nine-google-doc-page question

• You have tested how the ChatGPT-detectors work on human written posts and you did not like the accuracy.
• You came up with a heuristic that you use to determine the number of possible ChatGPT-posts on the site.
• Your heuristic did not show a surge of ChatGPT-posts, but the number of suspended users of a certain segment increased.
• The number of answers continued to decrease, although the outflow of questions did not.

Multiplying one by the other, you concluded that the moderators incorrectly suspended 16 times more users than they should have.

My thoughts

If the community is fighting something intentionally, then it will find more precedents than if the users stumble upon the problematic posts by accident

For the past few months, the community has been specifically looking for ChatGPT-posts. When we deliberately look for something, we will find much more of it by definition. Look at the data for any other community-driven initiatives. For example, look at the data for the plagiarism detection initiative that the community held recently.

Users get suspended for low quality content, plagiarism and more, not only for ChatGPT-posts

When the community deliberately works on something, other types of activities usually increase as well. By searching for potential ChatGPT-posts, users will definitely find plagiarism, low quality posts, and so on. Much of this will result in additional suspensions. Moreover, we will see suspended “now” users for their “old” posts.

The number of answers is most of the time late

Questions and answers are like eggs and chickens. It takes a while for an egg to become a chicken. In addition, the number of answer givers in the last few years has declined faster than those who ask questions. Especially those who post many answers.

Your heuristic may not be correct. And it is definitely not correct during the period of "active cleaning" of the site

According to the information in your question, all conclusions are based on some heuristic about the time it takes for a user to create a post. Have you thought that it may be incorrect or not very accurate, especially when the community proactively is working on cleaning the site?

Moreover, have you tried at least normalizing it by the number of processed flags over a period and presenting the data not by the suspension date, but by the date of posting the content that violates the rules? I think you would see a different picture.

With ChatGPT, those who would not be included in the group of "active users" will get there

And when they are suspended, they will create the very anomaly that you write about.

The term “false-positive” has a specific definition

To talk about a false-positive rate, you need to know exactly how many posts the moderators processed incorrectly out of the total number. But you didn't analyze their work! No way you can talk about the “false-positive rate”, you simply do not have data for that.

To be honest, I spent an hour and a half reading your post, but did not find the meaning of many of your "scientific” dictums.

What I see

Your data is a set of facts taken out of context, which are in no way connected with each other, and even more so with your conclusions.

Answer 13

I already posted an answer to this question, in which I try to assert why people create posts using GPT and why others do not find that behavior healthy for the community. I still wanted to post another one, as I am unable to move past what we can read in this quote from the question (emphasis mine):

Based on the data, we would hazard a guess that Stack Overflow currently [June 7th?] sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error. We can therefore say that the rate of GPT posts is far less than it was two months ago, and then it is less than it was two months before that.

So, under what conditions could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close. This also assumes every user who posts GPT answers are caught, and that GPT answerers post no more or less than three answers in a given week. In practice, the site should be seeing significantly more than 330 GPT answers per week to support this suspension rate.

It could be possible, either due to severe measurement error or due to an unexpected change in user behavior that obfuscates GPT usage using this method. But the evidence for this viewpoint does not appear strong.

Stop taking us for fools

I know for a fact how wrong you are and I find it extremely offensive towards the entire community (moderators, curators, flaggers, askers, answerers and readers) that:

1. You've dug in your heels so deeply that you base your entire policy on this belief, and
2. You don't respond to any of the overwhelming amount of constructive criticism shown in the answers to this question.

Stop trusting your tools

Whatever your methodology is, stop using it. Start listening to your moderators and users. You are off by orders of magnitudes, you are ignoring users who slightly edit and mark up GPT-generated text in the answer box, you are somehow severely erring in your measurements, you are seeing (or rather not seeing) at least hundreds of GPT-generated posts per week. You have been for half a year already.

My methodology

I am an English as a second language person. My native tongue is Dutch, and I have been hearing, speaking, reading and writing English for over three decades already. I love actively working with language, researching etymology, rewriting sentences twenty times to have the perfect cadence, throw in a joke or two here or there, and so on. According to multiple people, I'm doing a terrific job posing as a native English speaker (though not audibly; it is apparently pretty hard to get rid of a Dutch accent on your own).

As stated in my other answer, I have read and vetted literally tens of thousands of posts on this network, mainly on Stack Overflow. Besides that I am an avid reader of and poster on Reddit and Tweakers (a Dutch tech news site and forum).

Having played with ChatGPT for a couple of months, there are some telltale signs about the prose it generates, especially in the context of asking for help with programming problems.

One of the biggest signs is that it will almost never tell you that that what you're trying to do is a stupid idea, something developers can't hear often enough. It will do exactly what you ask for, and produce very readable code, often with comments explaining what that code does, and then proceed to repeat that code in the form of an explanation thereof in English. But the code will be conceptually and/or idiomatically wrong, if not syntactically.

This all is leading up to this: I know, through my 30+ years of being very active online, how users behave on forums and Stack Overflow, and how they write in English and Dutch. Sure, doing things a lot doesn't necessarily mean you're good at it; you'll just have to take my word for all of this.

For someone like me, it is not hard to recognize the obvious signs of ChatGPT-generated text and code. And there are dozens of us. Dozens! (There is room for magnitudes of error in this statement.)

Case studies

#1: 10 answers in 30 minutes

On May 19th, right about where the GPT-suspect posts graph converges to zero, I have flagged a post from a now-deleted user with the message:

Someone has discovered ChatGPT. Their last 10 or so answers within 30 minutes have been generated.

To clarify: they posted this answer (screenshot with moderator name and red haze removed using F12 for sub-10K users) that has obvious GPT signs, eight minutes after the question was asked, along with at least 9 other answers, in a 30-minute timespan, after an answering-hiatus of 6 years, ignoring a comment by the OP stating "It does not produces expected results", ignoring my comment that GPT-generated answers aren't accepted, linking to Temporary policy: ChatGPT is banned and https://stackoverflow.com/help/gpt-policy (left three minutes after the answer was posted), leaving all their generated answers up until a moderator deleted them more than 1.5 hours later.

You will be unable to convince me that this person, with their 10 answers showing typical GPT-generated prose in 30 minutes and all other signs, was not using GPT to craft their (incorrect) answers. I did not use tools to detect this, I used the heuristics outlined above (i.e. my brain).

#2: 10 answers in 10 minutes

Two days earlier, May 17th, I encountered an answer which I flagged with:

10 answers in 10 minutes? ChatGPT.

They answered the first revision of a question, basically stating:

[When] the server has an outage [...] my page [...] gets a Http 503 'service unavailable' returned, and that's how it stays [...] it's on a unmanned PC [...]

do I need [...] to redirect to a local page if, on refresh, http 503 (or other error) is returned?

They need code that refreshes their user-interaction-less web page when the server serving that page returns an error.

The answerer posting 10 GPT-generated answers in 10 minutes (an inhuman feat to begin with) after a nearly 2-year answering gap produced this answer (screenshot) which is basically a very generic rehash of every "how to set up error handling for ASP.NET" tutorial, entirely ignoring both the premise of the question and the question itself.

Again: extremely unlikely that this was a false positive, and that this was the only user on that day posting GPT-generated cruft.

Resuming

I'm not in the SOCVR chatroom nor do I do review queues anymore, so I encounter all my questions and answers organically: from searches related to things I'm working on, or from trolling the frontpage while I'm not working.

I have (only) 21 more handled flags on ChatGPT answers, which are older and usually don't point to as many answers as these two flags I expanded on above. Ironically enough, most of those remaining flags were on answers posted mid April, just when your graph starts sharply going towards zero.

If I, in my casual browsing during the day encounter users with such egregious display of using GPT, in the three weeks before when you claim the upper bound to be 10-15 answers a day, then one of us must be wrong. And by now I'm pretty confident it's not me.

Mea Culpa

I was in fact wrong at least once. I shouldn't have flagged this answer, accusing it and this other one of being GPT-generated. Both of those answers remain up and the writer unpunished, and rightfully so. Their writing style is what threw me off.

The moderator who handled my flag correctly noted that this was a false positive – clear evidence if there ever was that moderators were not simply rubber-stamping flags or triggering on insufficient evidence.

In conclusion

I'm not claiming to have superhuman properties. I'm not saying I'm free from faults. I'm merely saying that there's something in your analyses causing you to severely under-report GPT-generated content, causing unsubstantiated policy changes that the community rightfully disagrees with.

I don't know what's causing your data to be that off, and I don't know how to express my (and others') heuristics into definitive rules (nor whether we should publicly do so), but you need to reconsider your criteria.

Please listen to your moderators and users.

Answer 14

The one-size-fits-all policy is a problem

The data you present is somewhat compelling. An answer with many cached versions, showing the answer being fleshed out over time, is a pretty reasonable defense against ChatGPT. I imagine you will eventually turn this heuristic into a metric that mods have access to. I'm not sure I completely buy all your conclusions, but after a quick skim, the data is more compelling than I thought it would be. Certainly none of us mods want to suspend people who don't deserve it.

But different sites are different. On my site (Academia.SE), many of our askers are at a very vulnerable stage in their lives, and they're looking for career guidance from people who have been there before. This is not a programming site where they can say "thanks, but I tried it and it didn't work"; they are often facing life-changing decisions that can only be made once. It is unacceptable that they would make such decisions (unknowingly) based on auto-generated answers. In other words: on Academia.SE, I would much rather turn away someone who is posting content indistinguishable from ChatGPT (which we don't really want anyway) than let someone base a difficult decision on some auto-generated nonsense.

Our community can figure this out. We don't need you to tell us what the answer is. We need you to tell us what the problem is (which you have now done) and then we'll come up with the answer. Our goals are aligned: we don't want to suspend innocent users, and I'm sure you don't want bad advice to run rampant. So please retract your policy (at least for the smaller sites) and let us mods get back to work: we'll start a conversation with our community on meta and decide how we can better balance these two priorities.

Answer 15

Thanks for elaborating on your reasoning. However, I still have questions.

How was urgently overriding the moderators' mandate a conclusion?

Your exposition does not at all reveal how such an outrageous action could be the result of your analysis.

Why not merely halt suspensions?

If this is the part you were actually having trouble with, I'm sure temporarily allowing the users who posted AI-generated content to remain on the site would have been a more appropriate solution to the actual problem you were apparently trying to solve, and somewhat more palatable to the moderators, if not outright embraced by them.

If you also wanted to allow alleged AI-generated content to remain on the site, even that could have been more acceptable than what you ended up with. We already have mechanisms for marking content as contested; for example, disabling voting on the reported posts would prevent the OP from using these posts to improve (or ruin) their reputation.

Again, why is the accuracy of automatic detectors important?

Reading between the lines, I guess you are looking for ways to validate the moderators' actions. But why? You don't need a "rudeness estimation tool" to validate suspensions for rude behavior, or proof that promotional posts are genuinely spam, as opposed to honest mistakes by over-eager marketers (actually a really thorny problem in its own right).

You rely on the moderators to make these calls every day, and on the CMs to handle appeals for all of these other cases. Why is this process not acceptable for AI-generated answers? Because it's harder? That's precisely why the community regards them as particularly problematic, you know.

What's with the alleged cultural bias?

You originally claimed the ChatGPT suspensions might have "biases for or against residents of specific countries". In spite of requests to clarify this, I have yet to see any attempt to explain this allegation.

The best speculations I have seen are that some users who are not fluent in English might have been suspended because they used ChatGPT to create their posts. Is that what this is about? How is that a bias for or against residents of specific countries? I would think it would indiscriminately penalize anybody who is incapable of writing coherent sentences in standard English (which includes a portion of the native speakers of English as well).

---

More tangentially, I have some speculations about the dynamics of the ChatGPT eruption.

Where did we go?

Did users who previously posted answers actually stay on the site?

In particular, did they stop posting answers, but spend more time curating content?

You should be able to see which users stopped visiting vs which users merely stopped contributing answers, vs perhaps also which users flagged posts more often than previously.

Did ChatGPT users get better at evading detection?

Like in many adversarial scenarios, you would expect both sides to evolve.

In the first wave, you would expect many users to get the same bright idea, get caught, and learn from the experience.

While some would simply learn that the community didn't like their attempt at gaming the system, and stop doing that, others would take this as a new challenge to overcome.

My concrete speculation is that this is what accounts, at least partially, for the falling rate of detected ChatGPT answers in recent weeks.

Isn't it great if beginners got helped by ChatGPT?

If you are as enthusiastic as your CEO about the potential of AI, isn't it actually a good thing that Stack Overflow no longer receives mundane "where is the missing closing parenthesis?" questions? This translates to less traffic, but also fewer trivial typo / duplicate questions which are useless noise to everyone except the asker. Fewer junk bytes in your database, less curation time spent by your valuable volunteers on pointless rote content, and fewer newcomers who complain that their low-value, and in the worst case also low-quality, contributions got rightfully downvoted and closed. Thus, happier results all around, and more time for us to answer actually useful and unique questions.

Answer 16

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform.

While I'm still digesting much of the data, you seem to be failing to consider alternative explanations for it.

The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.

Or are users editing their GPT posts more nowadays rather than just copy/paste in an effort to evade detection? I've certainly seen the latter, although this week, with the floodgates opened, it's much more copy/paste once again. I cannot remotely imagine that you are seeing a "very small" number of GPT posts this week, as I've spotted more than 100 myself barely looking. These are clear copy/paste

Answer 17

there is no future for the network at all.

You might be closer to the truth than you think, or fear.

Regardless of the above, no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community.

Is "trying to actively participate" the only bar a Community Manager sets for the members of a healthy community? You know what Stack Overflow is missing, partially by design? Because it is a "no chit-chat" site, because members aren't allowed to talk on-site, but only post answers, that's what you get: a bunch of uncoordinated (as far as it looks on-site) people with no common goal. You cannot throw hundreds of thousands of members together without hardly any guidance (literally nobody reads FAQs and Help Centers), and expect them to generate answers and to have those answers have something in common:

Quality

Quality comes in many forms. Attention to detail and language, for one. Eagerness to teach, as another. Providing readable code and reasonable examples. Using abstraction and experience to craft an answer that not only answers the explicitly asked question, but also addresses any implicit properties of the problem, and translate that into knowledge that applies to the question as asked but also serves as a useful resource for later visitors.

There is an abhorrent lack of such educators. I've asked about this on Meta Stack Overflow, three years ago: Where's the new boatload of experts who can explain stuff to me like I'm five?. It's very problematic for the network in the long run.

Apart from that, there's another problem on Stack Overflow:

Quality control

I mainly browse Stack Overflow from the homepage. It has a filtered view with questions that match my areas of interest and expertise. I click many questions a day, I often abstain from voting. In my 13 years and 4 months of membership I have posted 3,591 answers, averaging 5 per week. Apart from answering, I downvote unclear questions and incorrect answers. My voting stats (viewable for everyone on my profile):

3,475 upvotes
14,543 downvotes (80%)
11,649 question votes
6,369 answer votes

From this you can deduct that I have read and assessed tens of thousands of human-written questions and answers. I downvote answers that don't explain what they contain. I downvote when answers contain code that omits a glaring problem from the question. I downvote when an answer contains an incorrect claim, especially when there's easy to find documentation that proves otherwise.

People do not downvote enough. Worse, people counter-upvote downvotes as they see them come in. When I downvote an answer "too soon" after it gets posted, it sometimes immediately gets a counter-upvote, while being not that great at all or flatout wrong.

Even worse than that is that most posts do not get the experienced eyes viewing them that they deserve, neither for appropriate voting nor for answering or correcting them.

As time moves on, and as posts remain on and remain to be posted onto the site, the average quality of posts will keep declining. Allowing GPT-generated answers to stay will only accelerate this.

We need more knowledgeable, invested people, not more posts, nor more copy-pasters.

none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.

I have one for you. It's actually pretty trivial. Most posts on Stack Overflow come from people who do not care about quality. Either through lack of knowledge, lack of interest, different education or upbringing, not enough experience, or any other combination of factors. Those people can answer only basic questions. They can do so unchecked, because nobody reads those posts anyway. Those questions now get asked to ChatGPT, instead of on the network. Now those people have nothing to answer anymore. So they venture into different tags, different topics, in which they actually have no experience, but GPT can generate nonsense resembling an answer. So that gets posted instead. Those people get suspended. Learn this is not the way they should behave. Leave.

This message from the moderators is akin to "Hey, you know what you're doing? We don't want that here", and for some users that might be the first time they ever hear that here.

Quality control in measurements

I have no idea how you came to this conclusion:

Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error.

But you could not be more wrong. Take this user. On May 30th, they have posted 6 answers. I suspect all of them to be GPT-generated based on writing style. From two of their answers, I know they are GPT-generated because of what's in them:

• How to set tags for Secret in Azure Key Vault using C#: they hallucinated a class and a method (SecretCreateOrUpdateOptions, CreateOrUpdateSecretAsync()) which don't exist anywhere on the internet except on that post.
• write List<T> to File: they answered the question to a T and perfectly explained what they changed, but missed a glaring problem: OP is asking for an explanation about generics. What use does a generic method have when it accepts a List<T>, but then constrain T to be PERSON using an if statement? Only an AI could make that up and be so meticulously reasonable about it.

You believe this user to be single-handedly responsible for generating 50% of GPT answers for May 30th? Then it should be trivial to find the other user and ban them both.

You know what Stack Overflow can do? Change. Trust your community-elected moderators to do the right thing. They know what their sites stand for. They know what they, and their community members, do and don't want to happen on their site.

Answer 18

This is based on a question that I asked on the Stack Moderators Team. It wasn't addressed there, but it may have gotten lost in the shuffle. I've trimmed it down to make it more appropriate for Meta Stack Exchange.

What, exactly, is the problem you are attempting to solve? There may be more than one problem. For each problem, can you express that problem in a single sentence or question? What questions are you trying to answer or what problems are you solving?

After reading this post, I see several different problems being expressed, such as, but not necessarily limited to:

• GPT detectors have a false positive rate that is higher than is acceptable for use as a moderation tool.
• User engagement is down.
• The policy of suspending users for posting algorithmically-generated content is causing potential contributors to leave the site.

It seems like the initial theory is that these are all connected. That is, the problem statement appears to me to be something like: The tooling that moderators are using to detect algorithmically-generated content has a high false-positive rate, which is leading to a large amount of erroneous suspensions, which is causing users to not convert to contributing members of the community.

I don't understand why you would combine these problem statements into one. They feel like three separate issues, although they could be related. Until there is evidence that explicitly links them, I don't see why the assumption should be made that they are related. When you combine problems, you built upon supposing truth for part of the problem that may not be there.

I do think that there are problems. However, we don't know what the formulation of the problem statement that led to gathering this data and doing this analysis was. And, although making the data and analysis public is a good step, it doesn't address concerns with the underlying methodology.

I would also encourage anyone interested in questions to check out The Art of Asking Questions: Ask Better Questions, Get Better Answers. It's really eye opening and informative about the effects on how questions are phrased to how people give answers. Unless we know the question, it's hard to assess the validity of the answer.

Answer 19

I have big doubts about this part of presented data:

Yet, at the same time, actual GPT posts on the site have fallen continuously since release

It's implied that posts with a small number of Drafts are occurring less and less on sites, so it means that we have less ChatGPT answers.

I would argue that the only conclusion that can be drawn is that we have less posts that are blind copies from ChatGPT, with no reading or edits to apply basic formatting.

I've executed a small experiment: I've gone to one of my latest answers, copied the question to ChatGPT, and attempted to create an answer with its output. It took me three drafts to paste and reformat the answer so that it would be adequate to my personal standards.

I don't have the exact number of drafts required to create an initial answer, but since it consisted of three rather conservative in length paragraphs, and ready-made code copied from an IDE, I'm pretty sure that it took me less than 6 Drafts to create the answer.

I'm open to the idea that while this study was conducted there were some additional internal indicators of answers being AI-made that were applied, but the published part, in my opinion, has major flaws regarding this point on drafts.

Answer 20

Analogy and motivation

Let's say I'm a dairy farmer in Wisconsin. I look at some data about milk output from my dairy and find that output is down 50% from last year. This is a remarkable drop, and it must be explained! What on earth is causing this? I need to call a vet to check that there isn't some infection afflicting the cows. I need to send our feed for analysis and make sure it's properly balanced. I need to have our milking machinery serviced and checked. This is all going to be very time-consuming and expensive, but it's very important to get to the bottom of it.

But first, I should count how many cows I have, because if I have half as many cows as the year before, then all these expensive solutions which actually might be good for solving a "less milk per cow" problem are not the right solutions for a "fewer cows" problem.

If our Stack Exchange cows are our answerers, the analysis presented in OP is suggesting that our very best cows, the ones that produce more than 3 answers in a given week, are being particularly affected. They're dropping faster than the overall answers! The proportion of answers by the >=3 answerers is down! But, before we go about an expensive way of figuring out what is targeting our best producers, is there any simpler explanation?

One could, perhaps, find these same effects due to an actual cause that is "answers in general are down" plus an analysis strategy based on thresholds that creates an illusion that the top answerers are most affected.

Recapitulating results

I've grabbed post data from SO in two weeks, one from Nov 2022 and one from April 2023. I'll refer to these as just Nov and Apr from now on, but note these are 1-week examples from each, not the whole month.

Details of the analysis are here: https://pastebin.com/mVEMVuFD I'll just include the results in the rest of the post.

33571 answers in Nov. 21001 answers in Apr, 62.6% of Nov.

2399 users with >=3 answers in Nov. 1360 users with >=3 answers in Apr. 56.7%

47.9% of answers in Nov are by users with >=3 answers. 39.6% of answers in April are by users with >=3 answers.

So far, these results are right in line with the data presented in the original post here: the number of users with >=3 answers has dropped off quite a bit. The proportion of answers by the top answerers is down to a lower fraction than overall answers (39.6% is much smaller than 62.6%).

Checking an alternative explanation

So, let's consider another possible explanation, which is that this result is really just a threshold phenomenon, introduced by the 3-answer threshold. If we want to know if the top answerers are really more affected than the general trend, we need to compare to a suitable null hypothesis.

We can simulate what the data would look like if the decrease is uniform by starting with our Nov data, and randomly tossing out 62.6% of the answers. Then, we can look again at the April results and check whether they are consistent with a simple across-the-board reduction (if they look like the simulated data), or whether the data are more consistent with an alternative hypothesis that the top answerers are being specifically targeted (if they don't).

I run the simulation 1000 times. The mean total answers in the simulation is 21000, 62.6% of the original count (good reality check there).

The mean number of answerers with >=3 is 1461, 60.9% of Nov.

The percentage of answers by the top answerers is 40.9% of Nov.

It looks to me like indeed, the reduction in top answerers, at least comparing just these two weeks of data, is more than what you'd expect just from an across-the-board drop. The simulation predicts 1461 users with >=3 answers, but the actual April data only had 1360 such users. If you were just expecting the number to drop proportional to the number of answers, though, you would have predicted 2399 * 0.626 = 1502 users, so some of the drop is accounted for by the general effect filtered through the threshold, rather than the specific one.

Similarly, for the "fraction of answers by the top users", the simulation predicts 40.9%. The actual observation 39.6% doesn't seem practically different from the expectation (though it would be statistically significant if you used my simulations as an empirical null distribution).

If we plot out the actual April versus simulated data we can get a better idea of what's going on. It looks like there are really no fewer answerers than predicted among the very top answerers, those producing over 20 answers in a week. Rather, it looks like there's a proportional drop in people posting 3-4 answers and an increase in those posting 1. I think these data would make me look for reasons that people aren't posting more than 1 answer, rather than only focusing on why people who post a lot of answers might be going away. Some people have pointed to the 30-minute timeout as a possible cause. One might look at historically how many of the people who post 3-4 answers in a week are new accounts posting in rapid succession to guess at that impact. You might also look at how many people experience the 30 minute block in a week. If the number seeing the block is similar to or greater than the 100-150 missing people who'd normally post >=3 answers, that might support it as a cause.

Importantly, the simulation predicts about 5300 people who would have posted in November just don't post an answer at all in April, and of the 1000 fewer people posting at least 3 answers, all but about 100 are following a general pattern of fewer answers rather than a specific pattern.

Summary

In summary, it does look like you're losing more of the "3+ answer" crowd than you'd expect just from the drop in answers, so it's important to figure out where they're going. Importantly, though, it may not be that anyone is going anywhere, really, but just that users who before would post 1 answer aren't posting more. Overall, the discrepancy in frequent answerers is still quite small compared to the overall drop in answers, which likely originates in a drop in questions, as others have pointed out.

I am very open to feedback and criticism on this approach. I didn't have a lot of time to play around in SEDE, but if someone wants to make a data set that has week-by-week data instead of just one week extracted, I'd be happy to tweak the rest of my code to plot this out over time.

Answer 21

TL;DR: Your analysis that concludes “automated GPT detectors have unusable error rates on the platform” is not sound.

I can’t easily find what exactly Hugging Face’s percentage signifies, but I assume that it tells you how likely a random non-AI text (of comparable length) is to appear less AI-generated than the given text by its internal metric. In other words, the percentage is the true negative rate. This is the same test you have been exposing it to, just with a restricted dataset (namely only SO posts). Thus, ideally, its false-positive rate would look like the green line I added to your plot:

First, given that the situation is not ideal as you used a restricted dataset (for a good reason), it is no surprise that the blue and green line strongly deviate in parts, but that’s no problem per se.

In the region up to 90%, Hugging Face performs much better than advertised on your data. I mention this only because you seem to focus on this region for estimating a false-positive rate, although hopefully nobody uses it for detecting AI posts. In brief: If Hugging Face says that a post has a 50% chance of being AI-generated, of course I would expect it to be human-generated in roughly half of the cases.

Now, the really interesting region is above 99%. Here, Hugging Face performs worse than advertised, although not drastically so. Importantly, as you mention yourself, you need much more data to make reasonable estimates here: Going by your numbers (500 posts in total), you have probably less than twenty posts in this region. Also any results in this region are highly sensitive to confounding factors such as posts being generated or assisted with simpler machines than ChatGPT and SO being part of ChatGPT’s training data.

Finally, I could be wrong about my initial assumption about what Hugging Face’s percentage means, but I fail to imagine any interpretation that would arrive at a different conclusion.

While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.

The only way I can make sense of this statement is if you assume that we feed Hugging Face with posts whose distribution of scores is the same as your pre-ChatGPT training data. But that’s hardly the case. Of course, ChatGPT exists now and some people use it to produce answers and many of those score very high on Hugging Face. In the last months, there has been no shortage of posts that score far above 99%. If we moderate only by AI detectors (which we don’t) and set the threshold to 99.75%, we would still find plenty of posts to delete, which I would expect to be mostly true positives. In fact, the last moderated post I was involved with scored 99.98%

Answer 22

The methodology presented is very interesting, but if the company really believes that human beings can't tell the difference between good quality posts that contribute to the site and poor quality or plagiarized posts (given a pattern of posts made by a particular user), then they might as well shut the doors, turn off the lights and go home.

This is not about sifting AI posts from non-AI posts and I can't understand why everyone is so fixated on that aspect. This is about humans judging content and recognizing patterns that indicate a user is trying to game the system and the moderation tools (for everyone, not just elected moderators) not scaling when it becomes easy for lots of people to generate low quality content that requires more than a glance to determine the quality problems.

It's disappointing that y'all are acting like this is some new problem that suddenly manifested because of new technology. SE has been struggling with moderating low quality content (incorrect, plagiarized, unsupported, irrelevant, outdated etc.) and onboarding new users for all of the years I participated on the network. The decline in frequent answerers is not going to be resolved by preventing moderators from moderating. Healthy communities require that spammers, bots and other bad actors be excluded.

The answer is not to tell the people working on the front lines that they can't use their judgement to try to keep things from falling apart. The volunteer moderators know more about how their communities work and what they need than the company will ever be able to determine by data mining (uh, don't get me wrong, data is still valuable). The LLM AI problem is very similar to the sock-puppeteer problem. It's really hard to detect a skilled puppeteer, and pretty easy to detect an unskilled or lazy one. We don't stop moderators from acting on sock puppets because they might alienate someone who is disrupting the site.

Either you trust moderators to use their judgement, or you don't. If you don't trust them, then you need to replace them with something or someone else that you do trust, because almost every aspect of what a moderator needs to do for a site requires a judgement call, especially now that the CoC prohibits "harmful political content" and other vague categories of speech that can only be moderated by human judgment. The answer (in my opinion) is to get back to Stack Exchange's core purpose, which is to crowd source credibility. With the current design, that means that the giant sites have to be segmented somehow into smaller gardens curated by communities of experts that are invested in them.

Answer 23

since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly

The data you show, along with system changes introduced since advent of GPT suggest that most likely cause for content collapse was identified incorrectly.

Much more plausible explanation is that collapse of frequent answerers content is the result of introducing new suspension reason.

Just think of it. Imagine some user willing to gain rep points without sufficient knowledge (for example if in their region these points impact job offers).

Prior to advent of new suspension reason such user could post low quality answers at a fairly high rate... well practically indefinitely - give or take some miniscule impact of soft rate limiting for those of them who are particularly unlucky. Really, what could stop them? Moderators surely could not because they don't intervene in low quality answers. Regular users could not either because of rep penalty.

Since the advent of GPT it is only natural to expect that many of such users will see it as an opportunity to gain reputation with even less effort than they invested before. So instead of their prior low quality answers they started posting GPT dumps. The difference this time is, because of a new suspension reason moderators now can introduce a substantial throttling (suspension) after their first few answers.

So what you've got since the advent of the new suspension reason is that the kind of users who previously could post tens if not hundreds of low quality answers at sufficiently high rate are now throttled by moderators much ealier - like, after posting just 5-10 GPT dumps.

There is your "% of content produced by frequent answerers has started to collapse" and it's not even close to happening "unexpectedly". (and if my assumption of such answerers historically tending to be from regions where rep points matter in the job market holds, there is also your observation about increase of region-specific suspensions)

GPT removal actions are not reasonably defensible

By same token, abrupt interruption of GPT related moderation is not reasonably defensible.

Quite the opposite - delay in such action is reasonably defensible. You have official procedure of introducing policy changes and this procedure justifies reasonable delay. You could (should) use such a delay to properly discuss this matter with moderators and find out how they would prefer to address it.

Answer 24

There are multiple things I have questions/concerns about, but I'd like to focus on just one. This still seems like throwing the baby out with the bathwater with regard to what the announcement/policy actually says.

The problem you have observed and are trying to correct here is negative user growth: active, answer-writing users are leaving the platform at a higher rate than you like, or that you think is sustainable, etc.

The cause you have attributed to that, based on this post, is suspensions for using ChatGPT, which have also increased by a marked amount in the same time frame. In fact, you even point out in the post, "since we greenlit the suspension on first offense..." (or something similar) to try and show a strong positive correlation.

However, your solution to this problem has a severe negative effect in terms of the underlying problem: ChatGPT content; it encourages ChatGPT content to grow and metastasize across the network, unfettered by moderation attempts. If moderators are no longer allowed to delete or suspend for ChatGPT at all, then we are tacitly (or even expressly) saying that ChatGPT-authored contributions are welcome on the site. As you say, Stack Overflow, at least as it exists today, cannot survive such a reality. It will absolutely become the next Quora (or whatever awful Microsoft forum iteration is around at any given time).

---

The CM team took the unusual step in concert with moderators at the advent of the ChatGPT problem of writing a site policy that outright banned the use of that tool in answering questions. The community largely agrees with this decision, as Stack Overflow and its sister sites across the network pride themselves in curated, expert content. As we all know, ChatGPT is neither expert nor curated. And because the bar for using it is so extremely low, and the quality of ChatGPT's English grammar and spelling is so extremely high, the absolute fire hose of ChatGPT-generated content on the network really mandated such an unusual response.

However, you also enabled a pretty severe enforcement option for moderators to mete out for violations of this new policy: suspensions for 1st-time offenses. If a new user gets suspended for their first or second post, I agree that it does make for a pretty unwelcoming experience (whether the suspension was warranted or not), and such users are not likely to stick around.

The metaphor of getting told to permanently or even quasi-permanently shut up as soon as you say something for the first time in a new place tends to have that effect.

So...

The problem I have here is that the new policy doesn't just revert the extreme suspension policy. It purports* to rollback the ChatGPT policy entirely. User retention is suffering, so you are going to undo the policy that helps protect the site's entire raison d'être: to provide free, expertly curated answers to every programming question there is.

Have you considered a half-measure somewhere between "no ChatGPT content allowed--you are banned the first time you use it" and "you get ChatGPT content, you get ChatGPT content, everyone gets ChatGPT content!"?

One thought is reducing the suspension threshold to only occur on the second or third offenses (based on severity, of course), similar to how they are for other infractions, while still allowing mods to delete the offending posts with a 'ChatGPT notice' and work out the details via mod mail if a user wants to appeal.

To reuse the same metaphor from before, this would be saying something for the first time in a new place and told that, while you are allowed to speak, the specific thing you just said is not OK. A far better and more useful piece of feedback than just instantly suspending someone.

I'm glad the suspensions are sometimes (usually?) temporary, but let’s improve on that even more. We are OK with the user, so long as they aren't posting content that isn't allowed. Have moderators show some grace here, and perhaps users will stick around a bit longer. It would certainly be a less harmful iteration, at least, of the policy than what is looming on the horizon, in my opinion.

The main benefit for the site is that ChatGPT copy-and-paste content is still not allowed, and the main benefit for the company is that your metrics of number of new/frequent answerers getting suspended will decrease, likely leading to better (read: less bad) retention numbers. Win-win!

---

* - I say 'purports' because, as far as I am aware, the moderator team (and certainly not the users of the network writ large) doesn't have clear guidance on what they can or cannot allow on the site (a core tenet of Stack Exchange sites/communities, I'm sure I don't have to tell the CM team, has always been that each site gets to decide for itself what kind of questions it finds acceptable). This, aside with the extreme speed at which the marching orders were foisted upon the moderator teams, understandably has a lot of people upset.

Answer 25

In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community.

In other words - removing people from the community makes the community smaller? I guess that's important if the thing you care about most is having a large number of members in the community. But here's the secret that anyone who has spent much time contributing on SE knows: it's the quality of the content that draws people here. And they'll stay here even in the face of negativity and having rules forced on them that they don't understand, so long as this is the place they can come to for high quality Q&A.

Seeking a set of root causes for the contraction in the network’s community size, and an explanation for how it affects different sites/community segments around the network, has been the object of study for dozens of people, and for many months now.

I assume by "dozens of people....for many months now" you mean this is something that Stack Overflow Inc staff have been doing? That sounds like a massive waste of effort. High quality questions and answers, and tooling that enables us to promote high quality questions and answers - that's all it takes. No studies needed. Invest in developers, and have them work on implementing features the community has been begging for. Don't waste time seeking answers to questions about how to grow the community when we've been saying all along it's about the quality. If you build the quality, the quantity will naturally grow. If you spend time focusing on the quantity, then the quality will shrink, and thus the quantity will eventually shrink too until you collapse.

No wonder you have such an antagonistic relationship with the community: you don't ask the right questions, you use terrible methods of gathering data (number of cached drafts, really?!) and you don't bother consulting with the community or even the elected moderators before making sweeping devastating changes. If you carry on like this, you WILL lose your company. I get that you have all this pressure from investors to grow the business, but it's your job to advocate for the best way to do that. And right now, what you're advocating for will do the exact opposite of what's good for the platform, and ultimately what's good for the investors.

Answer 26

Thank you for providing some details, but I have several issues, mainly statistically related. First, there's with

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

At this rate, we would still expect to see around 150 incorrectly-placed suspensions on Stack Overflow in the last six months. This value is still too high for comfort, and ideally, we’d see better rates than this. However, at this level of precision, conversations about how we may put such a system to practice can begin.

There is then a discussion about the HuggingFace’s GPT detector false positive rate, based on its "threshold score". However, this is only an important issue if a tool, like that detector, was a significant factor in any suspension decisions. However, I have not read about any diamond moderator doing this among the multiple related answers and comments. In fact, it seems many of them don't use automated checkers very often (and, if they do, they never rely on just them alone), or even don't use them at all (e.g., sideshowbarker's answer where they state "Background from a striking SO mod who handled 10000+ GPT flags" and "In the thousands of cases I handled, I never ever used any of the detection tools"). Thus, since you've not stated what your actually determined # of false-positives actually is (but with the discussion about the GPT detector apparently being used as a proxy for this instead), it's definitely quite possible, perhaps even relatively likely, that the number of incorrectly-placed suspensions is actually considerably lower than 150.

One other important aspect is you're assuming any diamond moderators using these automated tools are for the purpose of helping them to decide to potentially suspend a user, rather than the opposite. For example, although I don't know if it actually applies to any particular person, a moderator may not be using any automated tool to initially determine whether or not to potentially suspend a member but, only after deciding they should probably be suspended, they then use the automated tools to check, with a suspension occurring only if the results indicate with a sufficiently high confidence that the posts were generated by an AI. In this case, the use of these tools can potentially only decrease the incidence of suspensions, including also any false positives. Would this be an allowable use of those tools?

Note that I believe one important issue, which is a significant, underlying part of the problem here, is the relative importance assigned between having more answers, even at the expense of quality, versus ensuring their quality remains relatively high, even though the number of answers may decrease. The company seems to consider the first option to be more important (e.g., for it to generate more traffic, even at the expense of long-term quality), while I believe that many diamond moderators and relatively active curators (including myself) consider the second aspect to have higher relative importance (in particular, to help ensure the SE sites continue to be relatively high-quality, reliable sources of information).

Next, there's

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform. It produces a coarse estimate, and can’t be used to decide whether a given post or person is posting using GPT. However, in aggregate, it can offer us insight into the ‘true’ rate of GPT posts on the platform.

This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange. In practice, many users save few drafts routinely (for example, because some users copy and paste the answer in from a separate doc, or because they don’t stop writing until they’re ready to post), so it’s the ratio of large draft saves to small draft saves that actually lets us measure volume in practice.

Note that this metric may actually be quite misleading, regarding the number of drafts which may not be that much different between using a ChatGPT result and writing an answer from scratch, such as mentioned in markalex's answer

I've executed a small experiment: I've gone to one of my latest answers, copied the question to ChatGPT, and attempted to create an answer with its output. It took me three drafts to paste and reformat the answer so that it would be adequate to my personal standards.

I don't have the exact number of drafts required to create an initial answer, but since it consisted of three rather conservative in length paragraphs, and ready-made code copied from an IDE, I'm pretty sure that it took me less than 6 Drafts to create the answer.

Also, tripleee's answer indicates that at least some users will learn to get better at evading detection. This would involve things like making more changes to the text (e.g., to have it formatted better, look less like it was generated by AI, etc.), so there would then also be more drafts saved, thus affecting the validity of your metric.

In addition, with your "... was validated against other metrics early on at the peak of the GPT answer rate", the situation may have changed significantly since that initial start, as discussed in my several paragraphs above.

Finally, it seems to me that, overall, you're not appropriately taking into account that correlation does not imply causation.

Answer 27

It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for abuse prevention. If this change were causative, we would expect to see a sudden jump to a new lower level, and a return to the prior well-established rate of decrease. However, we do not see this, a strong indicator of deepening attrition. (We would also expect to see a discontinuity in other metrics not listed – this point is established by a confluence of metrics.)

(Emphasis added)

Can you elaborate on this further? I don't I understand how you've been able to determine that this rate limit would have a fixed decrease and not have a compounding impact on the rate of decline in the number of answers and amount of answers posted per user.

From a very human perspective, I would think this limit would be enormously frustrating. Users cannot participate in almost anything on the site without having reputation. Earning reputation is hard and answering questions was previously one of the most effective ways of doing so. Starting from a 1-reputation account, you need to earn 124 reputation to have the rate limit lifted.

Assuming you earn 1 upvote per answer with no downvotes (which is honestly quite difficult to do in some tags and especially for new users) you would need 13 answers to get above the threshold. The minimum amount of time to post these answers would now be 6 hours (up from a minimum of 36 minutes prior to this change). I would think that this would have a lasting discouraging effect on users beyond a fixed rate decrease based solely on time.

I understand that users "are obviously not going [to ChatGPT] to answer questions" but that doesn't mean that the rate limit is not frustrating or otherwise deterring our users from contributing on our sites.

Answer 28

I've read this twice, pondered for a while on a few of your more vague turns of phrase. Most of my specific constructive thoughts and criticisms have already been covered by other users. I found Kaya3's answer the most comprehensive in terms of breaking down the pitfalls in your data analysis, but there is really something for you to glean from almost all of these answers. For my part I'm going to focus on what seems to be the root drive of your policy:

You're experiencing user attrition, and particularly seeing erosion of "eager answerers". While this is understandably eye-opening from your perspective, I think you're very obviously drawing the wrong conclusions. I won't belabor the details, but suffice to say there is reasonable doubt as to whether the statistical group you're focused on is actually correlated with high-quality answers, or whether the "lost" posts were actually a benefit to the community.

Worse yet, your blundering and unnecessarily imperious attempt at "solving the problem" is actually inflaming the true cause of user attrition: waning community trust in the leadership of SE inc. If you want to reverse the tides, that's the area you need to focus on. The solution to all your woes is as simple as it is hard:

Humility.

You need to admit you were wrong. Acknowledge that the strike was justified by your error, and roll back your decision. Then reapproach this the right way, with transparency and community involvement from the outset.

Otherwise the dedicated and skilled users that make this site are just going to keep hemorrhaging away. Which is just a shame, honestly.

Answer 29

As to the future of the site part, I think it's imperative to be able to detect AI generated answers. Whilst it is true that voting can take care of hallucinations and plain incorrect answers, the problem lies in one of the core principles of the site: gamification through tying credibility to reputation.

Getting reputation for AI generated content is akin to getting reputation for plagiarised content: it is not your own. Reputation gives you on-site credibility and trust in the form of privileges. But not only that, more crucially, it gives you credibility to other users. An answer posted by a user with 100k reputation is trusted more by readers than when the exact same answer would be posted by a user with 1 reputation. Thus, at some point people would be seen as credible and trustworthy, when in fact it should be the AI getting the reputation.

This is in fact a known problem and has happened in a slightly different form, when a US teenager without speaking any Scots started editing Wikipedia articles in Scots . They got moderation privileges, allowing them to e.g. roll back correcting edits on their posts. At some point, even translation algorithms, such as Google Translate, started to use their, partly/mostly incorrect, posts.

Thus, if anything, fully AI generated content should still be disallowed and a detection mechanism should be found.

Note that I am not completely against the use of AI in all forms. Especially for non-native speakers having an AI clear up their language to make it more understandable and grammatically correct is a good use in my opinion (much like spellcheckers have been doing the past decades). Where the line should be drawn is up to discussion; e.g. is a user answering a programming question with their own code, but having an AI writing the complete explanation acceptable?

Answer 30

This is basically the same thing Mithical already said, and it is the first thing that came to mind as soon as I started to read your very long post. That said, I am posting this version to give a sightly different approach to the issue.

Your assumptions so far:

• ChatGPT detection tools have an unacceptable error rate. References:

  1- We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives
  2- Automated GPT detectors have unusable error rates on the platform - this very post

• Moderators' own judgment has an unacceptable error rate too. References: multiple moderators pointing out that the private version of the policy, that very conveniently for you the userbase will never see, doesn't just forbid the use of tools. Since I would not be surprised to see retaliation actions against those individuals you won't get a list here.

Logical conclusions from your premises:

• Since neither automated or human made judgment works, it is impossible to determine if any post is the product of automated generation without the author disclosing that.

Your next claims:

• The company was able to measure the rate of generated content on the site.
• The company was able to assume the expected number of false positive bans on the site based on that rate

Both claims seem to be founded in the strategy you used to detect/estimate the number of AI generated posts.

This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange.

On paper, this may work. There are two problems though.

• This idea, which based on your claims work better than any judgement the brains of more than a hundred mods managed to do, is coincidentally the first thing that came to mind when I was trying to imagine what the tells-by of a copy-and-paste post could be. I therefore assume that unless I am far smarter than you, the mods and everyone else that is also what most will have though of too, if anything else because this is exactly how the "are you a human" check usually work on sites: if someone is making some action too fast, they are probably not an human.
• Mods have been telling you that quite often users posting generated content try to edit it before posting to remove the more blatant commonly seen recognizable signs of automated generation (for example the usual final line that often sound like "it is worth to notice that the above is just ... and not an accurate ....") and that obviously increases the time spent on a post.

For these two reasons alone, I am not really convinced that your data provides much more value than the mods' own analysis and actually imho looks like a pretty disingenuous oversimplification that tries to prove causality while ignoring a ton of other variables, but it is still better than no data.

I therefore once again propose you an experiment: take one of the posts that based on your analysis should be a false positive, identify the mod who claimed it was a real positive, post an anonymized version of the post so that users can also see what the content looked like, and then have both parties explain how they came to their conclusions.

Corollary

I appreciate the shift in tone in this post. Sadly, I still have to consider this as a part of a bigger picture so I still find it important to write the following.

It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate.

The sentiment of this passage does not align at all with your actions. Once again you immediately jumped on the wagon to post what your own voluntary moderators apparently see as a direct disparaging of their work to a media press site. Those posts were not removed or rectified after multiple requests to do so, and therefore you current words feel as empty as they can possibly be. You don't know how to start to rectify the misrepresentation you gave because "once on the internet, it is forever"? YOUR PROBLEM.

Also, while we are at it, this is not the first time this happened. Remember someone called... Monica Cellio? How many years ago did that happen again?

Wonder what? That [redacted] article... it is still there.. Lucky, apparently the Register had the heart to follow up on that story, but neither article seems something the company requested to clarify things. It is my understanding therefore that you were perfectly fine with never taking action to tell the press that you were wrong.

• Flak overflow: Barrage of criticism prompts very public Stack Overflow apology
• Stack Overflow makes peace with ousted moderator, wants to start New Year with 2020 vision on codes of conduct

I therefore can't help but ask: how is that every time some "mishap" of this size happens we end up with an article posted somewhere that manages to portrait the company as the Heroes Of Light (tm) that fight for a better world against the evil forces of the Malevolent Community of Evils (tm)? I seriously doubt that the Register or any other site spends its day looking at Stack just to post a new story, so I assume that YOU are the ones to contact them and request an article, not the opposite. This obviously opens another question: what do you hope to gain from those coincidentally wrong representations of the issue at hand and the mods?