69

In a meeting with some moderators last week, I committed to releasing the data sets from our initial studies around the efficacy and false positive rates of ChatGPT detectors to them. Tuesday afternoon, we did so. This post contains as much information from that discussion as we are able to share in public, and represents the joint efforts of my community leadership team and other members of the community team.

We will open this post with a discussion about the baseline error rates in GPT detectors. Then, we are going to discuss some data coming out of Stack Overflow that can rule out many hypotheses about root causes for contraction.

We have done our best to look through the data and understand what hypotheses could hold water. To this end, we are engineers and community managers, not scientists. We will not claim these are formal studies that produce scientific data, nor will we claim that the data sources are perfect or uncontestable. Rather, that they are operational data sufficient to produce engineering decisions: the same standard that we use as a Community Management team. We’d like to be clear: nothing in this post is intended to cast aspersions on the moderator team of any site. We respect the work that they are doing to address this problem, as challenging as it may be.

This post will be a lot to take in. And even then, it’s a fraction of the work put into analyzing this question across multiple teams in the company. However, it’s a piece of the work we believe may help clarify some facts that have heretofore been obfuscated.

The tl;dr: Summarizing the results

  • The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small.
  • The rate of suspension for frequent answerers rose by a factor of 16 from ~0.4% to ~6.6% since the rise of GPT, and has held steady since its release.
  • If every GPT poster posted exactly three answers and were suspended within three weeks, this would imply a minimum GPT post rate of 330 answers per week on Stack Overflow; in practice, we would expect a significantly greater quantity. Measurements of GPT occurrences on the platform imply fewer than 100 GPT answers per week, in disagreement with this rate, implying the existence of many false positive detections.
  • The suspensions issued appear to have a significant and measurable impact on the demography and volume of answerers on the site, preferentially excluding frequent answerers.
  • GPT detectors continue to be ineffective for detecting GPT answers on the platform.

Automated GPT detectors have unusable error rates on the platform

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

At this rate, we would still expect to see around 150 incorrectly-placed suspensions on Stack Overflow in the last six months. This value is still too high for comfort, and ideally, we’d see better rates than this. However, at this level of precision, conversations about how we may put such a system to practice can begin.

HuggingFace’s GPT detector assigns a “threshold” score from 0 to 1 of the post being authored by GPT. Based on a random sample of 500 answers from before GPT’s release, each answer being no less than 400 characters long, the false positive rates follow. At the 0.50 detection threshold, around 1-in-5.5 posts are falsely detected. At the 0.90 detection threshold, around 1-in-13 posts are falsely detected.

A chart showing the false positive rate of HuggingFace's GPT detector by threshold.

Keen readers will note that 1-in-5.5 is a fair bit better than the 1-in-3 false detection rate we originally noticed. This is because we used two detectors during the original survey: ZeroGPT and HuggingFace’s detector. The new value of 18% +/- 3.4% false positives from HuggingFace falls within the 95% CI of the original smaller survey (27% +/- 8.9%).

While it is theoretically possible to achieve better baseline error rates than 1-in-20 by picking higher thresholds, the efficacy of the detector may fall off considerably. A detector that does not produce false positives is no good if it also produces no true positives.

For a 1-in-20 false positive rate, the detector threshold needs to be 0.97. Satisfactory rates (less than 1-in-50 false detections) could not be achieved in this test until the detection threshold was set to 0.9975, but the error due to low sample size at this threshold is considerable. Narrowing this window to a more precise value would require significantly more data points than we collected in this survey. At this point, we can’t endorse usage of this service either as a tool for discriminating AI-generated posts or as a tool for validating suspicions.


We thought it would be helpful to include some more discussion and context. Rather a lot of discussion and context, actually. Over the last few months, folks within the company have been working to answer the question, “What has been taking place in the data coming out of Stack Overflow since GPT’s release?” Considerable changes have taken place in demography: users use Stack Overflow at different rates than they did before; different sorts of users use Stack Overflow; and questions and answers reach different outcomes.

What follows is a single piece of the puzzle. The broader picture informs the company’s operational decision-making processes. This piece has a part to play, but only a part. It is not a full explanation of the question, nor is it a blame-first investigation. Neither can it explain every change in the data on Stack Overflow that we see. However, what it reveals calls into question whether GPT detection is possible or effective on the platform, and by proxy, whether a high false positive rate may be partially responsible for a decline in answerers and answerer retention on the platform.

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release

While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

A chart showing the relative change in the number of users who posted 3 or more answers a week, normalized to October 2022.

It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for abuse prevention. If this change were causative, we would expect to see a sudden jump to a new lower level, and a return to the prior well-established rate of decrease. However, we do not see this, a strong indicator of deepening attrition. (We would also expect to see a discontinuity in other metrics not listed – this point is established by a confluence of metrics.)

The number of high-volume answerers has seen a -2.4% average week-over-week decrease since December. In total, there has been a -42% contraction in high-volume answerers since the release of GPT. While users may go to ChatGPT to ask their questions, they are obviously not going there to answer questions. Therefore we can consider that…

The total volume of questions available to frequent answerers continues to rise

The alternative hypothesis for the above chart is that the number of questions available for users to answer has simply fallen, on account of question rates falling. This claim is hard to swallow given current data. What follows is the # of available questions posted per week, divided by the number of users who post 3 or more answers in a week. If this hypothesis were true, then this value should be falling, or at least not rising as quickly, as active users are crunched by a collapsing question rate.

However, even though the question rate is collapsing, there are still plenty of available questions for users to answer. It’s hard to tell exactly what the carrying capacity is for this number, but we certainly know that it is somewhere between 3 and today’s value of ~18. In other words, if users were willing and able to answer more questions, the evidence strongly suggests that they could in fact do so. Of course, on short timescales, factors like % of questions closed and % of questions deleted could cause fluctuations in this value. However, in the long view, the trend is clear and not violated in the post-GPT region.

A chart showing the total number of questions asked per week, divided by the number of users who answer 3 or more questions per week.

This leaves one question remaining: Where are they going?

7% of the people who post 3 or more answers in a week are suspended within three weeks

…and this value has held reasonably stationary since these suspensions were enacted.

In the 16 weeks before we enabled moderators to suspend users for GPT users, around 0.4% of users who posted >2 answers per week were suspended within three weeks. After we allowed GPT suspensions on first offense, 6.6% of users who posted >2 answers in a given week were suspended within three weeks, a 16-fold increase.

(Note that suspensions issued for users who posted fewer than three answers are not counted here; nor are suspensions issued three weeks after the user answered several questions.)

A chart showing the percent of users who post 3 or more answers in a given week, and are suspended within 21 days.

In a given week, 32% of the people who post 3 or more answers also did so during one of the last eight weeks. Supposing that these suspensions are distributed across users regardless of GPT usage, we should see (by rough order of magnitude) a ~2.2% decrease in the actual volume of users who post three or more answers in a given week. And indeed, real data (2.4%) are quite close to the theoretical percentage.

Instead suppose that no more than 1-in-50 of the people who were suspended for GPT usage were not actually using GPT. In order for this to be true, a large volume of users would have needed to immediately convert from being regular users to ChatGPT users; and then, a high rate of conversion would have to be sustained over time, long after the release of ChatGPT, in order to sustain present suspension rates.

Regardless of the above, no Community Manager will tell you that removing 7% of the users who try to actively participate in a community per week is remotely tenable for a healthy community. Supposing every suspension is accurate, the magnitude raises serious concerns about long-term sustainability for the site.

It is worth acknowledging that we did give explicit permission to suspend on 1st offense for GPT usage. However, even in the absence of these policies, this value alone rings a deafening number of alarm bells for potential false positive detections and contributor loss alike. If there are false positive detections, even removing users’ content incorrectly could prove harmful.

Users who post 3 or more answers in a given week produce about half the answers

First, a short detour. We are going to focus on users who post 3 or more answers in a given week for much of this post. While it may seem a bit odd to look only at this segment of users (and it is, of course, not the only segment of users we investigated), there is a rationale for doing so.

Users who post answers more than twice a week used to comprise about half the content produced on Stack Overflow. However, since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly. Given the absence of question scarcity as a factor for answerers (note the above chart), the clear inference is that a large portion of frequent answerers are leaving the site, or the site is suddenly not effective at retaining new frequent answerers.

A chart showing the percentage of answers posted by users who post three or more answers that week. It is relatively flat until late November 2023, when it starts to fall steeply.

(It is worth noting that GPT messages and suspensions disproportionately skew towards users who have posted more than two answers in a week, but we can’t discuss this in more detail publicly without revealing the details of how GPT posts are detected on the platform.)

Yet, at the same time, actual GPT posts on the site have fallen continuously since release

What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform. It produces a coarse estimate, and can’t be used to decide whether a given post or person is posting using GPT. However, in aggregate, it can offer us insight into the ‘true’ rate of GPT posts on the platform.

This metric is based around the number of drafts a user has saved before posting their answer. Stack Exchange systems automatically save a draft copy of a user’s post to a cache location several seconds after they stop typing, with no further user input necessary. In principle, if people are copying and pasting answers out of services like GPT, then they won’t save as many drafts as people who write answers within Stack Exchange. In practice, many users save few drafts routinely (for example, because some users copy and paste the answer in from a separate doc, or because they don’t stop writing until they’re ready to post), so it’s the ratio of large draft saves to small draft saves that actually lets us measure volume in practice.

This metric is sensitive to noise, but was validated against other metrics early on at the peak of the GPT answer rate. Additionally, it matches the volume of actual actions taken on the platform early on after the release of GPT.

This allows us to get a good, albeit coarse, understanding of the overall population trends in GPT answers. However, it does not allow us to identify which specific answers were drafted by or with assistance of GPT, because quite a lot of answers are ‘normally’ posted after saving very few drafts.

Between January 2023 and March 2023, it appeared that we were going to hit a floor of around 700 GPT-generated posts per week and stay there, which would have been quite a bad outcome for the site’s general health. However, as time progressed, it became clear that the actual volume of GPT answers was falling precipitously - even as a percentage of total answers (which, yes, is also falling comparably).

The number of GPT posts created week-over-week.

Some folks have asked us why this metric is capable of reporting negative numbers. The condensed answer is that the metric has noise. If the true value is zero, sometimes it will report a value higher than zero, and sometimes a value lower than zero. Since we know how much noise the metric has, we know what the largest value for GPT-suspect posts should be.

At the initial holding point, GPT answers reflected around 2.5% of the answers posted on Stack Overflow. Rates hovered around this level for several weeks, but such rates were not sustainable and the percentage of posts authored by GPT began to fall. The following chart shows the expected % of answers posted in a given week that are GPT-suspect.

These days, however, it’s clear that the rate of GPT answers on Stack Overflow is extremely small. In fact, it is difficult to estimate the true number of GPT answers posted on Stack Overflow for this reason. Based on the data, we would hazard a guess that Stack Overflow currently sees 10-15 GPT answers in the typical day, or 70-100 answers per week. There is room for error due to the inherent uncertainty in the measurement method, but not room for magnitudes of error. We can therefore say that the rate of GPT posts is far less than it was two months ago, and then it is less than it was two months before that.

So, under what conditions could it be the case that roughly 7% of frequent answerers on the site are still posting via ChatGPT? If this were the case, the site should be seeing at least 330 GPT answers per week, but the rate estimate is not close. This also assumes every user who posts GPT answers are caught, and that GPT answerers post no more or less than three answers in a given week. In practice, the site should be seeing significantly more than 330 GPT answers per week to support this suspension rate.

It could be possible, either due to severe measurement error or due to an unexpected change in user behavior that obfuscates GPT usage using this method. But the evidence for this viewpoint does not appear strong.

Many of GPT appeals sent to the Stack Exchange support inbox could not be reasonably substantiated

While we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox, we could characterize some of them.

As a platform, we have an obligation to ensure that moderation actions taken on the Stack Exchange network are accurate and can be verified upon review if we need to do so. We need to be able to see, understand, and assess whether the actions taken were correct. It therefore needs to be said that we are very rarely, if ever, in the position where we cannot do so. In all other areas for which we receive suspension appeals, moderator actions are easily verified and double-checked the overwhelming majority of the time. It is rare and notable if we are ever in the position of overturning a moderator’s decision due to insufficient or contradictory evidence.

So, when we say that many of the GPT appeals we receive could not be substantiated on review, please keep in mind that our baseline value for this is zero, and it’s been that way for years. It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

It is worth noting that we don’t believe this discrepancy is due to moderator misconduct or malfeasance. Our goal here is not to accuse moderators of wrongdoing or poor judgment. We respect the fact that they were, and are, working under difficult circumstances to achieve a goal we appreciate. Rather, it is important to remember that the company has a strong need to ensure moderator actions are verifiable and justifiable. And, on this point, we need to seriously consider whether these processes, in whatever form they are taking, do what they should.

True false positive rate of moderator actions

We want to clear up a particularly important point. We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these methods to the platform as a whole.

Under this assumption, it is impossible for us to generate a list of cases where we know moderators have made a mistake. If we were to do so, it would imply that we have a method by which we can know that the incorrect action was taken. If we could do this, we would share our methods in a heartbeat in the form of guidance to the moderator teams across the network, and then we’d carry on with things as normal. Instead, the most we can do is state that we just can’t tell. We lack the tools to verify wrongdoing on the part of a user who has been removed, messaged, or had their content deleted, and this is a serious problem.

This is a critical opportunity for us to inspect the processes that are contributing to the outlook for the site, and contextualize them in the overall state and progression of the network.

Stringing it all together

Look, the truth is, there are going to be many hypotheses about what could be taking place in the data on Stack Overflow. Heck, we’ve had several large analytics teams pore over the SO and SE data for any sorts of anomalies or possibilities when it comes to participation contraction. While it is absolutely true that Stack Overflow is losing users faster than it is gaining users, none of the hypotheses generated by the company can explain away the relationship between % of frequent answerer suspensions and the decrease in frequent answerers, in the context of falling actual GPT post rates.

In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community. The scalar factor here varies considerably from community to community, but it must be taken into account here as well. Deleting their contributions for reasons they feel unjust, or commenting on behavior that is not present, appears to have a similar effect to being suspended, and differences are often small when it comes to long-term user outcomes (such as the user leaving, and/or potentially other people leaving with them).

Is it still possible that the proportion of false positives is small? Maybe so – it can’t be completely eliminated at this time. Direct causative data are not possible to obtain for this problem. But for that to be true, it would require some very strange user behavior en masse around answering, by users who were otherwise answering questions normally. These are behaviors we do not have an organic explanation for after months of exploration, even under the scenario where question demand contracts at a rapid pace. Again, it’s possible, but the evidence is not favorable. We are forced to look at all possible exogenous causes of user attrition - including whether logic internal to the network needs to be altered (and we have made changes here, as well).

Finally, it needs to be said that the analysis presented here is far from the complete analysis we have conducted internally. These are pieces of a larger puzzle, selected because they best express our meaning (and contain no proprietary data). Seeking a set of root causes for the contraction in the network’s community size, and an explanation for how it affects different sites/community segments around the network, has been the object of study for dozens of people, and for many months now.

We hope by now the ultimate point here is clear: Suppose we are right in this assessment and GPT removal actions are not reasonably defensible. How long can we afford to wait? To what extent can we continue to risk the network’s short-term integrity against the long-term risks of the current environment? Any good community management policy must balance the risk of action against the risk of inaction in any given situation, and the evidence does not presently favor inaction.

What we know, right now, is that the current situation is untenable. We have real, justified concerns for the survival of the network. We’re not saying this to invoke despair, or to imply that these decisions are overly rushed. There are silver linings here, and there is significant potential for future growth and a more stable long-term path. The onus is on us to find a way forward that protects the communities and network as a whole.

If there is no future for a network where mods can’t assess posts on the basis of GPT authorship (where we would be after this policy), and if there is no future for a network where mods can assess posts on the basis of GPT authorship (where we are today), then there is no future for the network at all. Yet somehow, our adjacent communities are making this work. In this deadlock, something has to give. At the end of the day, it all boils down to this: We have to walk the middle path.

60
  • 121
    I see we're once again ignoring requests internally pointing out severe errors in the methodology
    – Zoe
    Jun 7, 2023 at 19:33
  • 80
    Why didn't you expose your metric for GPT-ness (the number of drafts) to the moderators and let them factor it in to their own decisions instead of declaring it to be the be-all end-all and override everyone's judgement?
    – pppery
    Jun 7, 2023 at 19:47
  • 56
    Although it is an interesting analysis, something is missing. I asked about this on the Stack Moderators Team and didn't get an answer. What, exactly, is the problem you are attempting to solve? There may be more than one problem. For each problem, can you express that problem in a single sentence or question? What questions are you trying to answer or what problems are you solving? My current thinking is that there may be assumptions inherent in the question that do not hold true. (For staff/mods, the Team post I reference.) Jun 7, 2023 at 19:55
  • 64
    Could you please explain why you didn't mention the actual numbers or percentages of appeals that you could not verify? I don't think "many" cuts it here, I can imagine users coming up with wildly different percentages here when interpreting this. Jun 7, 2023 at 19:56
  • 86
    Although mods have made clear several times that they are not relying on detection software like this, and several have stated that they never use the software at all, I challenge the characterization that this analysis shows the detector is not useful. It shows that an acceptable false-positive rate can be achieved when used on single questions as long as the threshold is set to 0.9975. That's a high threshold, but also one frequently exceeded by AI-generated content fed to the tool. If you tested instead sets of, say, 3 posts by the same person, you might find it improves substantially. Jun 7, 2023 at 19:58
  • 120
    Thank you very much for posting this publicly. Doing so will, hopefully, improve the overall discussion. It will allow a much wider audience to review the procedures used and conclusions drawn, which is a good thing overall, regardless of any position or opinion which we might or might not individually have.
    – Makyen
    Jun 7, 2023 at 20:08
  • 138
    I've found one user who has posted 14 obviously ChatGPT answers in the last hour. So that pretty much accounts for your entire daily quota even if no-one else posts anything.
    – DavidW
    Jun 7, 2023 at 20:35
  • 41
    It's very strange that you released your internal metric (thus ensuring that you can't use it anymore reliably), but decided to censor the actual number of suspensions in question. Are you worried that the public won't think it's large enough to be as big a deal as you are making it?
    – Chris
    Jun 7, 2023 at 20:50
  • 74
    Philippe, I was really hoping you would show some data and linked outcomes, as per all the requests in TL, but you have really just stuck to your list of items that you are somehow assuming are correlated, while not providing any evidence of any linkage. I would be more than happy to attest that every post I deleted, and every account suspended for misuse of ChatGPT was accurate. Further I am confident the quality of those posts was terrible. Sure, I don't moderate SO, but I do moderate a large number of smaller sites - and your new policy will only cause damage.
    – Rory Alsop
    Jun 7, 2023 at 21:28
  • 63
    "7% of the people who post 3 or more answers in a week are suspended within three weeks" <- I think one big piece of data needed here is how many of these are new accounts. Right now it fits both the "overzealous mods are banning longtime community members" and "a few bored GPT fans keep making new SE burner accounts". Ban evasion is... a common thing online
    – Kaia
    Jun 7, 2023 at 22:11
  • 141
    Thanks for posting this. I'll post detailed thoughts once the June data dump is released and I can sanity-check some of these charts, but for now I would strongly recommend y'all work on better record keeping - I was NOT expecting to read, "we could not recover all of the GPT suspension appeals sent to the Stack Exchange inbox". Y'all are the backstop for these things - it is crucial that you take them seriously. If you're losing appeals, that is extremely worrying. Also, unless moderators have gotten a lot better in 3 years, a baseline of 0 is unrealistic - which also suggests data loss. 😬
    – Shog9
    Jun 7, 2023 at 23:32
  • 42
    @Shog9 - to clarify the language used here, “recover” does not mean the remediation of a data loss in this case. “Recover” means “one of my staff went through our support inbox and attempted to pull out all the ones related to GPT”. To the very best of my knowledge, there are no “lost appeals” and there has been no data loss. (Our data protection and retention policies are pretty strict, and there’s no question that regulators and lawyers would probably be involved in that case.)
    – Philippe StaffMod
    Jun 8, 2023 at 1:27
  • 48
    That's a relief, @Philippe - might wanna rephrase that section, I'm sure I'm not the only one to interpret it that way! FWIW, should be possible to query mod messages for replies - that's not going to give you the same information, but I've found it to be a pretty good indicator of exceptional circumstances. Threads with more than 1 reply nearly always warrant a look...
    – Shog9
    Jun 8, 2023 at 2:56
  • 56
    "The actual rate at which GPT posts are made on Stack Exchange has fallen continuously since its release, and is now very small." (1) How do you know they are GPT posts? Did you use tools to check? How do you know the tools are accurate? (2) Did you include those GPT posts with added spam links in your data? Many GPT posts were deleted and accounts were suspended because they were spam posts, not because of GPT.
    – Nobody
    Jun 8, 2023 at 7:22
  • 30
    Question I need to answer now is whether I still care, @Chris. A walled garden isn't the Stack Overflow I signed onto almost 15 years ago; it is the antithesis of that. I'm gonna take some time to reflect.
    – Shog9
    Jun 9, 2023 at 18:50

37 Answers 37

1
2
30

Just some random points in no particular order, which I think could be relevant:

  • You spent much time speaking about the false positives of a specific GPT detector, but usually one displays precision-recall curves that give a more complete picture (the chosen threshold is the parameter to walk along that curve). Why didn't you do it (mixing GPT generated content with pre 2023 content in a fixed ratio)? What is the recall at a desired precision?
  • The issues with specifically SO are many, many years old. Now they seem to accelerate a bit but why do you think that not the same underlying forces are still mainly responsible? Activity on SO is shrinking since 2016 or earlier (if you count positively scored answers and questions for example). Maybe users simply don't like the platform anymore (after GPT as before)?
  • Why the odd concentration on answerers with more than 3 answers per week? The ratio of new questions to new answers is only slowly declining. Does it mean that infrequent answerers are saving the day?
  • Have you measured other properties of the draft process (amount of text changed, time between drafts saved, number of clipboard operations, ...)? Or the number of edits afterwards? Could this also be used? Do you see something there?
  • How does this high/low draft saves ratio correlate with the suspensions? Did moderators indeed suspend the people you think were posting GPT messages in the beginning?
  • You seem to assume that there are only very few people if at all posting GPT output on SO now. Any idea why people stopped? It may not be that novel anymore but the temptation should still exist. If people stopped then maybe because of the suspensions? But now the suspensions stopped, a reversal might be expected. Can you show how the draft save ratio developed or the number of answers since last week?
  • It's not a good idea to let people post GPT content. There is no guarantee that the solutions work. Sometimes they do, often enough they don't. The quality is rather low currently. Now couple this with difficult to understand questions and what you get is only noise. More answers can actually reduce the usefulness of the content in SO if they aren't helping. What does the traffic say? Do any high quality Q&A pairs are getting generated still or does SO live exclusively from the past?
  • SO excels over AI not so much for the simple questions but for the complex problems where you really need to be an expert, not just a very, very diligent but slightly stupid word cruncher who happens to be also able to speak Oxford English. So unless you want to compete with GPT (very, very challenging) you better go for the very high quality niche.
  • And finally, I was here for the library of programming knowledge. I saw the data dumps as an insurance. You took that away and I don't trust it will come back. For me it's the last straw. I don't feel obligated to stay either. And I stayed for a long time and probably posted more than I should have and wasted more time than I should have but I also enjoyed reading contributions from so many nice persons, that helped me and always made me thinking. I'm very thankful for that.
3
  • I agree with the first few points - but on the latest few points re. quality of SO answers vs ChatGPT - I think that would be correct only for ChatGPT3.5 or GPT3, not ChatGPT4. For me ChatGPT4 was able to correctly answer 83% of my questions on Cross Validated that never received an answer & out of my questions only 25% of my questions were ever answered correctly by other users (a further 15% was eventually answered by myself). See here for some examples & results. meta.stackexchange.com/questions/387575/… Jun 20, 2023 at 8:02
  • 1
    So a big part of the explanation of the declining traffic may be that (1) ChatGPT now solves most questions faster & better than SO and (2) users may find ChatGPT a more welcoming platform than SO - I often see people comment "ChatGPT is like Stack Overflow, only less infuriating" (whether that is right or wrong) - ChatGPT always responds, never closes down questions for being duplicates & is always friendly and helpful. Many users do not have the same perception of SO unfortunately. Jun 20, 2023 at 8:07
  • 1
    "ChatGPT always responds, never closes down questions for being duplicates & is always friendly and helpful. Many users do not have the same perception of SO unfortunately." - they shouldn't, because we do close questions as duplicates (which is helpful) and it is not in the job description to indulge nonsense or accommodate every question. It is in the job description to be correct and to actually think. A pity for those who, reciprocally, indulge AI hallucinations. Aug 4, 2023 at 5:50
26

It is exceptionally strange for us to look at a moderator’s action and find ourselves unable to verify it – yet this is the situation we are frequently in with respect to GPT.

This raised a question for me: Are the people who review moderator actions people who know the fields of the SE sites they take place in?

If so, then that's a valid concern. I don't know if you notify those moderators, or if they then respond to you. And I understand there's limited bandwidth for going back and forth over each case, but I am curious if you've been able to explore the disconnect.

But if the reviewers are generalists, and don't know the fields, then I wonder if there's a gap here. Because ChatGPT output is tuned to appear valid - they have worn away most of the "tells". I would not be surprised if they can't verify the GPT identification - there would be an entirely new set of skills required to do so, and people would need training to gain them.

I should mention that I'm mostly active in math.se, and we frequently find ourselves completely unqualified to judge the quality of this or that question. We had one poster that dropped like eight answers in less than an hour. I was able to find one with an out-and-out error that you could find if you paid close attention, one that completely missed the point of the question but was oddly long-winded, but most of the others, had I seen them in isolation and didn't have other users spotting them, I wouldn't have been able to reliably verify a claim that they were ChatGPT.

Discerning ChatGPT is hard. I can't say much about the behemoth that Stack Overflow is, where you have to resort to stats-based speculation, but at least in the SE sites where I hang out, I can't see how you're offering an improvement on the moderators' judgements that we currently rely on.

5
  • 11
    A bit more info about whether and how Stack Overflow moderators were notified can be found here, in a collaboratively-written answer by the SO mod team. And, no, the staff who reviewed these appeals were not subject-matter experts in the relevant fields. To a large extent, I think we'd agree that is fine; even SO mods don't know every language/FW/tool that is on-topic for the site. But answers that are "not even wrong" could certainly explain the edge cases where staff were unable to verify a mods' work. (See also: meta.stackexchange.com/a/389844) Jun 8, 2023 at 8:35
  • @This_is_NOT_a_forum - thanks for the general clean-up, but I had intended the word "voluble", (which I misspelled). Replaced it with something less obscure.
    – JonathanZ
    Jun 8, 2023 at 20:05
  • The kind of people who report AI-generated posts on SO tend to be subject-matter experts though. I have reported some 3-4 suspected ones, all had the nature of "cocksure gibberish" which would be hard to call out based on the technical content alone, if you weren't familiar with the specific programming language.
    – Lundin
    Jun 9, 2023 at 11:17
  • 1
    @Lundin - Exactly. And the moderators are subject matter experts too (at least I've always assumed so). But then they're saying there's this disconnect, where the people who review the mods evaluation don't see it. So I'm saying this is what to investigate, instead of guessing about correlations. And in fact there's a better answer in here, I think from actual mods, that much more explicitly says "start letting us know when this happens, so we can start understanding and fixing it".
    – JonathanZ
    Jun 9, 2023 at 12:45
  • @JonathanZonstrike Well yeah they are often domain experts too but not necessarily of the particular topic they are currently moderating. If you are lucky, the Java expert mod is reviewing the Java-related flag, but I don't think they can afford the luxury of always passing such flags on to the moderator most suited - there are way too many flags and way too many topics. But I would think that flags that actually require domain knowledge to moderate are few. AI-generated content - probably not applicable, since you'd recognize these by other means than examining the technical content.
    – Lundin
    Jun 9, 2023 at 12:52
23

All statistics and chances of chances of false positives aside...

I find it very hard to believe that moderators of any site in the network would suspend a user for posting high quality, non-plagiarized content for the benefit of others.

The kind of users suspended will extremely likely be the ones who post low quality content, AI-generated or not.

Having come to that conclusion, then is it really such a big deal if moderators accidentally suspend a user accusing them for posing AI-generated content, while they had in fact written their low-quality, possibly incorrect, of no-value-to-anyone content all by themselves? Or perhaps they had just blatantly plagiarized it without the aid of AI.

Even if such suspensions would occasionally happen, I'd say that's perfectly acceptable collateral damage to keep the content quality up. 1 week suspension is not the end of the world. Concerned users can protect themselves from this by writing better answers. Ideally with actual references to sources, which you won't find in AI-generated or plagiarized answers.

And here comes the quantity vs quality discussion. Those users posting low quality content do generate site traffic. Who cares about quality of the product over time when we can have short-term profits!

5
  • 5
    "Welcome wagon" aside, the experience of getting suspended for trying to be helpful will surely suck; so in that sense, you will want to avoid false positives even at some cost. But ultimately the pressure to suspend violators seems to have been coming mainly from the company, and should be tangential to the core question.
    – tripleee
    Jun 9, 2023 at 11:16
  • 11
    @tripleee "the experience of getting suspended for trying to be helpful will surely suck" If so then why isn't the review audits on SO ever fixed? They have been broken since the dawn of time and I find that a much more serious problem than the one discussed here. If SO the company had spent the same energy into fixing review audits as in discussing & blogging about AI and AI-related moderation, those audits would be pure gold by now. But no, review-suspending reviewers who step on some particular bad audit mine was always regarded as acceptable collateral damage.
    – Lundin
    Jun 9, 2023 at 11:21
  • 5
    What you have hard time believing in is an actual policy in math, in responce to users posting their homework without an attempt to solve it and other users posting high quality original solutions. But this has nothing to do with AI and/or the current debate. Jun 9, 2023 at 14:44
  • Re "in math": Presuming the Mathematics site, Mathematics is a homework site. If demonstrated work is required, why isn't it enforced? I had the impression that homework dumps were happily accepted (that is, there are a lot of eager answerers (with whatever motive, but most likely sweet, sweet reputation points)). Jun 11, 2023 at 10:44
  • 1
    @This_is_NOT_a_forum: You are mistaken about MSE being a homework site. Yes, some problems are homework (personally, I try to avoid answering these) but many are not. Some are deep math questions which would be also appropriate for mathoverflow. Some are coming from applied side, etc. As for enforcement, mods are trying, but it is hard. Jun 11, 2023 at 23:14
19

Aside from the post authorship question, let's turn our focus on user behavior, in hopes that it might explain some anomalies.

Here are some motivation factors that strongly influence people's behaviour on this platform:

  • This platform is addictive, as it was designed to be so. For this platform to work well, questions need relatively rapid interaction, and relatively rapidly produced answers. This can be facilitated by making the platform addictive, so that people keep hanging around, and remain available for interaction.

    How do addicts behave?

    First of all, they may spend more of their time with these sites than they reasonably intend. Accordingly:

    • You will see behavior of frequent engagement, aimed at soothing their urges for whatever they gain from participation.

    • On the other end of this tension, you will regularly see people attempting to regain control over their time, and thus, life, breaking away from the platform, and spending their time elsewehre.

      Then, after a few months of hiatus, they may return again.

  • The reputation system can impose a significant frustration, even if it's well-intended for the purpose of fostering and facilitating disciplined and responsible behavior.

    Due to the limited privileges associated with low rep, from their first day on this platform, new users are chasing after the value that they invested time-wise, to somehow get to the point where they can participate with dignity. That, for me, comes at a minimum of 1000 points, which unlocks the ability to see up-/downvotes breakdowns on posts.

    Anything below 1000 points is a painful place to be in, and thus people will be urged to earn those points as fast as possible.

  • Stack Exchange is well known in professional contexts, and a highly valued (high rep) user account is something that can boost people's CVs.

    This will bring in people who hope to quickly and cheaply build such a boost for their CVs.

Each of these motivations can be fulfilled by posting ChatGPT content, and I offer that these are major motivators for people to do so.

You can factor these insights into building assumptions about what is really happening, and then you can attempt to verify those new, inconvenient-but-honest assumptions through new rounds of data analysis.

3
  • 14
    There is also the possibility that the (potential) increase in ChatGPT answers is de-motivating new users who would otherwise register, and post real human-written answers. An attitude like, "Why should I even try to get some SO reps when others will just post quick AI-gen answers and take the credit that I would otherwise have earned?" Jun 8, 2023 at 5:42
  • 1
    "factor these insights into building assumptions about what is really happening" But how exactly does one do that? How much is 1 rep really worth to people? I have no idea. It might vary a lot. I think it's impossible to accurately model this. In the other hand, it would be possible to calculate how much rep people may have gained by posting GPT content since last week. Jun 9, 2023 at 10:46
  • 2
    @Trilarion I'm no (data) scientist, so I can rely on only what I seem to have read somewhere: some of science is said to work such way that you develop an assumption, then attempt to make and run a model of it, in order to verify the truthiness of that assumption. This process relies on smart and insightful people to recognize such compositions of models that are likely to be correct. Thus, this process is sabotaged if someone, due to their biases, are unwilling to build the most plausible models. I believe SE might have fallen into this trap, and that's why I urge a bolder, unbiased approach.
    – Levente
    Jun 10, 2023 at 15:10
8

I just want to mention the existence of a paper on the arXiv (pdf): Guo et al., How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection, Jan 2023.

TL;DR: This paper tested humans detecting ChatGPT content, and found an accuracy ranging from 34% to 98%, depending on experimental conditions (not to be confused with the false positive rate). They also ask humans whether a ChatGPT-written answer or a human answer is more "helpful", and found ChatGPT's answers were more helpful 23% to 72% of the time, depending on experimental conditions. One dataset used in the experiment partly used finance-related Stack Exchange Q&A data. Note this was uploaded to the arXiv in January 2023.


From the results in this paper, I want to highlight Table 2 (screenshot); the top part is reproduced below (the bottom part is similar but for the Chinese language):

Pair-expert Single-expert Single-amateur Helpfulness
All 0.90 0.81 0.48 0.57
reddit_eli5 0.97 0.94 0.57 0.59
open_qa 0.98 0.78 0.34 0.72
wiki_csai 0.97 0.61 0.39 0.71
medical 0.97 0.97 0.50 0.23
finance 0.79 0.73 0.58 0.60

Rephrasing the paper to be more succinct, the columns labels are:

Pair-expert: The "expert" tester needs to determine which answer (one from humans and another from ChatGPT) is generated by ChatGPT.
Single-expert: The "expert" tester needs to determine whether the answer (a single answer randomly given by humans or ChatGPT) is generated by ChatGPT.
Single-amateur: The "amateur" tester needs to determine whether the answer (a single answer randomly given by humans or ChatGPT) is generated by ChatGPT.

And the fourth column is distinct:

Helpfulness: Each tester is asked to pretend that the question is proposed by him/herself, and needs to determine which answer (one from human and another from ChatGPT) is more helpful to him/her.

The participants are...

... 8 experts (who are frequent users of ChatGPT) and 9 amateurs (who have never heard of ChatGPT).

Importantly, they're "experts" in ChatGPT (in the sense that they're familiar with using ChatGPT), not subject-matter experts.

They're all asked a series of 30 questions for each of five different datasets (the rows in the table above): (a) Reddit's Explain it like I'm 5, (b) WikiQA, (c) sourced from computer-science topics on Wikipedia, (d) Medical Dialog, and (e) finance-related Q&A "built by crawling Stackexchange, Reddit and StockTwits". So I would say (b) and (e) are the most relevant, especially (e) since it partly uses Stack Exchange data.

The authors didn't explicitly say this in their paper, but I expect the first three columns in the table list accuracy rates, i.e., the proportion correctly identified as ChatGPT (note that this is not equal to 1-FPR, where FPR is the false-positive rate). I also note that this paper was published in January, and ChatGPT has improved considerably since then. I also point out the possibility that ChatGPT may have encountered the questions it was asked for this research previously during training.

1
  • 2
    Very interesting paper. I'm going to have to dig more into how they created their datasets with human answers. Their summary of Distinctive Patterns of ChatGPT and Major Differences between Human and ChatGPT is quite good, and I think the paper points to several other topics that would be interesting to research (like how human answers were rated more "helpful" in medical domain questions even though ChatGPT was more helpful for other domains).
    – ColleenV
    Jul 7, 2023 at 15:03
5

Why, instead of making stuff up about how confident our detection methods are, (human or otherwise) did we not test them? SE already has an incredibly large library of high quality QandA content from before November 2022. A simple test, have ChatGPT answer questions and take thousands of human generated answers where we know with certainty how it was created and test our detection methods.

Even if the detection methods are 1 in 20 false positives the point would still be moot. If a user wanted a generated answer, they could ostensibly use ChatGPT themselves for free. No, instead they came to the SE network for high quality answers from real people that are actually accurate.

Finally, what does SE have to gain by allowing GPT content? So far, every message to us has been about how it is essentially "not bad" for the site. Really? We're shooting for "not bad"? What incentives are there, as from my perspective, there are none, other than reduced work for the moderators who already work so hard for this community.

4
  • 2
    Which posts have you read which say it's "not bad" for the site?
    – kaya3
    Jun 13, 2023 at 21:24
  • 4
    @kaya3-supportthestrike In all of the message from the staff they are discussing the negative implications of GPT content without ever questioning the positive implications and whether allowing it brings any good at all
    – ajgrinds
    Jun 13, 2023 at 21:27
  • 1
    This whole ordeal is predicated on users doing exactly this on SO and failing miserably.
    – Kevin B
    Jun 13, 2023 at 21:28
  • @KevinB yes but without any actual A/B testing. It is all conjecture
    – ajgrinds
    Jun 13, 2023 at 21:29
-5

False assumption: LLM outputs are distinguishable from human content

In order for us to consider using detectors of any kind (automated or human) on the platform at these volumes, we’d need to see less than a 1-in-50 false positive rate from them. We’ve selected this rate as a ballpark estimate for acceptability.

Although you argue your point, what evidence do we have that such a rate is even possible?

GPT and similar LLM models are designed to mimic natural human language, and although there are detectable patterns, the fundamental reason those patterns exist is because they are probable pathways of human communication.

If one in fifty human written posts did not resemble the word choice, ordering and structure of a modern LLM model, that would be surprising. In real life situations we will always have non-LLM posts that resemble LLM posts, and that's to be expected.

Hopefully detection technology will improve - but so will GPT and similar LLM models improve in generating more natural written text. Setting our baseline target to be so radically high that it's unachievable today virtually guarantees it will be less achievable in the future. We need to be prepared for what the problem is likely to resemble in 2024 or 2025, not just what's happening today.

Recommendation: Reduce the target false positive rate

I would suggest that a 1-in-10 false positive rate would be better, but still may prove too high in the long run.

Perhaps this seems untenable to SE from the prospects of incorrectly placed suspensions, but perhaps we need a more blended approach to the problem:

  1. Set a clear threshold for how many posts a user must have posted to make the result valid
  2. Set clear expectations for which detectors should be used and how high each detection result must be to count as valid

And ideally these types of tools should be automated and then supplied to moderators to inform their decisions.

9
  • 17
    (Raw) LLM output follows a specific formula. It's very easy to recognize once you've seen a few of'm. The problematic users that resulted in this ban on GPT-generated content generally aren't the ones to put any effort into rewriting the output. So yes, "LLM outputs are distinguishable from human content"
    – Cerbrus
    Jun 28, 2023 at 8:29
  • 1
    Language learners like me are using LLMs to learn foreign languages. Do you think their writing style could consequently resemble LLMs? In the Chinese education system, it's common to memorize phrases and even whole paragraphs and articles (that are considered excellent examples) and tweak them in writing---maybe they're doing that with LLM-generated material. Or what if they use LLMs to polish, rewrite, or translate their posts prior to posting? Assuming good faith, is it not possible there are other reasons for posting content that resembles LLM-generated content? Jun 28, 2023 at 9:02
  • 11
    It is true that distinguishing AI generated text from human-written text is difficult to do by algorithm. All you have to do is look at how poorly most automated spam filters perform without human input and extrapolate. It is relatively easy for a human with some experience to detect AI generated content however. Being able to reliably fool humans is the holy grail for AI and we’re not there yet.
    – ColleenV
    Jun 28, 2023 at 10:29
  • @Cerbrus with respect, your answer seems to assume all LLM output is like ChatGPT. In 2022 we had to ban a prominent user for utilising bespoke AI/LLM tools to generate site content, and for much of it the distinction was far more subtle. Give it a year or two and you'll see increasing quantities of LLM content that leave you shrugging your shoulders as to whether it's LLM or not. We're only at the edge of the precipice today. My point still stands - the target false positive rate is unachievable and unsustainable into the future. Jul 16, 2023 at 8:37
  • 5
    With all due respect, LLM output still is full of hallucinations. We're not writing rules for in "a year or two". We're writing rules for how the state is now.
    – Cerbrus
    Jul 16, 2023 at 11:52
  • 1
    @Cerbrus You can have rules for how the state is now and also take into account that those rules need to change eventually, as Steve seems to have been arguing. "These standards will be open to revision as technology changes and data is gathered on identifying AI-generated content." (from here). Not sure why you two were having at it that much in the comments here, but the sidesniping (dumb, toxic, adding two and two, limited experience, bold, etc.) didn't do the comment thread much justice. I've cleaned up the nasty bits, keep the rest clean.
    – Tinkeringbell Mod
    Jul 28, 2023 at 8:02
  • @Tinkeringbell thanks for the cleanup. Of course, I'm open to reconsidering policy if and when AI gets more capable, but that's a discussion for another time when that's the case. IMO, it's pointless to speculate about the future of AI in a debate on current policy.
    – Cerbrus
    Jul 28, 2023 at 9:26
  • It comes across that you're seriously arguing that things like "mimics the word choice, ordering and structure" is the only test humans could apply to detect LLM content. In reality, text that perfectly conforms to expectations of word choice, ordering and structure can still completely and utterly fail to make sense in context. Prose can also apply expectations that are relevant to the wrong context - e.g. sounding picture-perfectly like a high school sociology essay in the middle of what is supposed to be a technical manual. Aug 4, 2023 at 6:05
  • @KarlKnechtel nope, that's not what I'm arguing and not remotely the point I'm making. I agree with you wholeheartedly and don't really understand why you think that's disagreeing with my point in any meaningful way. Aug 4, 2023 at 21:54
1
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .