296
votes
GPT on the platform: Data, actions, and outcomes
What makes you so confident that your methods of identifying GPT posts are more accurate than how moderators have been identifying them?
So far, all of the data I've seen hinges on the assumption that ...
- 84.5k
262
votes
GPT on the platform: Data, actions, and outcomes
Thank you for posting the methodology! That's a step forward.
Unfortunately, there's a critical flaw in your methodology: you assume that
In principle, if people are copying and pasting answers out ...
231
votes
GPT on the platform: Data, actions, and outcomes
Appeals of moderator actions that cannot be validated need to be shown to the moderator team
When taking action against a user, we need to have strong evidence that we are correct. That evidence ...
- 28.6k
201
votes
GPT on the platform: Data, actions, and outcomes
Notes about my following SEDE queries: Upon new SEDE refresh, you may have to run the cross-site queries a few times for them to finally complete). Ignore the datapoints for the current month (cropped ...
- 29.2k
143
votes
GPT on the platform: Data, actions, and outcomes
I came off of read-only mode for this.
My initial take on this is that the analysis tries to affirm that user retention is valued higher than user quality, but the data points are still reasonable ...
- 58.3k
126
votes
GPT on the platform: Data, actions, and outcomes
Copying my response from Teams, with some small edits to remove things that you chose not to reveal publicly for some reason:
I am skeptical about several points in your analysis. But most troubling ...
- 2,994
122
votes
GPT on the platform: Data, actions, and outcomes
What percentage of users who have been suspended (for any amount of time) for GPT use were active answerers as you described them immediately prior to ChatGPT's introduction?
I wasn't a very heavy ...
- 11.9k
87
votes
GPT on the platform: Data, actions, and outcomes
What about everything else?
We've known for a long time that detectors (HuggingFace is the one I'm most familiar with, but others as well), have an extremely high rate of false-positives. Having data ...
- 16.2k
85
votes
GPT on the platform: Data, actions, and outcomes
The "actual GPT posts have fallen" graph is the lynchpin of your entire analysis. Without that graph, you've proven exactly nothing (aside from "engagement is down and the CM team has ...
- 5,907
84
votes
GPT on the platform: Data, actions, and outcomes
OK, finally there is something to actually discuss. This discussion should have happened before the policy was decided, and the policy should have been decided with community-elected moderators' input ...
- 6,286
74
votes
GPT on the platform: Data, actions, and outcomes
We are, as of right now, operating under the evidence-backed assumption that humans cannot identify the difference between GPT and non-GPT posts with sufficient operational accuracy to apply these ...
- 4,980
72
votes
GPT on the platform: Data, actions, and outcomes
My summary of your nine-google-doc-page question
You have tested how the ChatGPT-detectors work on human written posts and you did not like the accuracy.
You came up with a heuristic that you use to ...
- 8,192
71
votes
GPT on the platform: Data, actions, and outcomes
I already posted an answer to this question, in which I try to assert why people create posts using GPT and why others do not find that behavior healthy for the community. I still wanted to post ...
- 4,931
69
votes
GPT on the platform: Data, actions, and outcomes
The one-size-fits-all policy is a problem
The data you present is somewhat compelling. An answer with many cached versions, showing the answer being fleshed out over time, is a pretty reasonable ...
- 1,341
64
votes
GPT on the platform: Data, actions, and outcomes
Thanks for elaborating on your reasoning. However, I still have questions.
How was urgently overriding the moderators' mandate a conclusion?
Your exposition does not at all reveal how such an ...
- 15.1k
60
votes
GPT on the platform: Data, actions, and outcomes
What follows is the internal ‘gold standard’ for how we measure GPT posts on the platform.
While I'm still digesting much of the data, you seem to be failing to consider alternative explanations for ...
- 4,980
59
votes
GPT on the platform: Data, actions, and outcomes
there is no future for the network at all.
You might be closer to the truth than you think, or fear.
Regardless of the above, no Community Manager will tell you that removing 7% of the users who try ...
- 4,931
58
votes
GPT on the platform: Data, actions, and outcomes
This is based on a question that I asked on the Stack Moderators Team. It wasn't addressed there, but it may have gotten lost in the shuffle. I've trimmed it down to make it more appropriate for Meta ...
- 55.4k
51
votes
GPT on the platform: Data, actions, and outcomes
I have big doubts about this part of presented data:
Yet, at the same time, actual GPT posts on the site have fallen continuously since release
It's implied that posts with a small number of Drafts ...
- 1,791
48
votes
GPT on the platform: Data, actions, and outcomes
Analogy and motivation
Let's say I'm a dairy farmer in Wisconsin. I look at some data about milk output from my dairy and find that output is down 50% from last year. This is a remarkable drop, and it ...
- 14.6k
47
votes
GPT on the platform: Data, actions, and outcomes
TL;DR: Your analysis that concludes “automated GPT detectors have unusable error rates on the platform” is not sound.
I can’t easily find what exactly Hugging Face’s percentage signifies, but I assume ...
- 28.3k
45
votes
GPT on the platform: Data, actions, and outcomes
The methodology presented is very interesting, but if the company really believes that human beings can't tell the difference between good quality posts that contribute to the site and poor quality or ...
- 27.1k
44
votes
GPT on the platform: Data, actions, and outcomes
since the advent of GPT, the % of content produced by frequent answerers has started to collapse unexpectedly
The data you show, along with system changes introduced since advent of GPT suggest that ...
- 11.3k
44
votes
GPT on the platform: Data, actions, and outcomes
There are multiple things I have questions/concerns about, but I'd like to focus on just one. This still seems like throwing the baby out with the bathwater with regard to what the announcement/policy ...
- 16.9k
37
votes
GPT on the platform: Data, actions, and outcomes
Thank you for providing some details, but I have several issues, mainly statistically related. First, there's with
In order for us to consider using detectors of any kind (automated or human) on the ...
- 14.9k
37
votes
GPT on the platform: Data, actions, and outcomes
In the Community Management industry, it is a well-known fact that removing a person from a community, even for a short time, has an outsize impact on the contributor community.
In other words - ...
- 1,423
36
votes
GPT on the platform: Data, actions, and outcomes
It is worth noting that, early in the release of GPT, we changed the Stack Overflow rules to require new users to wait 30 minutes between first posts, instead of 3 minutes as was originally set for ...
- 5,000
36
votes
GPT on the platform: Data, actions, and outcomes
I've read this twice, pondered for a while on a few of your more vague turns of phrase. Most of my specific constructive thoughts and criticisms have already been covered by other users. I found Kaya3'...
- 1,281
34
votes
GPT on the platform: Data, actions, and outcomes
As to the future of the site part, I think it's imperative to be able to detect AI generated answers. Whilst it is true that voting can take care of hallucinations and plain incorrect answers, the ...
- 1,658
33
votes
GPT on the platform: Data, actions, and outcomes
This is basically the same thing Mithical already said, and it is the first thing that came to mind as soon as I started to read your very long post. That said, I am posting this version to give a ...
- 34.8k
Only top scored, non community-wiki answers of a minimum length are eligible