-585

Update: GPT on the platform: Data, actions, and outcomes


As you may be aware, a number of moderators (on Stack Overflow/across the network) have decided to stop engaging in several activities that they had taken on, including moderating content - in fact, almost all moderation tasks. The primary reason for this action is dissatisfaction with our position on detection tools regarding AI-generated content, and discontent with how that was rolled out.

We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions. People with original questions and answers were summarily suspended from participating on the platform. We stand by our decision to require that moderators stop using that tool. We will note that it appears to be very rare, however, for mods to use ONLY the ChatGPT detection tool, and frequently their own analyses were in use as well. We will continue to look for other, more reasonable tools and are committed to rapid testing of those tools and any suggested heuristic indicators.

The moderators who are engaged in this action served this community collectively for many years on the platform. Personally, I consider a number of them friends, and anytime friendship is tested like this, it’s difficult. I would like to say to them clearly that I hope they know how much I, and the whole staff and community, appreciate their collective decades of service to this community, and I hope that we are able to come to a path forward. I regret that actions have progressed to this point. The Community Management team is evaluating the current situation and we’re working hard to stabilize things in the short term. We’ll be working collaboratively with the community on the long-term solution.

I’ll be honest, the next few days and weeks might be a bit bumpy. Both sides share a deep and unchanged commitment to quality on this platform. Additionally, it’s important that we guarantee that anyone who has meaningful contributions to make has the opportunity to do so. As we have updates, I and the team will be sharing them here.

86
  • 254
    Reminder that Philippe is trying to be civil here... while I am one of those on strike and don't intend on doing any curating. People should still be nice, even if critical. Jun 5, 2023 at 18:31
  • 121
    Nice words, but we also got nice words last time the "friendship" was tested, and this brought us, well, here. Point is, words are not enough. At this point I also simply don't believe what management says, but that's already my own problem to solve. Jun 5, 2023 at 18:34
  • 72
    So, forgive me for being blunt but does the company any actual new insight to post other than the same generic and without evidence note about the tests (reminder: users already asked for supporting data, it is kinda useless repeating that without any number or insight on how those data was achieved) and a "we still love you" that comes out as a tad patronizing?
    – SPArcheon
    Jun 5, 2023 at 18:35
  • 299
    "We ran an analysis and the ChatGPT detection tools have an alarmingly highly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions" - I am deeply disappointed that you've opted to ignore our many comments both before and after the AI policy going live stating that we do NOT blindly use GPT tools for suspensions. You've been involved in several of these discussions as well.
    – Zoe
    Jun 5, 2023 at 18:35
  • 116
    Honestly, you are just making everything worse with this post. You still refuse to listen to us, and continue to spread the same lie about how moderation on AI content was done. Jun 5, 2023 at 18:36
  • 88
    Nothing here even attempts to address the issue. The first half is a repeat of what was said before, except they added very rare so at least there's that. The second half is either sincere or emotional manipulation. As someone who doesn't really know Philippe, I'll leave that to someone else to decide Jun 5, 2023 at 18:36
  • 45
    @StephenOstermilleronStrike That's not what Philippe said to the media
    – Zoe
    Jun 5, 2023 at 18:39
  • 95
    Also, after at least FOUR post that asked you to avoid namecalling mods (the ones willing to spend their time for you for free) as sheep that blindly follow the outcome of tools to decide on suspensions instead of using their brain you continue with that same claim, to the point of even once again involving a media/news site too (didn't the company repent about misrepresenting Monica on a news site in the past)? Please do share your proofs or stop this now
    – SPArcheon
    Jun 5, 2023 at 18:44
  • 71
    "Personally, I consider a number of them friends, and anytime friendship is tested like this, it’s difficult.": Reading this from outside of the friend sphere, it seems like you conflate the strike with malice, and take it as a hostile action from us. Jun 5, 2023 at 18:51
  • 57
    "I would like to say to them clearly that I hope they know how much I, and the whole staff and community, appreciate their collective decades of service to this community" -- oh please. Why does everyone have such a need to remind verbally of their love and appreciation exactly after they've done something that shows just the opposite?
    – ilkkachu
    Jun 5, 2023 at 19:39
  • 90
    @Philippe Your post focuses on ChatGPT detection tools being banned, and I don't see it as a huge issue. But the linked policy says something much more dramatic: it bans the "use of moderators' best guesses based on users' writing styles and behavioral indicators". E.g. if a user puts out 20 long answers of dubious quality over the course of 40 minutes (which are clearly generated; something I've seen once or twice), we can't ban him, because that's a "behavioral indicator". Is that correct? Jun 5, 2023 at 21:56
  • 38
    Please don't close this question. We want users to be able to post responses to the announcement as answers. (cc @cocomac ).
    – user202729
    Jun 6, 2023 at 3:26
  • 123
    @Philippe: Want people to calm down? Then prove your claims. Show us your analysis, and - more importantly - the underlying data. Show us why you think countless innocent users have falsely been suspended, how this is an dramatic uptick and problem. Show us, goddamnit!
    – CharonX
    Jun 6, 2023 at 7:14
  • 63
    "Both sides share a deep and unchanged commitment to quality on this platform." citation needed for one side. Jun 6, 2023 at 10:00
  • 125
    Have you considered replacing moderators with payed employees to whom you can give orders ? At this point this sounds like the more honest solution. Expecting volunteers to implement company policy (that apparently they are not even privy to) seems a mismatch in expectations when talking about people who see themselves as part of the community. If you cannot offer job satisfaction for something that looks like a mindboggling amount of work, you should at least offer money. Jun 7, 2023 at 10:27

21 Answers 21

580

People with original questions and answers were summarily suspended from participating on the platform.

We've been asking for data here for a week. As far as I know SE has not provided a single case of an unjustified suspension. Rather the argument has been that it is impossible to detect AI-generated content at all, and therefore it is impossible to produce data on unjustified suspensions here. Which is a very frustrating point to argue against.

This statement here contradicts what SE has told us, it is a statement of fact that alleges that suspensions were unjustified. I expect from SE to back this statement up with data or retract it.

26
  • 53
  • 61
    I think others have already made this point elsewhere, but it's worth repeating with each dishonest statement the company makes: the persistent blaming of faulty AI detection tools (with no effort to try and back up the alleged corollary of high false positive rate for suspensions) is gaslighting at this point. Repeating the fallacy (if we're being charitable; blatant lie otherwise) with the hopes of people outside meta not realising the smell of organic fertilizer that the company reeks of. Jun 5, 2023 at 23:57
  • 42
    SE Staff can't provide data, because they have no data. The claim of a "dramatic upswing in suspensions" of "People with original questions and answers" is simply an ass-pull.
    – CharonX
    Jun 6, 2023 at 7:03
  • 166
    Honestly, a dramatic upswing in suspensions wouldn't be strange, given that there's a new, suspension-worthy action being performed on a large scale. A dramatic upswing of the rate of unjustified suspensions would be a problem. It's not unexpected that the whole ChatGPT thing has created a major increase in justified suspensions, and that is a problem, but one in which the moderators should be praised and ideally assisted with better tooling.
    – Erik A
    Jun 6, 2023 at 8:01
  • 8
    "We've been asking for data here for a week" show your communities also the data. How you were applying, what was the process, what were the results. You would get more sympathies if you make your own process transparent too. Show SE how is it done.
    – Braiam
    Jun 7, 2023 at 15:02
  • 3
    @Braiam: Answers and comments here ask for data. It's been 2 days. I will bet you that no information will be forthcoming when a full week passes - and that's ignoring what MadScientist wrote, which I tend to believe.
    – einpoklum
    Jun 7, 2023 at 21:34
  • 4
    @Braiam While possibly true, it's not the community that's throwing accusations around, it's SE. To the press. BTW, there's also this.
    – Passer By
    Jun 8, 2023 at 7:42
  • 2
    Playing devil's advocate here... I have a vague suspicion that they company doesn't want to share specific data because the Monica case is too fresh in everyone's memory... I would guess that the community team is stuck between doing what they want to do (release real data and talk about constructive steps forward) and the legal department telling them they can't release specifics as it could be construed as libel.
    – anonymous2
    Jun 9, 2023 at 17:51
  • 2
    @PasserBy the community is throwing accusations around, more so than the company. See the title of the question you link. And it is true that the striking moderators keep saying that "moderators are not relying solely on ChatGPT detection tools", which they cannot actually prove. SE has already shown the data, and it makes no sense to keep finding "flaws" in SE's data collection while the striking side has provided zero data to substantiate their claim (ok well, maybe the one data point that you link).
    – wimi
    Jun 11, 2023 at 7:37
  • 3
    @PasserBy I don't agree with the new policy that forbids moderation of AI-generated content. But I find all this talk about defamation and missing data wildly inaccurate and overly dramatic. The data is there, and saying "the use of AI detection tools correlates with an increase in the number of suspensions" is not defamation or libel by any reasonable interpretation of the word. Saying that someone "lies, mistreats and maligns", on the other hand...
    – wimi
    Jun 11, 2023 at 7:42
  • 5
    @wimi I don't know how you manage to interpret the linked title as an unfounded accusation. If implying mods are blindly following GPT detectors and trigger-happy about suspensions to the press aren't mistreatment, I'm not sure what is. By suggesting mods needs "proof" they didn't use detectors is to suggest you think they're lying about their actions. You don't need a thousand words of analysis to remember what you did, which is quite contrary to interpreting data about the impact of suspensions.
    – Passer By
    Jun 11, 2023 at 10:19
  • 2
    @PasserBy it doesn't matter. I've asked for data on this very same thing, and was met with serious resistance by moderators. I want moderators to do some dogfodding too. They should also have to defend their policy with data. Show the rest of the community what is being done. Both SE and moderators being too cagey with the information doesn't help. Everyone should just show their hand on the table and we will see exactly what's going on.
    – Braiam
    Jun 11, 2023 at 22:59
  • 2
    It is quite possible that unknown AI test used by mods returns too many false positives, but to ignore the problem as the staff do is worse.
    – kludg
    Jun 12, 2023 at 8:53
  • 2
    @PasserBy No, the current conundrum is that neither side is transparent. Nor the mods nor the company. Mods should put up or shut up if they want transparency of the company. At least for their communities. Also, ChatGPT doesn't create useful content? Are you sure? Because that seems to be a kind survivor-ship bias were you are only seeing when it fails (and yes, it does fails spectacularly), not the times where it was correct and yet was distrusted.
    – Braiam
    Jun 14, 2023 at 14:19
  • 1
    @PasserBy if you think I didn't try both SE and moderators are shielding themselves on the moderator agreement to not show the data. Something that I and you are able to do: get a list of deleted posts. I'm just asking for the sample, to do some analysis.
    – Braiam
    Jun 14, 2023 at 14:28
419

We ran an analysis

Alas. This is the moment where I miss the times where y'all used to publish Data Science times! (see e.g. November 2018 and December 2018) which not only showed some professional and responsible analysis of the data that you had, but did so in a way which was transparent to the community.

Ever since the announcement of the new policy imposed upon moderators, preventing them from using their skills to handle the surge of AI generated content, there was still no disclosure of the research that would justify it.

ChatGPT detection tools have an alarmingly high rate of false positives,

Moderators know this, and they do not make decisions based on such tools alone. In fact, some moderators claim that they don't rely on automatic detectors at all. This makes the argument a non sequitur.

which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions

And rather than interpreting this as "hey look, a lot of people are creating accounts and posting lots of answers with ChatGPT to exploit the system", it is appalling that your reaction to this was "hey look, a lot of people are suddenly interested in contributing to the site and moderators are suspending them for no good reason". In a scenario where a platform with gamification is already pretty loaded with answers to most questions, and an alternative platform (ChatGPT) is available to quickly obtain answers to the most frequent and simple ones, even Occam's razor would have guided me to the first interpretation. This is the part I would consider not very responsible. And without concrete evidence of undue suspensions taking place, it is also not transparent.

At this point, in the absence of factual data and research to back the claims made, this question would have been a bit more honest if it did not include the second paragraph. Because I understand that "the damage is done", and reverting the policy alone won't revert the damage, making that too a bit reckless. I shall wait to see what's coming next.

See also:

13
  • 3
    "Moderators know this, and they do not make decisions based on such tools alone." the question post here also says "We will note that it appears to be very rare, however, for mods to use ONLY the ChatGPT detection tool, and frequently their own analyses were in use as well." Jun 5, 2023 at 18:49
  • 57
    "even Occam's razor would have guided me to the first interpretation.". Did you try with the "Stakeholder's Wallet" instead? Just assume that all that matters is expanding the userbase and the second option immediately becomes the correct choice.
    – SPArcheon
    Jun 5, 2023 at 18:51
  • 2
    @starball So yes, they know this. I guess my main goal here was to emphasize that the argument made is... just bad.
    – E_net4
    Jun 5, 2023 at 18:53
  • 4
    Well, can someone come up with a better KPI in place of the one that lumps LLM regurgitations and content together in the same metric? Or a method to more clearly visualize the actual meaning of this one, so that it cannot be used to save face in the next growth targets accomplished meeting?
    – anx
    Jun 5, 2023 at 19:25
  • 91
    The analysis has been provided to moderators, now, so we're mostly in the loop. We can discuss things more meaningfully with the CMs. (Can't share details, but wanted to let people know about the positive development.)
    – wizzwizz4
    Jun 5, 2023 at 21:20
  • 8
    ^ @wizzwizz4 I hope the fact shared with mods is made more visible than a comment. Thank you for letting us know. Appreciated.
    – QHarr
    Jun 6, 2023 at 3:20
  • 4
    @wizzwizz4 we don't need details, but can you share some rough numbers? E.g. how many users are claimed to have been unjustly suspended (ideally as both an absolute number as well as a percentage of all GPT-related suspensions)? Also, what's your gut feeling about the analysis - is it solid and raises valid and important points that were missed / mishandled by the mod team, or is it yet more BS?
    – l4mpi
    Jun 6, 2023 at 13:51
  • 20
    @l4mpi What's already public: the number of users claimed to have been unjustly suspended is zero, but the number supposed to be unjustly suspended is higher. (As I've stated before, and I'll say again, it's theoretically impossible to be certain of provenance without information about the provenance.) We have no analysis on the actual suspensions, though that's being worked on (delayed slightly by people having lives outside their jobs / diamonds). I've been asked not to editorialise the analysis, so I won't be sharing my gut feeling at this time.
    – wizzwizz4
    Jun 6, 2023 at 14:47
  • 1
    @wizzwizz4 thanks for the update. Let's see what more information drips out over the next days, but if they were not even able to identify a single instance of an unjustly suspended user thus far, that's a pretty bad start.
    – l4mpi
    Jun 6, 2023 at 14:50
  • 1
    @l4mpi "Not even able to identify" is a slightly unfair characterisation. There's a certain nobility to the actual reasoning (which hasn't been leaked yet, to my knowledge, but you can probably infer if you know the CMs well enough). But the ultimate effect is the same.
    – wizzwizz4
    Jun 6, 2023 at 14:52
  • 14
    I think it's totally fair for people to make adverse inferences about what the data or analysis says, when SE is hiding it from us. If they want to refute those inferences then they should publish their analysis. In the meantime, giving them the benefit of the doubt only gives them less reason to publish it. So I will continue operating on the assumption that SE have no factual basis for any of their claims.
    – kaya3
    Jun 6, 2023 at 15:50
  • 45
    Yeah the biggest gaffe here seems to be attributing causation or even meaningful correlation between "ChatGPT detection tools are inaccurate" and "lots of people are getting suspended for allegedly using ChatGPT". Conveniently missing from this equation is "ChatGPT was recently made available for free" which is the real reason why people are suddenly posting ChatGPT content (and getting suspended for it) at such high volumes.
    – TylerH
    Jun 6, 2023 at 19:50
  • 2
    But @E_net4isonstrike, sounds you're not being welcoming enough to the bots. AI has feelings too you know! :-P
    – einpoklum
    Jun 7, 2023 at 21:31
332

This post is in violation of our Misleading information policy. Relevant sections quoted:

To ensure the integrity of our network, we do not allow any content that promotes false or misleading information or coordinated misleading information campaigns.

Broadly speaking, we do not allow and may remove misleading information that:

  • Is likely to significantly harm democratic institutions or voting processes or to cause voter or census suppression
  • Promotes disproven claims of election fraud or manipulation as factual or severely misrepresents the safety or validity of results from voting machines or other voting processes
  • Promotes the specific views of a party, government, or ideology by using false claims regarding others
  • Misrepresents, denies, or makes unsubstantiated claims about historical or newsworthy events or the people involved in them

Please correct this error.

  • We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions.

    That is not what that word means. There is no correlation. Quoth Wikipedia:

    In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data.

    You have observed a fact, and a trend, and believe them to be related. What you have is a hypothesis, and many alternative hypotheses exist. You are playing off the popular conflation of "correlation" and "causation" to make this sound more impressive than it actually is.

  • The primary reason for this action is dissatisfaction with our position on detection tools regarding AI-generated content, and discontent with how that was rolled out.

    No, that is the primary focus, goal or aim of the strike. It is not the primary reason; merely the proximate cause that broke the camel's back. You are downplaying the myriad reasons – systematic problems stretching back years – that led us to this point.

  • People with original questions and answers were summarily suspended from participating on the platform.

    You have not provided evidence of this. You have been asked repeatedly, and last I heard (from you!), Stack Overflow had no evidence of this. Either substantiate your claims, or retract them.

  • Your statements to the press, and leaked internal messages, make it clear that this is a deliberate, coordinated campaign of misleading information.


On an unrelated note:

  • We will continue to look for other, more reasonable tools

    We've tried to explain to you that no such tools exist, and no such tools can exist in public (nor be created with the resources available to Stack Overflow). Any non-avalanche discriminator can be run backwards, and used to cause that discriminator to give arbitrary output; and empirically, most transformer-based tools can fool most other transformer-based tools.

    If you do not understand this by now, you as a company do not have the expertise to be working with AI.

    and any suggested heuristic indicators.

    I can confirm that this, at least, isn't made up. Discussions about heuristic indicators have occurred, where suggestions have been made, and the CMs have been looking through those discussions (and even participating).

  • I regret that actions have progressed to this point.

    We know that you, personally do, Philippe. But the company overall does not appear to regret it sufficiently; or else it would have acted on that regret.

    Update 2023-06-12: I now have reason to doubt your sincerity. Friends don't allow – or, perhaps, writethis kind of internal communication about their friends, even if they're "just doing their jobs". I have to say, though: props. I really believed you. (And I'm still inclined to forgive, if we get an apology.)

    Update 2023-07-26: From a 2023-07-20 Moderator Strike update:

    Apparently, the statements sent out in Philippe's name did not undergo his approval and were put together by a PR team.

    My ire may be misplaced. (If that's true, I'll owe Philippe an apology.)

8
  • 2
    can you expand on your argument about the misinformation policy? What do mod-issued suspensions have to do with voting procedures? Or are you talking about flagging as a voting process? Or are you instead saying that the network is a democratic institution? (sort of true I guess?) The argument has the potential to pack some punch, but right now it's a bit vague. Jun 5, 2023 at 20:20
  • 13
    @starball It's not just mod-issued suspensions. They have told one thing to the public and said another thing in private, and I don't know how much other mods have leaked so I'm not going to elaborate. I meant the flagging thing, that the network is a democratic institution, and one other thing – but that was just for that bullet point. The other bullet points all obviously apply.
    – wizzwizz4
    Jun 5, 2023 at 20:26
  • 6
    I was confused about the "correlation" part for a bit. If anyone is also confused, basically: you can only say there's a correlation if "a year ago detection tools was accurate and suspension rate was low, now detection tools is inaccurate and suspension rate is high". The suspension rate is definitely [edit: no data but "likely"] correlated to the general availability of LLM tools to the public though.
    – user202729
    Jun 6, 2023 at 1:26
  • @user202729 Correct-as-I-understand-it up until the last sentence; I haven't been shown an analysis that shows any such correlation, and I'm certain you haven't seen the data either, so "definitely" is stretching it.
    – wizzwizz4
    Jun 6, 2023 at 1:30
  • 9
    @wizzwizz4 Right, but it's definitely believable -- in the sense that "one year ago ChatGPT was unavailable and suspension rate is low, now ChatGPT is available and suspension rate is high" -- most likely because of all those illegitimate users who repost low-quality ChatGPT answers verbatim.
    – user202729
    Jun 6, 2023 at 1:32
  • I don't think the point about non-avalanche discriminators is correct. One assumes that most of the people that are using ChatGPT to write answers are just...using ChatGPT. They're not rolling their own language models. They're not going to be using sophisticated means to evade detection. Jun 7, 2023 at 1:53
  • @SteveBennett They don't need to. Until the hype dies down, VC-backed corporations will do the heavy lifting for them.
    – wizzwizz4
    Jun 7, 2023 at 11:06
  • Alas, your suggestion cannot be followed due to the strike. Or will someone make an exception here? :-P
    – einpoklum
    Jun 7, 2023 at 21:32
223

The primary reason for this action is dissatisfaction with our position on detection tools regarding AI-generated content, and discontent with how that was rolled out.

No, it isn't, and I think the community would appreciate if you didn't misrepresent the actual reasons for the strike, which have been clearly expressed. The primary reason for the strike is because Stack Exchange, Inc. has imposed a secret policy which de facto allows behaviour which the community overwhelmingly thinks should not be allowed.

That is, regardless of whether "detection tools", subject matter knowledge and experience, the user's post history, post frequency, or other evidence shows that a user has plagiarised from an AI tool, the users caught doing it can't be suspended.

You have nothing to gain from lying to us about our own motivations, so honestly this looks more like you are talking past us, and lying to some other audience about us.

6
  • 64
    Yeah, this is a blatant lie from the company. We agree with the company about the detection tools, simply because that's the truth; decisions cannot be made solely based on them. Jun 5, 2023 at 20:42
  • 1
    "The primary reason for the strike is because Stack Exchange, Inc. has imposed a secret policy" I don't think so. The primary reason for the strike is that the site is exponentially overwhelmed with AI generated crap answers and the company has hamstrung those who have been trying to defend against it. So now we say, fine, go ahead and drown in the crap.
    – matt
    Jun 10, 2023 at 14:41
  • 2
    @matt The rest of that sentence is: "... which de facto allows behaviour which the community overwhelmingly thinks should not be allowed." I am not sure how you missed that. The behaviour which the community overwhelmingly thinks should not be allowed is, of course, plagiarising answers from generative AIs.
    – kaya3
    Jun 10, 2023 at 14:42
  • I'm on your side. But this answer is still primarily about the company lying, behaving underhandedly, etc. I'm saying that's not the heart of the matter. The heart of the matter is that the site (esp. SO) has gone to hell in a handbasket.
    – matt
    Jun 10, 2023 at 14:48
  • @matt From the linked post explaining the reason for the strike: "The new policy, establishing that AI-generated content is de facto allowed on the network, is harmful in both what it allows on the platform and in how it was implemented." From the strike letter itself: "Specifically, moderators are no longer allowed to remove AI-generated answers on the basis of being AI-generated, outside of exceedingly narrow circumstances. This results in effectively permitting nearly all AI-generated answers to be freely posted, regardless of established community consensus on such content."
    – kaya3
    Jun 10, 2023 at 14:49
  • 1
    I don't find the phrase "hell in a handbasket" anywhere on either page, and that would be a very broad, vague complaint to form a strike over. The strikers' stated reasons are specific and directed, and the demands are concrete and actionable. This answer here is about the company lying about the reasons for the strike, not saying that the strike is because the company has lied about those reasons. In any case, my answer already includes the point you are making about AI answers being allowed when we don't want them to be; I do not see any way to improve my answer to address your comment.
    – kaya3
    Jun 10, 2023 at 14:53
191

You write:

The primary reason for this action is dissatisfaction with our position on detection tools regarding AI-generated content

I will remind you that the position we are dissatisfied with comprises not just the public position on detection tools, but very importantly the broader policy and interpretation, made clear only in private, that goes far beyond particular detection tools and effectively forbids treating the vast majority of AI-generated content as unwelcome on our sites, overruling community-determined policies on individual network sites.

I hope this was an inadvertent lack of clarity in your post here rather than an intentional misrepresentation of our position; understandably there are a lot of different pressures on your time and it's often possible to dissect written words more deeply than they were intended. Thank you for acknowledging that these tools are not the only basis for identifying AI generated content, I see this as an important step forward that was not clear in previous company statements here or to the press.

3
  • 5
    Yes, clarification on this would be good. Philippe's post above suggests that this is merely about how mods are enforcing bans on AI-generated content, but the way it's been handled leaves me wondering whether SO's real beef is with the very existence of those bans. If that's not the case, SO could probably take at least some of the heat out of this by publicly stating that they support the AI bans and discussing what their proposed plan is for enforcing them.
    – G_B
    Jun 6, 2023 at 5:30
  • 33
    SO mods have already noted explicitly to staff that the primary reason we've been jumping straight to suspensions for users who post AI-generated content is because staff told us to shortly after ChatGPT was released for public use. We also have already indicated, repeatedly, that we'd be more than willing to stop issuing suspensions for this and instead simply give warnings. In fact, many of us (like yours truly) had been suggesting this already (despite arguably going against staff guidance) because we'd noticed that a large number of people apparently didn't know of the ban. Jun 6, 2023 at 11:16
  • 4
    @CodyGray-onstrike that makes this situation 10-fold more infuriating in my mind. Having seen (mostly on smaller, non-SO sites), some "heavy" suspensions for apparent first time offences, I figured SE was panicking about some overzealous mod behaviour and this was a (bungled as usual) attempt to pull things back. To hear that was their own policy is mindboggling.
    – mbrig
    Jun 7, 2023 at 20:34
174
+50

I am writing this not just a reply to this specific post, so please consider the following in the bigger scheme of things that happened in the last years. Also note that unless otherwise stated whenever I say "you" I am referring to the company as a whole and not to any specific individuals.

Premise

Let's just start with a simple premise. I no longer trust your words when you say that you didn't expect this, when you say you are sad about the outcome and so on.

I don't trust you. I can't trust you any longer after every single time you shove in our face the same theatrical act, with the same characters, the same plot, and the same twists.

Don't get what I am talking about?

Stack Exchange Scandals 101: a proven formula

Historically, most recent scandals have been following the same formula, with some minor changes every now and then.

  • The staff announces some controversial changes out of nowhere. The announcement is usually made on Meta.SE and it is carefully crafted to be as invisible as it can imaginably be to the rest of the network. Optionally, the post is made because some meddling user caught the company red-handed. If an official post is made, it will be written in the most confusing way possible so that users won't even be able to agree on what was said ("did they want to prohibit moderation of AI generated content" VS "did they suggest to stop using detector tool until we can work out a better policy" is just a recent case).
  • Corollary: whenever possible, the user base is expected to get only an incomplete picture of the issue. If applicable, mods will get some full guidance that contains the more controversial parts that they will be required not to share.
  • The announcement is met with a huge backlash as multiple users start posting related questions on Meta. The company enters the "silence" phase — no answer is given, no comment posted, no "we heard you, please give us time to write back" is said. This phase usually lasts a few days as the company "sits-out" on the issue while letting the bigger fires die down.
  • (Optional: while ignoring its own community, the company still finds time to talk with some external media news site and publish a misrepresentation of the reality that constitutes actual slander against a single or a whole category of users. Most of the time this seems to be done as a PR move to show that the company is doing their best on social issues.)
  • After some time the company comes out with a "We are aware of the issue" post that on surface looks not completely unlike an apology. Under more careful scrutiny it is more often than not a non-apology in the form of "I am sorry if you were offended" or an indirect attempt to share the blame like "It is sad that we escalated so far". The post also carefully avoids giving out any info about what is planned to do to actually resolve the problem.
  • At the same time the users start to plan their action: sometimes this can involve an actual moderation strike or users using their profile to spread the word.
  • In a desperate attempt to reduce the visibility of the protest, SE revokes the right that was given to mods to advertise posts that they deemed important for the community to see. Removing links for a solidarity fundraising campaign from users' profiles, unfeaturing posts, anything goes in this step.
  • The fire continues until the company concedes on some (minor) details. More often than not this is not a real win for the community, just an "it is better than nothing" scenario with most users preferring to accept whatever little Scoobie Snack the company dispensed instead of going into an endless fight. Sometime it may even go as far as look like a fake concession that was already planned to make the userbase more yielding in whatever negotiation is going on at the time.
  • The company then starts a final memory-clearing campaign. The community will be rendered unable to fully know what actually happened, sometimes even going as far as to use law to be sure that no one can disclose the full picture by force-handing the actual victims into silence.

Obviously any new tire fire is different, but I expect them to more-or-less follow this same script. And with that behind us, let's move on to...

A Bastard's explanation (yep, that's me)

I know that what will follow will be seen as evil by some users. "Never assume ill intent" they will say.
Yet let me also tell you that there is an old saying that goes like "Once is ingenuity, twice is stupidity, three times is malice" or, if you prefer another version, Hotoke no kao mo sando - "Even with a Buddha nature (you can forgive) only three times".
With that said, feel free to downvote the following if you wish, but I will try to explain what all of this feels like and why I think history keeps repeating itself.

Assumption #1: the attempt to make problematic posts less visible is deliberate

First of all, let's start with a basic assumption: the company knows in advance what topics will cause backlash. It is quite evident from how they are presented. Why is there no secrecy involved when they announce some new feature they are proud of, or when the company makes some donation to charity? At the same time, things like unilateral changes to existing content licensing are posted without any attempt to actually make them visible and followed by a total silence while waiting for the bigger flames to go down.

Assumption #2: the actual motivations are never the ones being told

Let's go a little further. We know that the company knows that whatever they are going to post is problematic. But how can they be so sure that it will be problematic?
There are only two possible answers:

  • The issue can be easily misunderstood, but if that is the case I would expect that instead of closing up into silence you would want to try to clear up confusion. Why keep your users angry if you really think that you can agree?
  • The issue is clearly going against the will of most users, either by removing their agency to moderate content, by shoehorning unwanted content on them, or other similar things.

Since I don't think the company is stupid I will have to exclude the first irrational behavior. That leaves us with the second one.

Assumption #3: the company already knows in advance how the users will react and then try to act surprised when it goes that way

This is just a direct consequence of what we just said and is confirmed by the fact that the scripted non-apology post will usually contain some carefully written lines to look surprised, like "We didn't expect this" or even better "We didn't expect we (== you) would actually escalate to this".

You can see where this is going.

And a Tin Foil Hat one: the hidden agenda

By this point I have to admit that all so far gives a fairly bleak picture.
Thinking back at how Monica was picked out and slandered on a news site, thinking about how the company apologized for that only to do that again now, and looking at how users now even have to say that "actually they never promised to not slander user groups anymore, only individuals".

Comments discussing the semantics of the company's agreement forbidding comments (to media) mentioning "any individual moderator's actions" vs "actions of moderators as a group"

I can not reasonably avoid seeing some form of actual malice at work here: it is my blunt conclusion that none of these changes were actually made in the interest of improving the site content or the user experience.

What I see instead is a reiterated attempt at pleasing the stakeholders in the worst ways possible.
The Monica case was never resolved, and still today it looks like Monica was just a carefully picked scapegoat that the company decided to use to get some free advertisement in an attempt to present itself as caring for the rights of the LGBT+ community that not only backfired on them but also on the ones it was supposed to protect by further fueling rage and abuse.
Years later we now had to resort once again to a new moderation strike, and once again we see the company name-calling its mods to put the blame on them. Once again we see them post multiple inaccurate representations of mods' actions on the Internet to shame them and circulate the message that "The company is improving even if some (bad) mods are trying to fight back". And I admit that I went as far as to expect someone to be used as the scapegoat again to be the new Monica.

Sadly, I don't think that I am just hallucinating things as the feeling that these actions are part of a hidden agenda that not only is willing to go against the users for the sake of profit but is also quite open to publicly shame and even sacrifice them, seems to be quite a shared sentiment nowadays based on the comments I read on the site. Promises are usually made to stop the fighting but personally I find it rather unclear whether those are actual concession the company had to make, or just planned eye-candy, smoke and mirrors, used to make the userbase yield and accept the "better than nothing" resolution.

Also it is not unusual for the company to hide themselves behind grammatical technicalities worth of the "Evil Overlord" manual: after all, after the Monica scandal they just promised to never again slander an individual on the Internet, so slandering all the moderators is fine, right?

Conclusion

Over the course of the last years, the company has made their moves to show off how nice they are, how they care about defending others, how they work on social issues. Things like the Be Nice policy, revising how new users are guided in their first steps... all great things. On paper.

Yet, in the current scenario I can't see them as anything other than empty actions that didn't actually come from the heart but just from a need to "show off", to get media to talk about how Stack is doing the right thing, to win more users -> more ads views -> more money.
At the same time the userbase is portrayed as the enemy, whose toxicity causes nightmares, and who makes newcomers feel unwanted and leave.

I'll ask you a question...
Have you ever thought about the nightmares Monica probably had? Ever considered the users that went off the site after each of these scandals to never return? Ever listened to the mods asking you to not be toxic to them?

Once again, I ask you to stop and reconsider. You can work with the users and you can choose to make actual improvements together. The media awareness that you are "good" needs to be an effect of your good actions, not the target that you have to reach by any means by faking things — then happy users will suggest your site to their friends. Currently, the only thing I can suggest is to never get involved with an actual account here.

You probably still have time. But I fear that the candle is reaching its end.

13
  • 67
    +1, though you forgot the part about announcing stuff on major holidays, like the present fiasco. Or like kicking a Mi Yodeya mod in the face on a major Jewish holy day. I could understand nobody at SO knowing about Jewish festivals. What I don't understand is launching predictably controversial policies on a major US and CA holiday. Jun 6, 2023 at 14:15
  • 1
    I have to admit that imagining elusive stakeholders sitting on some mega yacht in the bahamas and demanding this and that without even having visited SE once in their life is a solacing thought. At least, much better than all these recent decisions being made by the company itself.
    – MaxD
    Jun 6, 2023 at 14:18
  • 2
    @StephanKolassa It's a major holiday in the UK as well, so there you have two major countries regarding the percentage of the moderators from them on the network. Jun 6, 2023 at 14:26
  • 32
    I was going to log out and not look back, but I had to upvote this. The comment about announcing on a holiday made me also remember Chipps' comment about "shipping on a Friday". This latest iteration has made me realize I'm still bitterly disappointed about what went down in 2019. I desperately wish we could just roll back the clock to when "Be Nice" was an accurate summary of the CoC and the biggest controversy I had to deal with was trying to get people to stop tagging everything "grammar" on ELL. (Yeah, I lost that battle but it didn't crush my spirit like this crap does).
    – ColleenV
    Jun 6, 2023 at 17:33
  • 5
    this time, hidden agenda is probably to transform the site into ChatGPT frontend
    – gnat
    Jun 7, 2023 at 15:06
  • 4
    @ColleenV "Be nice" was the start of the slippery slope that led us to this point.
    – Ian Kemp
    Jun 7, 2023 at 19:23
  • @IanKemp I wasn’t asserting an opinion about whether it was a good thing that the policy could be summed up that way. I was just remarking on how far back my nostalgia would like to go.
    – ColleenV
    Jun 8, 2023 at 1:17
  • 15
    No downvotes here. I think it’s been fairly obvious in several of these SE scandals that malice cannot be doubted, and that the true goal of SE is precisely the opposite of what they start out touting. In the Monica case, the CoC was deliberately written in an absurd way they knew would cause a furore and backlash against LGBTQ+ users, driving in a flurry of new users, which increased revenue. Currently, a network full of AI-generated posts created by AI-driven bots is exactly what they want because, again, it provides an influx of new users and increases revenue. Jun 8, 2023 at 2:53
  • 2
    Given all the flood of AI-centric blog posts from the company, and capitalists' well-known disregard for long-term care of their holdings, I agree this doesn't seem like much of a "hidden" agenda.
    – miken32
    Jun 8, 2023 at 19:13
  • 10
    @StephanKolassa "I could understand nobody at SO knowing about Jewish festivals." I couldn't. Not when Joel Spolsky (an outspoken Jew) was still with the company at the time. Jun 8, 2023 at 20:43
  • 2
    Now that I think about SE's actions in the past 3-4 years and how insanely illogical they were, it reminded me of a popular saying in our country that even if our leaders intentionally planned to destroy this country, they couldn't do it more perfectly than what they have already done. Jun 11, 2023 at 11:55
  • @JanusBahsJacquet I would NOT say that the CoC was crafted to cause backslash against LGBTQ+ users, but the golden opportunity of picking out a public example to hang in the main plaza like they used to do with pirates in the old day seemed a very careful planned act. Stir a mess with a poorly written policy, cherry pick a victim and then call in media coverage to present themselves as the ones who discovered this bad actor and are taking serious measures against them.... All of that looks like it was too much of a luscious opportunity to sit out.
    – SPArcheon
    Jun 21, 2023 at 7:50
  • 2
    @SPArcheon I don’t think it was done in deliberate malice towards the LGBTQ+ community specifically – that is, I don’t think hurting the LGBTQ+ community was the goal. But I do think they deliberately looked for a topic they knew would easily become highly inflammatory and had potential to draw in a massive crowd of reactionists, and trans rights and pronouns fit that bill to a T. The fact that it ended up being hugely detrimental to LGBTQ+ users was just immaterial to them. Jun 21, 2023 at 8:35
173

As a moderator, I don't think I've ever used an automated GPT detector when investigating flags from users highlighting potential uses of AI to generate answers. And yet, my moderation practices are now all banned by the new SE polices. This makes no sense to me.

The false positive rate of automated GPT detectors is irrelevant if I don't use them. I don't understand why I'm prohibited from using practices that don't ever involve any use of any automated GPT detector, ever. A high false positive rate would be a reasonable basis for prohibiting mods from using automated GPT detectors; but it is not a reasonable basis for the current policy, which goes far beyond that. I shared this feedback with the company privately as soon as Stack Exchange informed moderators of this new policy, and I have not yet seen a response.

I remain dissatisfied both with the process of rolling out the new rules, and the substance of the new rules.

4
  • 2
    "users highlighting potential uses of AI to generate answers" Could it be that these users used GPT detectors to highlight the potential uses? Then, while not directly using them, you could still have been biased towards false positive detector results. Just saying, I don't think it's a big problem. It's good to see that so many mods actually didn't use the detectors, so it will be interesting to see if the company can maybe come up with a different explanation for their abrupt policy change one week ago. Jun 6, 2023 at 15:53
  • @Trilarion, yes, it's certainly possible!
    – D.W.
    Jun 6, 2023 at 16:42
  • 3
    Isn't this then about the right/imperative to delete content not because it is incorrect, inappropriate, offensive, or plagarized, but simply because you don't like the idea of AI doing a task you think only live people should do? It looks to me like the debate has been obfuscated because that is a more tenuous position than debating whether or not AI detection tools work, which implies that if it could be proven then everyone would be copacetic about weeding out the evil AIs amongst us.
    – goldilocks
    Jun 13, 2023 at 15:06
  • 3
    @goldilocks, No, that is not what this is about. The rationale has been well explained in other locations, and I can't put it any better than it has been explained there, e.g., stackoverflow.com/help/gpt-policy, meta.stackoverflow.com/q/421831/781723, meta.stackexchange.com/q/389811/160917, meta.stackexchange.com/q/384922/160917
    – D.W.
    Jun 14, 2023 at 6:59
136

We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions. People with original questions and answers were summarily suspended from participating on the platform. We stand by our decision to require that moderators stop using that tool. We will note that it appears to be very rare, however, for mods to use ONLY the ChatGPT detection tool, and frequently their own analyses were in use as well. We will continue to look for other, more reasonable tools and are committed to rapid testing of those tools and any suggested heuristic indicators.

So, share the data. You obviously have it, and you obviously have it tagged, if you've gone through impact analysis on it. What are you waiting for?

At this point, it's the third time that I've heard in a week that there's an "unacceptably high false positive rate" without ever hearing:

  • What the rate was before ChatGPT went mainstream
  • What the rate is today
  • How you know that these suspensions were made by using a tool
  • Standard statistics values (r2, etc.) to assess correlation

So far, there's been a lot of "moderators bad" and very little quantitative data for mods to even consider or try to see your side. You've provided unquantifiable, unqualified hearsay, which, if I'm not mistaken, is in breach of your own Code of Conduct.

Like, I don't want to be mean or anything, but the intended reaction to this repeated exposure to unsubstantiated statements that aren't even remotely verifiable has a name: Illusory truth effect

7
110

I came across the news of the strike on a different site, and I thought I could resist pointing out that the company is making exactly the same mistakes it made in 2019, but I can't. I'm probably wasting my breath, because I tried to make these points in 2019 when I was a mod. Maybe repeating them will help.

  • Stop making official policy statements off the cuff in the Teacher's Lounge. That chat room should be for mods to get support from each other and the company, not for official company policy setting. Not every moderator participates in that chat room and chat messages are inherently a terrible channel for official announcements.

The company should make it clear that participating in chat is optional, and that mods who don't will still be included when discussing and distributing official company policy.

  • Stop making unsupported assertions disparaging volunteer moderators in the press. Surely you can talk about what you believe is good for the future of the company without throwing the volunteers who made Stack Exchange the success it is today under the bus.

The data behind the assertions that new users were being unfairly suspended by moderators should be made public, or the assertion should be retracted and apologized for.

  • Do not unfeature posts on site metas. It doesn't help contain or stabilize the situation; It just makes people angrier and more frustrated.

There should be a process that's followed prior to staff reversing the actions of site moderators and the first step should be a conversation with the moderators.

  • Stop treating volunteer moderators like they work for the company. Site moderators are elected representatives of their communities, not extra labor that can be directed to achieve the company's goals. They know what their communities need to stay healthy, and rightly prioritize that.

The company should not be dictating policies intended to prevent moderators from using their judgment to do what is best for their communities. Either the mods can be trusted or they can't. If the company can't persuade the experts that a course of action is good for their community, maybe it's a bad idea.

A few parting thoughts The company does not act like they value the expertise and dedication of the volunteer moderators as a whole. Individual staff may be friendly with individual moderators, but that does not translate into the company including the community and their elected moderators as partners in business decisions that impact them. Many of the veteran moderators are as (if not more) invested in the success of Stack Exchange as the shareholders and management.

Maybe moderators should join together in a formal independent organization that can speak with one voice to the company for its members. That might make it easier for the company to hear their advice on various policy decisions before things escalate.

8
  • 15
    IMHO any mod who does not regularly participate in the TL should simply continue to adhere to the public version of the policy as if the non-public version does not exist. Let the company explain why this is problematic.
    – Kevin
    Jun 7, 2023 at 3:33
  • 6
    The official (private) announcement is on the Moderator Team (an instance of the "Stack Overflow for Teams" product, with access restricted to staff and diamond moderators), not in chat. There have been many official policy statements made off the cuff in TL, but this main one is documented on the Mod Team. It is, of course, unclear whether participation in the Mod Team is a requirement for moderators, just as it remains unclear whether participation in chat is a requirement. Then again, those of us who are involved aren't interested in rules-lawyering this. We just want to be reasonable. Jun 7, 2023 at 10:06
  • 1
    @CodyGray-onstrike Thanks for the correction. I’m not sure which post led me to believe it was the Teacher’s Lounge again. The Mod Team is a fine place for policy discussion because you don’t have to be in the Eastern time zone to have a chance to be involved in the discussion, and if you aren’t a native English speaker, you can read at your own pace. Chat is for hanging out and getting immediate help with something.
    – ColleenV
    Jun 7, 2023 at 10:23
  • 1
  • 1
    @msh210 Moderators can not sue for defamation if they are smeared as a group. That agreement is for individuals. And saying a group is in general incompetent doesn't really rise to that level regardless. It's just disrespectful and indicates that there's either a serious disconnect between reality and someone's perception of it, or someone thought blaming mods for the company's failure to manage the situation was going to be the easy way out.
    – ColleenV
    Jun 7, 2023 at 18:44
  • @ColleenV suing for defamation has nothing to do with the moderator agreement: I mean, one could sue for defamation even if the agreement didn't exist. The agreement goes beyond that and says SE will… well, no point in quoting it again. It's in my previous comment. I don't see anything in that clause about such newsmongering being disallowed only if names are mentioned.
    – msh210
    Jun 7, 2023 at 20:45
  • 1
    @msh210 The reason that clause exists is because SE was getting sued/taken to arbitration. Regardless, the press policy has no indication of what recourse moderators have if it is violated. You’ll have to go through the arbitration process to find out I suppose. Since I’m no longer that active here, I have not been keeping up with changes to the legalese.It says… clear directives in place to not speak about individual Stack Exchange/Overflow network moderators or users without the express written permission of the user.
    – ColleenV
    Jun 8, 2023 at 1:28
  • @Kevin why limit this to mods who haven't been regularly active in TL? Seems to me that ALL mods should simply follow the publicly stated policy, and ignore any secret policies to the contrary.
    – Wildcard
    Jun 25, 2023 at 16:26
75

Apparently people have been upvoting spam. I don't know if this is an isolated thing or a coordinated effort, but it's not at all helpful. Indeed, I'd suggest that CMs consider warning and (if not heeded) suspending people who upvote obvious spam. It's functionally the same as a voting ring after all. Personally I find that behavior counter-productive to the reasons I signed the letter.

I got to thinking about voting rings because they present similar problems to detecting machine-generated content. It's less of a binary switch and more of a Bayesian inference. My memory might be fuzzy, but here's how voting ring investigations typically went down:

  1. A user would flag posts that seem to be getting more upvotes than expected.
  2. A moderator would see that flag and look into the situation using moderator tools built into the site. They'd also bring experience and past history into their judgements.
  3. If the moderator thought it worth more investigation, they'd contact a CM to (potentially) invalidate votes.
  4. We'd use a variety of tools and techniques passed on from one CM to another (thanks Shog and Tim!) to make a final determination.
  5. If, in our judgment, voting fraud had taken place, we'd invalidate some votes and message the users we suspected of being in the ring.
  6. The users would deny that they'd done what we said they did.

Each step would increase or decrease confidence that voting fraud had happened. For instance, an overly-aggressive flagger might decrease our estimate of the odds because they cry wolf so often. In my experience, moderators usually identified voting rings accurately, so when they brought one to my attention, it raised my estimate considerably. Some of the tools were extremely reliable and others weren't worth listening to most of the time. You learned to make these evaluations only through experience.

I remember one time we'd caught a pair of users dead-to-rights. One of the users (we'll call them A) said they had no idea what had happened. The other user (B) confessed that they were married to A, saw that A's posts were getting downvoted and upvoted all of A's posts to help them feel better. Now. That's a textbook voting ring, but I wanted to give that couple a hug and let them off the hook. We invalidated the votes anyway and told B to maybe try something else to cheer up A.

We'd also see things like two experts in a specific technology upvoting each other's posts. If we just used the voting fraud tools, we'd be compelled to suspend them. Another time we applied one of our tools to employees voting on Meta and found that the voting ring was coming from inside the house! Since we didn't just blindly do what the voting fraud tools recommended, it was an inside joke rather than a soul-searching investigation. Context matters so much!

We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions. People with original questions and answers were summarily suspended from participating on the platform.

One of the things that has made this issue go from 0 to 100 in a week is that we haven't seen any details of this analysis. Like the vote fraud tools I just mentioned, false positives are not surprising if used in an inappropriate context. In my experimentation with GPT and GPT detectors, I found that my own original writing could be flagged as likely machine generated. So that's not news.

What would be news is if we could see evidence of moderators blindly using GPT detectors without other signals to make their decisions. That sort of behavior is not unheard of, but its rare on Stack Exchange sites. Moderators here tend to be extremely diligent. Given the volume of flags, mistakes are inevitable, but most moderators are quick to own up to them when confronted. That's one of my priors that makes me suspicious of the analysis even before it's made public.

The final sentence about "original questions and answers" also raises red flags for me. I guess what bugs me is that if your standard is "original" you can't go far wrong with machine-generated content. How many times have you searched for something (recipes are particular offenders), found a promising page and then scroll (and scroll) until you find the actual information you need? (I get it! You spent a summer in Tuscany and that's how you know so much about spaghetti carbonara.) Since it's so easy to generate reams and reams of verbiage using ChatGPT, "original" seems to skew the playing field away from (most) humans.

I don't know if this is intentional, but the most recent CEO blog post (CEO Update: Paving the road forward with AI and community at the center) seems to argue that people in the community serve AI rather than the other way around:

Our community has given us feedback through the evolution of this tool, and their feedback is critical to how it scales. As the AI landscape continues to evolve, the need for communities that can nurture, inform, and challenge these technologies becomes paramount. These platforms will not only offer the necessary guidance to refine AI algorithms and models but also serve as a space for healthy debate and exchange of ideas, fostering the spirit of innovation and pushing the boundaries of what AI can accomplish.

The words are fine, but in the context of disagreement between moderators and the company about how to moderate machine-generated content, it sounds dismissive. It's as if the company believe it would be making great strides in technological development if only those pesky humans would get on board. Or maybe I'm being unfair and this is just an "AI meets community" pitch. (I suppose I'm not the audience for the CEO's blog, though.)

I agree that one value of communities is improving a product. But that doesn't happen if the community isn't free to reject the product or wait until it's reached some level of maturity. When the community signals that "original" content isn't enough, it doesn't do the product any good to ignore that signal. Because with few exceptions, the people who curate content on Stack Overflow also consume it.

9
  • 9
    I have been traveling most of the day so haven't been paying attention to spam today, but unfortunately up votes to spam aren't entirely uncommon. They usually just get removed fast enough via flags that it doesn't matter.
    – Andy
    Jun 5, 2023 at 23:17
  • @Andy: That, unfortunately, isn't surprising. It is hard to determine whether voters are confused, malicious or just random. (I don't have first-hand knowledge either.) Jun 5, 2023 at 23:53
  • 2
    I would tend to imagine that the spammers who care are also operating voting rings, or will soon be doing so, because now they know that 90% of the network's anti-spam tech (SmokeDetector etc.) is offline. Who's going to stop them?
    – Kevin
    Jun 6, 2023 at 1:25
  • Please let's not overload "spam" to deviate from the strict definition we use on the site itself; unsolicited promotion (of products, services, religious beliefs, mathematical "proofs", conspiracy theories, etc etc)
    – tripleee
    Jun 6, 2023 at 4:59
  • 1
    @tripleee: What word should be used instead (not a rhetorical question)? Jun 6, 2023 at 14:11
  • 2
    @This_is_NOT_a_forum I guess Jon means "AI-generated content" in this instance. More generally, content which is disruptive can be trolling, rude, abusive, off-topic, or just annoying for other reasons. Perhaps I would suggest "disruptive content" as a collective term for all of these phenomena, but it's also possible that Jon means something else here.
    – tripleee
    Jun 6, 2023 at 14:21
  • 3
    @tripleee: I'm talking about literal spam that CMs needed to delete. It's a natural consequence of the strike that spam will sit around longer than normal, but it's not helping anyone to encourage it in any way. Jun 6, 2023 at 15:22
  • 2
    Thanks for clarifying. It was not obvious to me how upvotes on spam were pertinent to this particular question but I think I see now.
    – tripleee
    Jun 6, 2023 at 15:27
  • 1
    Upvoting spam in protest is definitely a misuse of the voting system, but addressing misuse of the site's features is a moderation task (e.g. warning or suspending users). So I would encourage mods who are on strike from performing moderation tasks to not act on this until the strike is over. (I would also encourage these voters ─ if they are indeed doing it as a protest ─ to cease doing so, because it doesn't really harm SE, Inc. in any way for spam posts to receive upvotes, what harms them is for spam posts themselves to go unmoderated.)
    – kaya3
    Jun 7, 2023 at 13:44
73

We stand by our decision to require that moderators stop using that tool.

On the site I moderate, a perfectly well written, but entirely incorrect, answer can literally ruin the lives of anyone who decides to follow that misleading information. I have seen this happen on another immigration site that I've been highly active on. I should be able to use any tool whatsoever to prevent that kind of disaster from happening. If Stack Exchange has a problem with this kind of responsibility, maybe it would be best if they simply shut Expats down.

And there are probably a few other sites that they should take a hard look at because of the potential repercussions.

9
  • 4
    I feel this answer is a bit problematic. They're still "allowing" deleting incorrect answer and deleting AI-ban, just not allow using the detection tools. (of course the "internal rules" is internal, maybe it's more restrictive, I've no idea.)
    – user202729
    Jun 6, 2023 at 9:20
  • 8
    @user202729, I don't know about Expats, but at least at SO, incorrect answers are not deleted (and not even a subject for flagging, such flags are consistently declined), but rather subject for simple downvoting. Technically lying is allowed on network, unless it's political (according to new CoC)
    – markalex
    Jun 6, 2023 at 9:32
  • 4
    @user202729, regarding "internal rules": there are no public statements about it, but based on related statements, I will guess that internal announcement prohibited mods from taking any action, regarding AI content unless said content violates other clearly established policies or rules.
    – markalex
    Jun 6, 2023 at 9:36
  • 39
    The internal rules are much more restrictive @user202729, to the point where, effectively, we have no grounds for removing content that we strongly believe was generated by an AI. There is one narrow exception that would allow us to delete such posts, but this has not been disclosed publicly, and, in practice, it is meaningless. There is no objection among mods to avoiding the use of "detector" tools; in fact, none of the SO mods have ever relied on them. Some of us tried them in the early days after ChatGPT's public release, but determined they were useless. Others never even tried. Jun 6, 2023 at 11:25
  • 15
    It will be wonderful when we will be forced to leave ai-generated posts live on sites like Medical Science...
    – SPArcheon
    Jun 6, 2023 at 12:32
  • 7
    @user202729 Incorrect posts aren't supposed to be deleted in most cases; they're supposed to be downvoted. The problem is that AI-written answers fool enough users that the voting system often rewards instead of penalising them, regardless of their actual correctness. Mods are not supposed to be subject-matter experts, and are not generally expected to be able to judge the correctness of a post. Also, deletion is not enough to deal with a flood of superficially-good-looking nonsense from AIs, the users doing this need to be suspended; but users can't be suspended for making incorrect posts.
    – kaya3
    Jun 7, 2023 at 13:49
  • @user202729 the new rules implicitly disallow the deletion of any AI generated content, no matter the reason...
    – jwenting
    Jun 13, 2023 at 9:42
  • 4
    @kaya3 and jwenting, The problem for sites like Expats, and maybe a few other sites like the aforementioned Medical Science, and perhaps Finance, Information Security, Interpersonal Skills, Home Improvement, ...probably several others... is that leaving incorrect information - especially perfectly formatted/written - can mislead readers into making disastrous decisions. Not everybody who visits the site will understand (or ever care to understand) the voting system and what it means. And we can't assume anybody reads the comments.
    – ouflak
    Jun 13, 2023 at 9:56
  • 4
    TBH I have seen this pattern a lot of late. "It's probably fine for Underwater-Basketweaving.SE, but it's definitely not okay here on Drupal.SE".... and then later someone from Underwater-Basketweaving.SE chimes in pointing out that it's super important not to leave these AI-generated answers up because of the consequences. The simple conclusion is that everyone wants correctness in SE answers, on every site. If you think Philosophy is exempt, or Parenting, or Academia, or SO for that matter, you just don't know enough about that site. Jun 15, 2023 at 7:11
60

There's plenty to be dissatisfied with, but I've raised what I have to raise about that already. So I'm going to try to put on my "Pollyanna hat" here. Update from future me: You're making it pretty hard for me to keep the hat on.

We will continue to look for other, more reasonable tools and are committed to rapid testing of those tools and any suggested heuristic indicators.

Will you be making the mods part of that work? I'm sure they'd have much of value to share (Ex. @sideshowbarker (see also their post here)). Other community members also come to mind (Ex. @NotTheDr01ds (see also their posts here, here, here, here, here, and here)). (This doesn't have to be a public conversation- if I understand correctly, we've been trying to avoid showing the people violating the policy how we sniff their violations out).

The Community Management team is evaluating the current situation and we’re working hard to stabilize things in the short term.

I'm thankful that the CMs are picking up on some of the moderation work during the strike (Ex. @Yaakov) on top of their other work. I keep needing to mentally remind myself not to shoot the messengers (or anyone for that matter). A nice side-effect we might see here from the CMs dogfooding is some empathy for pain points in mod tooling. Maybe something nice will come out of that.

We’ll be working collaboratively with the community on the long-term solution.

That's good to hear. I look forward to it. When will we hear more on this? (Please also don't forget about the questions we've asked about the new policy).

53

The Community Management team is evaluating the current situation and we’re working hard to stabilize things in the short term.

If, by "short-term stabilization," you mean ending the moderators' and curators' strike, then the solution is relatively simple: Just apologize – publicly and unreservedly – for the confusing and insulting manner in which this so-called policy change was announced: privately (to the moderators) and publicly (in a way that was at odds with the private announcement).

I regret that actions have progressed to this point.

Although this may be considered the seed of a beginning of a vague apology, it simply doesn't go far enough. Also, it carries an implicit hint that the moderator team has played a significant part in creating this mess – which it hasn't.

We’ll be working collaboratively with the community on the long-term solution.

This "community collaboration" should have been present from the get-go (as it was when first implementing the ban on AI-generated content). Maybe it is now time to hit the "rewind" button and ask for suggestions on how to address the (unsurprising) "large upswing" in account suspensions.

In the meantime, restore the moderators' rights to use suspensions (the vast majority of which, I gather, have been for periods of seven days or shorter). You (the staff) need to trust your moderators to not issue suspensions (of any length) trivially and without full consideration. In the longer term, there may be a way to avoid such suspensions by using and/or fine-tuning the in-built "answer ban" system, instead; from what I understand about that (the details of its algorithms are, understandably, kept secret), this can actually work out to be a more effective "blocking tool" than a suspension, as it is: (a) potentially for an unlimited time period; and (b) covers all posts from a given IP address. And we certainly need some sort of block on the vast majority of AI-generated gibberish that gets posted on our network.

3
  • 25
    Some SO mods were originally (in early days, say Dec 2022 – Jan 2023) issuing suspensions for 30 days, consistent with site policy as set by staff and endorsed by Phillippe. We discussed this internally, and decided that it was too steep, so we ramped it down to 7 days (the normal duration for a first-time suspension). We'd even been discussing internally not suspending at all for first offense and simply sending a warning—and would have been more than willing to start doing this all the time—when this edict was handed down. Staff didn't discuss with us. Jun 6, 2023 at 11:22
  • 6
    Apology alone isn't enough. Things need to change to rebuild trust.
    – Ian Kemp
    Jun 6, 2023 at 15:44
  • @IanKemp The first part of my last paragraph addressed that, partially. Jun 6, 2023 at 16:06
46

People with original questions and answers were summarily suspended from participating on the platform.

You still are yet to give a single example. Wow, that gives this all a ton of credibility.

UPDATE:You have given some data, but I see no reason to think that it is realiable. I concur with others complaints on the post and won't restate them here.

7
  • There was an example on MSO where someone claimed that. In all fairness that MSO question ought to be found. I don't remember if the claim was substantiated or not. Jun 6, 2023 at 13:56
  • 2
    @This_is_NOT_a_forum Do you mean Why was my answer deleted for using ChatGPT even when I didn't?? Jun 6, 2023 at 16:01
  • 2
    Oh, one example, that no one can even confirm is really true. Well, that makes it all okay @Trilarion Jun 6, 2023 at 16:02
  • 3
    @Starship-OnStrike No need to be picky. I just try to be helpful. You probably know as well that the absence of evidence is not the evidence of absence. Jun 6, 2023 at 16:05
  • 5
    Okay, fair point, I'm ust saying that the absence of evidence contributes to the absence of trust @Trilarion Jun 6, 2023 at 16:50
  • 2
    Fully agreed. The company claims a connection somehow. It's their duty to provide the proof. I hope they come around and eventually see the light. Jun 6, 2023 at 17:25
  • @Trilarion While most of us understand this to be about a much larger moderation issue, the official explanation focuses on the supposed problem of moderator tools regularly leading to such erroneous deletions. Do you have any reason to believe a false positive result led to the issue with this case? From what I read, it was not distinguishable from regular human error. It is possibly an example of overzealous moderation against potentially generated content, but since Philippe didn't actually address that level of policy, it's almost pointless to ponder. Still, good share.
    – BryKKan
    Jun 7, 2023 at 18:51
43

Which other groups’ interests are missing from this post?

In many complex subjects a number of constituencies need to be considered.

From this post, one can identify the following groups/interests:

  • moderators

  • users that are, correctly, or possibly incorrectly, flagged as having gone against the rules.

  • implicitly Stack Exchange, Inc. corporate interests. While others may want to dismiss those I will say that, for all the complaints, the family of Stack Exchange sites have been of tremendous value to professionals and hobbyists all around. Second, a corporation that does not have a plan to make money, or at least fund itself according to expectations, risks disappearing or takeover.

However two other groups are missing and are very much impacted by this policy:

First, users that come to SE looking for answers to a problem that they have.

Stack Overflow, while sometimes criticized for fostering cut and paste programming, generally fits the bill. Some primary ways to judge the quality of answers are: your own knowledge, the up down votes, the presentation of an answer, along with supporting quotes and program outputs and the poster's reputation. Even if the poster is not the most competent on the subject, a user can assume repeat research and posting will gradually have improved their knowledge.

With AI allowing cheaply-produced, but perfectly worded answers, neither the presentation, nor the poster's reputation has any great meaning, leaving the user to their own to judge how helpful an answer is liable to be, except for community upvotes which can be swayed by confidence and presentation, something generative AI excels at.

Respectfully, that constituency is not benefiting in the least from your proposed hands-off approach.

And, much more so than just the moderators, if regular users cease to see Stack Overflow as a valid source of information, they will go elsewhere.

Second, users that answer questions.

Their motivation may vary from reputation seeking, interest, evangelism, desire to help, challenge, pursuit of professional credibility.

They decide to join SE sites and "grind" their way up, by answering question after question.

How is that group likely to react to a situation where they are systematically pre-empted by people who can write up a complex and perfectly-formatted answer - that they often don't really understand and don't bother vetting - in 5 minutes or less? While it takes even a knowledgeable person 3x the time to write up a concise answer, with some minimal code and showing output?

Like the first group, it is quite possible they will not bother. And without a solid core of them, there is no site.

Though it is a very imperfect comparison, I would note that schools that have, for whatever reason, taken a relaxed attitude towards cheating, and academic standards in general often end up devalued as a result: in the eyes of prospective students, and in those of prospective employers.

Generative AI content is going to be a big challenge, for this site, and others. You have indicated concerns with moderator measures as done so far.

Fine, to an extent. Work together on a better solution. But the answer is most certainly not throwing out AI content moderation as a whole.

The striking moderators, to me, ultimately seem to be motivated by looking after users in the largest sense: regular users posting questions and answers, without using shortcuts, and especially not knowingly using shortcuts with well-known imperfections at the current time.

As a regular, non-moderator, not particularly "community" minded, user, I would strongly encourage SE, Inc. to do the same: do well by your users.

Better said on this answer:

p.s. I intentionally stayed out of considerations about communication issues, perceived or real shortcomings in how moderators were treated, community stewardship, etc…. I do have my opinions, but I wanted to keep the focus on why the proposed policy is bad for regular users and a better collaborative solution needs to be found.

6
  • 11
    As (primarily) an "answerer", I can tell you that it's annoying to have to inspect AI answers. In my experience they are usually wrong or, at best, very generic on the general topic, and of course they have no apparent sensitivity to the particular question or the probable sources of confusion that led to the post. It's also annoying and telling that it's more and more frequent for a question to explicitly include a claim that the person asking first asked ChatGPT. All I can do is downvote, and actually it's not uncommon for people to delete an answer if they honestly didn't know.
    – Pointy
    Jun 6, 2023 at 13:59
  • 1
    @Pointy So, what you are saying is that ChatGPT, as it exists currently, is flawed as an aid to answering questions, even if you are knowledgeable and taking care to correct for its shortcomings?
    – JL Peyret
    Jun 6, 2023 at 17:12
  • 1
    No, that isn't what I meant. In fact if you re-read my comment, I specifically stated that I inspect Chat-like answers. Once or twice in recent memory, the answers were good enough that I didn't downvote even, and simply left a comment linking to the policy page. Most of the time, the answers are simply not good however. Of course I have no way of knowing how much effort the poster expended on verifying the answer, just like with any other answer. All I can do is read the text and use my experience to determine the quality. And for what it's worth I don't downvote answers very often.
    – Pointy
    Jun 6, 2023 at 17:18
  • 9
    Furthermore, one of the flaws (in my opinion) with the generated answers is that they're deceptively long. There's a lot of "fluff" text. Good long answers (I invite you to look for some T.J. Crowder answers) are packed with information and sensitive to the specifics of the question involved.
    – Pointy
    Jun 6, 2023 at 17:20
  • 1
    Oh, sorry, I thought you had experimented with starting out answers w Chat.
    – JL Peyret
    Jun 6, 2023 at 17:20
  • 3
    Oh gosh I see what you mean now. No, I haven't bothered with that; I meant looking at answers that I "sniff" as being ChatGPT, and reading through to see if there's anything worthwhile. I'm not a moderator so I can't ban anybody; all I can do is comment and (sometimes) downvote.
    – Pointy
    Jun 6, 2023 at 17:21
36

I am not a moderator, and I have no affiliation with any of the moderators - involved here or otherwise. This means I have no access to the "private policies" behind this. The upside is, I have no obligation or commitment to non-disclosure. On the other hand, everything to follow is by its nature speculative.

Edit: Update: Since writing this I have read a number of other related meta questions and answers, authored by both community members and staff. I also finally read the moderators' strike letter. Importantly, I have still not seen any "secret" materials. Nonetheless, at this time, I feel confident saying I was correct below, regarding the secret policy. SE Inc is very likely preventing moderators from removing posts that violate our community expectations, and harm the health of the community as a whole. Thus I have also since signed in support of their open letter.

From the body of the official "question", one might be forgiven for thinking that mods were in an uproar over "access to a [flawed] tool". You skillfully framed the issue as a move to protect the community from self-inflicted harm. However, having observed the community reactions to past debacles like this, the story simply isn't credible. Such a massive strike would only be undertaken if the policy issue were truly existential. Something is missing. Even without the other comments here, we might guess what that is: some "private policy" which mods feel bound to withhold [but which I'm free to speculate about].

So what is this secret new policy direction? Well, we can actually just read between the lines of the official statement, applying a basic understanding of standard PR tactics. This is clearly a deflection, and those work best to defuse a reaction if they are focused on something "innocuous" which is very near to the true subject. This is to avoid a sense of "narrative novelty" when people encounter discussions of the "real" issue, in hopes they'll dismiss it without consideration as something they've already evaluated. By this reasoning, we might say it likely has something to do with AI generated submissions.

Now the cynic in me comes out, and I ask myself "Why would SE want to hide this policy shift?" After all the community issues of the past few years, surely anything so contentious as to prompt mass strike would have been appropriate to discuss openly first? If they're playing the corporate trump card, and trying to gag the mods, then it must surely be rooted in profit motive. So really the question to ask is "What does SE or it's leadership think they stand to lose (or gain) in this area?"

Bringing this all together, I feel pretty confident hazarding a guess as to the policy, and mindset of the SE side:

Basically, they don't want AI generated content removed at all, unless it's so flagrant as to make the site appear useless.

Their reasoning is likely based in a combination of factors, ranging from fear of direct competition by AI tools to a more straight-forward play to assuage investor concerns about userbase growth. It's also likely they want to feel free to lean into that themselves eventually, and use AI tools to answer questions directly. The cynic in me wonders if they already tried it without telling anyone, but so far I haven't seen anything to suggest it. In any case, from a "catalog" management perspective, I'm sure they see having generated answers to orphan questions as better than having none, and more "active users" as better than fewer. It's easy to see why they might view moderator efforts to remove AI "contributions" as a nuisance to their bottom line.

Again, I could be completely off-base. But if I'm not, this is a serious error on the part of SE, and I hope they recognize this and reverse course in time.

From a business perspective, this move may appear savvy on the surface, but it actally is an act of brutal self-sabotage. The value of SE isn't being the place with all the answers. It's the community. Disregarding it in this way can only erode any long-term value you hope to create in the company. As a "regular end user", I'm watching this with an eye towards walking away entirely (and advising everyone I know to do the same).

Whatever is going on, now is the time for honesty and transparency. The idea that mods here are being coerced into hiding any policy at all does not sit well with me. Tread carefully SE.

9
  • 7
    The cynic in me wonders if they already tried it without telling anyone Yeah, that's not cynical at all. I'm betting half the companies on the planet with a development department have tried spinning up some kind of PoC. Right now there is a huge drive from investors and market analysis agencies to push AI as a tool to be used. I'd be more surprised if SO hadn't tried playing around with it.
    – DavidG
    Jun 7, 2023 at 14:11
  • 9
    My current personal theory is that this truly is more about the user suspension angle than the content side, which feels consistent both with the language in the policy post, as well as Stack the Company's broadly laissez-faire attitude towards improving site content quality. There seems to be a very strong force in the Company right now that is seeing the "large amounts of user suspensions" metric and panicking, despite the fact that the suspensions are justified and useful to maintaining the site as a resource that people widely use, which erodes when quality wanes.
    – zcoop98
    Jun 7, 2023 at 15:38
  • 8
    I'm imagining that with the influx of AI posts, suspensions went through the roof, in a way that hasn't been seen internally much before, and if someone, let's say someone higher up for sake of argument, sees that metric without understanding the context, without knowing that the suspended users and their content were never authentic in the first place and weren't beneficial to the site, without getting that the removals are a net positive to maintain Stack's "community trust", the thing the CEO cites as Stack's biggest asset... that's something I could see causing this debacle.
    – zcoop98
    Jun 7, 2023 at 15:43
  • 7
    The value of SE isn't being the place with all the answers. As another regular user, I fully agree. My use of this site isn't for its having a lot of content. It's because it has correct answers, experts presenting commentary as needed to fix misconceptions or fill in gaps implied in the question, and a minimum of fluff I have to wade through to get to what I need. AI might sometimes give the right answer, but my overall experience will be degraded.
    – ojchase
    Jun 7, 2023 at 16:15
  • 4
    @DavidG I think moderation strike turned this whole affair into PoC. Because this is exactly how it is going to work: ChatGPT in / content curation out. Because nobody in their sane mind would agree to mode/rate loads of automatically generated content for free
    – gnat
    Jun 7, 2023 at 16:41
  • @zcoop98 I don't disagree with you, and I'm certain that corporate misapprehension underlies at least a good part of the sense of fatalism and grim determination to err we see in Philippe's attitude.
    – BryKKan
    Jun 7, 2023 at 20:16
  • 3
    @ojchase Thanks for expanding on that specifically, I wasn't sure how to state it positively. You're exactly right. The greatest value of having human experts answer your questions is not getting the answer, but rather having them analyze the question itself for cues to hidden personal knowledge gaps.
    – BryKKan
    Jun 7, 2023 at 20:25
  • 3
    As long as we're speculating, I think there are at least two things going on here. One is very likely a reaction to a spike in user suspensions, much as zcoop98 suggests. The other is that the current CEO is a big fan of generative AI, so although they may not specifically want bad answers on the site, they don't want to ban generative AI as a general policy. But I suspect there is also a third disconnect: I think that corporate views SE as a service providing answers, whereas the invested membership tends to view it first as a group effort to curate knowledge. Jun 10, 2023 at 13:45
  • @JohnBollinger Agreed 100%. Couldn't have said it better myself. It's disappointing because the SE community is positioned to provide tremendous value over the next few years if the tide goes the other way. Training models from data that's contaminated by the output from earlier ones is a fool's errand. And SE Inc, by cooperating with the community effort to check this contamination, could position itself for several interesting and potentially lucrative moves into the sector. They have real-time access to the data, and they can prune that down any way they want to hone custom models, for one.
    – BryKKan
    Jun 10, 2023 at 18:10
29

We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives, which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions. People with original questions and answers were summarily suspended from participating on the platform. We stand by our decision to require that moderators stop using that tool.

I think this is a serious concern that should be addressed, but I think it also misses the point a little bit.

The main reason I signed the letter (something I rarely do) is the heavy-handed approach that seemed to have been taken. I think your concerns are completely valid and very much should be raised, but on the other hand moderators are also struggling with "ChatGPT spam". I don't really know what a good solution for that would look like exactly, but I do know that a heavy-handed "we have decreed that ..." is not it. It's a difficult problem and there are no quick and easy solutions.

It's not really about the "ChatGPT policy" or whether or not these tools are good or bad per se, it's about a mismatch between "we're all in this together and let's decide together what the best course of action is" vs. "we decided, end of discussion" (crudely put). Moderators are volunteering their time and in return it's understood they have some say in policy. You can't "just" order people like they're employees, so you need to spend time and effort convincing people you're right. This is work, but still cheaper than paying people to moderate the site. And sure, in the end Stack Overflow Inc. has the final say and that's fine, but if you're not even spending the effort...


As for this specific policy, it seems to me that the problem is that moderators are suspending users based on too little evidence and are, apparently, not taking the full situation in to account when making their decision, relying too much on the output of a single tool. I can't really judge how often that happens, but it certainly seems more than plausible that moderators have made mistakes here and that's obviously not a good thing, but the alternative is a fuzzy "this looks like ChatGPT" underbelly feeling, and that'll have a lot of false positives too – perhaps even more? I've certainly seen plenty of "ChatGPT paranoia" around, and most of this probably didn't use any tools.

The tool is not the problem; "How do we deal with ChatGPT?" is the problem, and as far as I can tell, there is no good solution right now. Any tool can be used inappropriately. If there is a problem with moderators making bad decisions then that should be taken up with those specific moderators.

For what it's worth, personally I don't even think using ChatGPT is necessarily bad in and of itself; see my other answer. I'm all for judging posts more "holistically" rather than just "you're using ChatGPT!" as that seems too simplistic. It's really about the heavy-handed approach as far as I'm concerned.


Edit: thinking about this some more, I'm also reminded how conflicts were handled in the past. I was involved in the launching of the Vi & Vim site quite a bit and a number of employees were pretty strongly against the concept initially. But the disagreement played out different: it was well argued, and we were allowed a chance to prove "you're wrong", and this turned out well.

The controversial Stack Overflow Documentation effort also springs to mind; I felt it wasn't the right direction, but Jon Ericson tried very hard to make it work by engaging with the community and I always very much appreciated that (even though the community was overly harsh at times, to the point I felt rather bad for Jon).

I'm not saying any of this is easy or that the People of Stack Exchange can't be a difficult bunch to deal with – they can – but that's kind of what you get when you rely on volunteers.

In comparison, this feels like ... bland marketing-speak. I get that community management is hard, but by being bland and generic you're not really engaging with anything substantial and nothing ever gets hashed out.

9
  • 2
    Agreed with this; there isn't a "perfect" solution for determining what is and is not ChatGPT-generated content. Whenever I flagged an answer, I looked at more than the content; user's post history (if any) to get an idea of writing style, syntax, etc., timeline of activity (i.e. 2+ multi-paragraph answers from a user within seconds). 100% of the cases I flagged resulted in a suspension (I looked at the user's profile following my handled flags). Is it possible those were false-positives? Of course! Aside from the user saying it's ChatGPT (which did happen), we can't know with 100% certainty.
    – Tim Lewis
    Jun 5, 2023 at 21:06
  • 20
    The company just wants you to think that AI detection tools are front and center. See e.g. this first-hand account. The company's persistent focus on AI detection tools is gaslighting at this point. There's "AI detection tools are unreliable" (what everyone agrees with), and then there's a unicorn leap to "moderators are suspending all these innocent people"---except there's zero data backing up the second part despite repeated, emphatic requests from the community (including mods!) to do so. Even "orders" would be fine if they made sense. Jun 5, 2023 at 23:46
  • 2
    Personally, I don't really like divining motives @AndrasDeak--СлаваУкраїні; maybe you're right, but who knows? I think it's entirely plausible there's just 1) a (spectacular) communication failure, 2) a failure to understand what the relationship between Stack Overflow-the-company and Stack Overflow-the-community looks like from the community's perspective, 3) an error in their interpretation of the data, or 4) a combination of any of the above. Jun 6, 2023 at 0:11
  • 3
    The last time there was a major conflict (Monica, which I saw up close as a moderator at the time) I think everyone involved was fundamentally well-intentioned, even though Stack Overflow-the-company clearly acted badly, IMHO. Jun 6, 2023 at 0:15
  • 13
    There have been way too many attempts from mods and general community to get a decent response/clarification from the company, to no avail. They are doubling down. We're way, way past good faith here. Ask yourself why this post is featured when a bunch of child meta strike posts are being silently unfeatured by Philippe. Probably another misunderstanding. Jun 6, 2023 at 0:16
  • 1
    @AndrasDeak--СлаваУкраїні Featured posts here are shown on every single SE site; do you really want 4 different featured posts on the same topic? Probably not. You can find "bad faith" behind every action if you try hard enough, but that's rarely a constructive path forward. Jun 6, 2023 at 0:39
  • 7
    Do I really want 4 different featured posts on the same topic? No. Do 4 featured strike posts on 4 different child meta sites (e.g. Academia meta) show up on all other sites? Also no. Jun 6, 2023 at 7:23
  • 1
    I think you've missed the point here. There's no evidence moderators were actually "going overboard" at all.
    – BryKKan
    Jun 7, 2023 at 12:55
  • It was intended to be a qualified "if that's the case" @BryKKan; I'm really not in a position where I can judge one way or the other. To be honest I can't be bothered to edit it now to clarify that, as it seems engagement from the company is basically zero – so what's the point? Jun 7, 2023 at 13:21
29

Honestly Philippe, knowing you as I do, I think you know better than this.

If you're right, and if you believe in what you wrote, this is a case of self-feeding vicious circle of knee-jerk reactions to rapidly evolving patterns of contribution and moderation, with the community and the "powers that be" going in opposite direction.

From our experience at Wikimedia you too can probably easily list a dozen past cases just at the English Wikipedia, starting say from the Seigenthaler incident. We need a de-escalation soon, and it's quite rare for things to de-escalate on their own while the opposite sides just dig in. (That tends to happen only if the proximate cause of the incident turns out to be a tempest in a teapot in hindsight, and even then it often takes a long time to recover.)

25

To respond to a few things here:

We ran an analysis and the ChatGPT detection tools have an alarmingly high rate of false positives...

How did you do that analysis? Do you have sample posts that you know for a fact that were human-generated and posts that you know for a fact were AI-generated, or were you using a different approach?

...which is correlated to a dramatic upswing in suspensions of users with little or no prior content contributions.

Was the content they were generating good quality? The cases of new accounts posting suspected ChatGPT content I've seen weren't exactly producing stellar answers. Is it really that problematic to suspend someone who was producing bad content anyway (regardless of whether it was ChatGPT)?

People with original questions and answers were summarily suspended from participating on the platform.

Again, are you aware of specific individuals that you know for a fact were incorrectly suspended? How did you determine that, unless you yourselves have a way to reliably distinguish between human-generated and AI-generated AI? If you don't have a way to do that, aren't you kind of doing the same thing you're accusing moderators of doing?

7
  • 13
    to be fair, while I expect that a detection tool test was made on labelled samples (samples you know the nature) at the same time your post opens a tangential far more interesting question: since tools don't work and the mods were apparently told that their logical thinking doesn't too, how was the company able to validate that the false positives banned users on the site were indeed false positive? Surely not just because the banned users told them so, right?
    – SPArcheon
    Jun 7, 2023 at 15:28
  • 1
    You're expecting logical consistency from SE? Well, I guess the illogic just proves they're human. @SPArcheon
    – W.O.
    Jun 7, 2023 at 15:36
  • 2
    I have no intent to offer defense for SE in this case, but I also feel it's worth noting that it's possible for the moderators and Staff to come to the same conclusion (that "AI detection is nigh-impossible to do with absolute certainty"), and arrive at two different forms of action from that conclusion (E.g. Mods with "Let's deal with what we can, a small amount of false positives is a necessary cost to stem the flow" vs. Staff with "We can't hope to do this accurately or evenly, the least costly option is to let everything through to avoid benching real users").
    – zcoop98
    Jun 7, 2023 at 15:51
  • 4
    @SPArcheon false positives banned users actually they didn't say that, they just formulated their unsubstiated claims in a way that readers might come to such a conclusion, and they can back out If needed. They talk about a) false positives by tools (agreed by all) b) correlation of suspensions to false positives .. what/how exactly? cant make much sense of their sentence c) (new) posters with original content (both answers and questions - never seen the latter, btw) were suspended - implying they were mistreated.
    – kleopatra
    Jun 7, 2023 at 15:52
  • 3
    it's nastily clever spun pitch ..
    – kleopatra
    Jun 7, 2023 at 15:57
  • 10
    @kleopatra pointing out the contradiction there was exactly my point :P. First you claim that there is no sure way to see if something is AI generated even after human scrutiny. Then you say that you are sure that some of the bans were false positive. Unless you are to imply that the bans were part of a test sample of human written post that the company itself crafted to look like AI generated content the company has by its own assumption no way to know if the false positive bans were indeed false positive bans
    – SPArcheon
    Jun 7, 2023 at 16:05
  • @SPArcheon I agree, just looking at theirs words from a slightly different angle :)
    – kleopatra
    Jun 7, 2023 at 17:17
11

I'm not a moderator, but I think SE has got this subject backwards.

I have on SU detected and signaled 5 cases of AI "answers", with confirmation in all cases by the moderators and even sometimes by the poster himself.

In the first two cases, I used no tools. For the next three cases, I discovered ZeroGPT and GPTZero and used them for a second opinion.

In other words, I have used the tools not for detecting AI answers, but for detecting whether I was wrong in assuming the answer was by AI.

No one runs all the answers through ZeroGPT and GPTZero to detect AI answers for fun. One only goes to this effort when the answer seems weird, without a central theme, with a very weak or no connection to the question and even sometimes just plain wrong and misleading.

The tools are used just the opposite of that postulated by SE : To check whether the answers are not AI.

6
  • «In other words, I have used the tools not for detecting AI answers, but for detecting whether I was wrong in assuming the answer was by AI.» But isn't that exactly what these tools are bad at? They can tell you something /might/ be fishy, but you need to confirm with your own judgement.
    – Nemo
    Jun 13, 2023 at 8:37
  • @Nemo: It's the opposite : I use the tools for finer control. For example, I once found out that one paragraph was added by the user to an AI answer, and this was confirmed by that user. The tools are usually correct, and in these 3 cases their judgement was the same as mine, but mine came first. I would have flagged them just the same without using the tools, but it seems irresponsible not to take every possible precaution before accusing the user.
    – harrymc
    Jun 13, 2023 at 8:48
  • 1
    In other words, AI answers will be flagged, with or without these tools, except that the tools can help avoid mistakes. Disallowing the tools doesn't mean that (bad) AI answers won't be flagged and removed.
    – harrymc
    Jun 13, 2023 at 8:51
  • 1
    Exactly why can't you continue to identify such answers by their content, downvote them and flag them in need of moderator intervention as wrong, misleading, or not answering the question? If an answer is bad, we need to get rid of it no matter how they were created. Why should a horrible answer survive just because a tool says it's not AI generated?
    – Philippos
    Jun 19, 2023 at 11:59
  • @Philippos: I can and I did. Reverse question : Why shouldn't I use a technology when it exists? I downvote and flag bad answers, but reporting a user as an AI spammer is more serious.
    – harrymc
    Jun 19, 2023 at 13:19
  • I'm not opposed to use that technology, so I'm not the one to answer your question. Just tried to understand your concern.
    – Philippos
    Jun 19, 2023 at 14:03
8

I thought it would be educational for the SE deciders to show them how GPT4 answers their own announcement here.

This "answer" is manifestly not useful or helpful, and doesn't even have a discernible central theme. It's a collection of texts that GPT4 found on the internet that loosely relate to the subjects in the announcement.

This is an example of rubbish AI "answers" that I have been seeing on the SU site. If this kind of drivel is allowed to pollute our sites, we'll only be destroying ourselves.

Does SE really think that we should let such answers deluge our sites?


The concerns expressed by the moderators regarding the detection tools and AI-generated content on Stack Overflow are valid and should not be dismissed. While it is unfortunate that some moderators have chosen to disengage from their moderation tasks, their actions highlight the seriousness of the issue at hand.

The claim that the ChatGPT detection tools have an alarmingly high rate of false positives is questionable. Detection tools, including AI algorithms, are not infallible, and false positives can occur. However, it is essential to consider the overall impact of these false positives and weigh them against the potential harm of allowing AI-generated content to proliferate unchecked. The purpose of such tools is to maintain the quality and integrity of the platform, even if there are occasional false positives that need to be addressed.

The suggestion that suspensions were imposed on users with little or no prior content contributions solely based on ChatGPT's detection tool is misleading. Moderators typically employ multiple methods and analyses to make informed decisions. While ChatGPT's detection tool may have been part of the process, it is not the sole determinant of suspensions. The goal is to ensure a safe and reliable platform for users, which sometimes requires precautionary measures.

The statement about seeking other, more reasonable tools and commitment to rapid testing of those tools is encouraging. It shows a willingness to address the concerns raised by the moderators and find better solutions for content moderation. Continuous improvement and adaptation to new challenges are essential for any platform to thrive.

Expressing appreciation for the moderators' years of service and acknowledging the difficulty of the situation is a positive gesture. However, it is important to focus on resolving the underlying issues and finding a viable path forward. Merely expressing regret and hoping for a resolution may not be sufficient. Concrete actions need to be taken to address the moderators' concerns and restore trust within the community.

While it is acknowledged that the upcoming days and weeks may be challenging, both sides should prioritize open dialogue and collaboration. Ensuring quality and inclusivity on the platform should be the shared goal. Transparent communication with the community about updates and progress is crucial for rebuilding trust and working towards a long-term solution.

In conclusion, it is crucial to acknowledge the concerns raised by the moderators and address them seriously. Balancing the need for effective content moderation with minimizing false positives is a complex task, but it should be approached with a commitment to finding the best possible solutions for all stakeholders involved.

4
  • 14
    "I thought it would be fun to ask GPT4 about its own opinion of the announcement here." It is not. You're not the first one to do that kind of stuff under this or related posts, and all such answers were receive extremely poorly, and later were deleted by authors. This joke/idea is out of date for nearly half a year.
    – markalex
    Jun 11, 2023 at 10:25
  • 6
    @markalex: It's not a joke, it's an example of AI "answers" that I have been seeing on the SU site. If this kind of drivel is allowed to pollute our sites, we'll only be destroying ourselves. But apparently this kind of stuff is what SE thinks we should let proliferate.
    – harrymc
    Jun 11, 2023 at 13:03
  • 2
    Then consider editing you answer, and adding as a first sentence something like "Look what incoherent sh*t chatGPT generated as an answer to this statement" and maybe more of your statement from last comment.
    – markalex
    Jun 11, 2023 at 14:17
  • 5
    @markalex: I tried to more clearly state what was my intention here. I hope I've succeeded this time.
    – harrymc
    Jun 11, 2023 at 14:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .