51

As a moderator on two sites, I am following the Stack Overflow community's policy regarding the ban of ChatGPT closely. However, does the community and its moderators have any handles on how to identify ChatGPT-generated answers? One recurring theme in answers (suspected to be) generated by the chat engine were devoid of references.

I'm pretty much only active on the science Stack Exchange sites, and there we basically enforce that posts (but especially answers) be supported by valid, credible, and preferably peer-reviewed sources to allow others to fact check and background read on the topic. I think referencing is a bridge too far for a chat bot, yet I don't know for sure.

One way or another, barring a possible loophole of OpenAI's ChatGPT in citing credible sources, how can we as a community identify posts based on a chatbot?

Improve this question
11
  • 9
    Sadly that's the hardest part, unless the author admits using ChatGPT, it's probably not really possible to prove it. However, repeated low quality or subtly wrong answers deserve a suspension all on its own, regardless how their author wrote them.
    – Shadow Wizard Love Zelda
    Dec 6, 2022 at 12:45
  • 4
    I would say that discussing privately with Stack Overflow mods would be helpful. They have pretty god ideas on how to detect them. Once you get the grip you can very easily detect AI generated content. I am not posting tips here as they might help in avoiding detection.
    – Resistance Is Futile
    Dec 6, 2022 at 12:53
  • 27
    NOTE: "Temporary policy: ChatGPT is banned" on Stack Overflow is imposed by SO moderators on SO only. It was imposed, effectively as an emergency measure, because SO can't function with the flood of ChatGPT posts. It was not imposed by Stack Overflow, the company. As far as I'm aware, the company has not taken a position on the use of ChatGPT. Stack Overflow moderators can't, and are not trying to, impose such a policy on other sites in the SE Network. Without a company statement, each site will need to decide for itself if ChatGPT is acceptable.
    – Makyen
    Dec 6, 2022 at 13:18
  • 4
    @Makyen you might have missed this comment. (re "the company has not taken a position on the use of ChatGPT".)
    – Shadow Wizard Love Zelda
    Dec 6, 2022 at 13:46
  • 2
    @ShadowTheKidWizard I appreciate you pointing me to that comment (I had seen it, but it was good to re-read it), but I don't see how that comment would be seen as the company taking a position on the acceptability of using ChatGPT specifically, or AI generally. Yes, they are considering what to do, but haven't actually done anything, or made any policy statement, as of this time. So, I don't really see that comment as changing or affecting anything I said above.
    – Makyen
    Dec 6, 2022 at 14:14
  • 6
    I do understand that such things take time. Hopefully, the company will, not too long from now, have a policy and do something, but we have to deal with things as they currently are, even while we hope for change and solutions. The big picture situation is quite complex, may take a lot of work, and is something that will probably need to change and adapt over time as technology progresses. So, it is reasonable for the company to take a while to figure out what to do and what policies to have on a general basis. But, we all also need to deal with the situation as it is right now.
    – Makyen
    Dec 6, 2022 at 14:14
  • 1
    Same question posted a few hours before this question on SO Meta: meta.stackoverflow.com/q/421880/395857
    – Franck Dernoncourt
    Dec 14, 2022 at 8:39
  • I don´t think you´ll get an answer here, since such internal information is always hiden from the public.
    – convert
    Jan 3, 2023 at 22:15
  • 2
    I've protected the question - We've had 3 answers generated by chat gpt so far from 'new' users , and if the newbies keep wanting to stick the nice silverware in the sockets, maybe its not a good idea to let them access either.
    – Journeyman Geek
    Jan 9, 2023 at 5:56
  • I don't have rep to answer the now-protected question, but I think we should reframe the question from "Identifying an answer as coming from ChatGPT," which is hard (and that's part of the problem that's leading to ChatGPT being banned), and "Identifying accounts who act as sockpuppets for ChatGPT," which is a much easier problem to solve.
    – Tim C
    Jan 20, 2023 at 23:19
  • 1
    Another problem is that ChatGPT was multiple times proved to produce made-up nonexistent references, so if you don't have means of verifying them, you shouldn't trust it.
    – Kusavil
    May 2, 2023 at 23:27

5 Answers 5

Reset to default
39

We (mods and some users working specifically on handling these posts) have not been publicly sharing information as to how we detect such posts, because doing so would give people using ChatGPT information that would be directly helpful to avoid detection. There is a significant amount of information available about detection of these posts in the Moderator Team and other locations mentioned in there. You can also get information through the Teachers' Lounge chat room.

Improve this answer
7
  • 1
    Thanks, this helps a lot. But giving the community at large handles to work with may increase our eyes and ears to help flag suspicious content.
    – AliceD
    Dec 6, 2022 at 15:27
  • 9
    There are significant trade-offs in whatever position is taken. Others looking for such posts does help. IME, the main thing is people being aware this happens, so when something feels funky, they know of this as a possible issue, rather than wondering "what the bleeep is this?" SO is dealing with literally thousands of posts created using ChatGPT, so we're definitely trying to make it easier to identify and deal with them. Doing that takes time. While our concentration tends to be on SO, it's known that all of SE needs a solution and there are mods from other sites participating.
    – Makyen
    Dec 6, 2022 at 15:41
  • 5
    For non-moderators, is mod-flagging suspected posts helpful or does it just add more noise? If it isn't helpful, is there anything else that community members can do to support moderators? (Bake cookies?)
    – Andrew Myers
    Dec 6, 2022 at 18:19
  • 6
    @AndrewMyers On SO, and I assume other sites, an "in need of moderator intervention" flag which explains that you believe the post to have been created with ChatGPT would be welcome and highly likely to be marked "helpful". On SO, you only need to raise a single flag on one of the user's posts, not flag each of their posts. On SO, you don't really need to collect evidence (i.e. don't expend your time), because mods will investigate and the mods handling ChatGPT issues are quite familiar with such posts. On other sites, more context and evidence in such flags is probably helpful.
    – Makyen
    Dec 6, 2022 at 18:37
  • 3
    As for sites other than Stack Overflow, I'm not a mod on any other site, so I can't speak authoritatively about how they handle things, or even their policies on ChatGPT content. On the other hand, I'd, personally, assume that the handling would be roughly similar, but that a bit more context might be needed in the flag, because other sites, fortunately, aren't, currently, seeing a flood of thousands of these posts, so how familiar with ChatGPT the particular mod who's handling your flag on another site will be is uncertain.
    – Makyen
    Dec 6, 2022 at 18:41
  • 4
    @Makyen As a mod on a few non-SO sites, I agree with Makyen's comments above and I'd guess that mods anywhere would welcome such flags. (The only site where I wouldn't have been sure is SO, with its massive volume of posts and flags, but given the above comments from Makyen, they're happy to get such flags too.) Mods can't be everywhere, and flags help to bring our attention to things we might not have seen otherwise. Even if you're mistaken about a post being ChatGPT generated, I believe most mods would mark a flag helpful if it's raised in good faith.
    – Rand al'Thor
    Dec 6, 2022 at 18:50
  • 13
    A major reason behind SO temporarily banning ChatGPT content entirely is so we can deal with it as a class of content, rather than have to individually evaluate the accuracy/inaccuracy/quality level of each individual post. To evaluate each individual post in that way takes orders of magnitude more time and effort, and often requires significant levels of expertise in the specific technology the question is asking about. There's just no way we can do that. OTOH, identifying that the user has posted ChatGPT content is, usually, quite fast and easy, once looking at the user/content.
    – Makyen
    Dec 6, 2022 at 19:05
15

Detect ChatGPT:

The creators of ChatGPT have written an improved tool to detect AI generated text, the AI Classifier.

Source: https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text/

"Our classifier is not fully reliable. In our evaluations on a “challenge set” of English texts, our classifier correctly identifies 26% of AI-written text (true positives) as “likely AI-written,” while incorrectly labeling human-written text as AI-written 9% of the time (false positives). Our classifier’s reliability typically improves as the length of the input text increases. Compared to our previously released classifier, this new classifier is significantly more reliable on text from more recent AI systems.".

ChatGPT isn't the only worry.

Improve this answer
0
8

Examples of programs for automatic detection of machine-generated text:

I'll add below some research pointers.

Most recent work:

For a survey of automatic detection of machine-generated text, see Section 4 of {1} and Section 4 of {2}.

Examples of hints that an answer came from ChatGPT:

  1. Spot typical ChatGPT output structures (example).
  2. No quotes.
  3. No images.
  4. No typos.
  5. No grammar mistakes.
  6. No links.
Improve this answer
10
  • 1
    Or just use tooling that can tell you, very accurately, whether or not something is AI-generated. GPT never outputs the same result twice. It can even be completely different depending on small input changes. You'll get different results when you do step 1 and 2 in different conversations vs the same conversation...
    – Cerbrus
    Jan 4, 2023 at 8:13
  • @Cerbrus yes, see the references for the tools
    – Franck Dernoncourt
    Jan 4, 2023 at 12:13
  • Or just google "Language model detectors"
    – Cerbrus
    Jan 4, 2023 at 12:16
  • @Cerbrus The two references give a nice survey, but are more research-oriented. I agree that is the type of questions that AI is pretty good at answering.
    – Franck Dernoncourt
    Jan 4, 2023 at 12:18
  • My point is that you have a convoluted list of steps, followed by two articles you'd have to dig through with a bucketload of theory (And no actual implementation on a glance), whilst detection is in fact trivially simple: Just google a detector, dump the text in there and bob's your uncle.
    – Cerbrus
    Jan 4, 2023 at 12:25
  • @Cerbrus point taken, I was trying to explain how the detection may work.
    – Franck Dernoncourt
    Jan 4, 2023 at 12:32
  • Your "most recent" is not yet available they say "Coming soon". It's useless for now.
    – Shadow Wizard Love Zelda
    Jan 30, 2023 at 8:37
  • @ShadowWizardChasingStars fixed.
    – Franck Dernoncourt
    Jan 30, 2023 at 8:41
  • Not really lol, you just made it much worse. Now you are linking to the research, still useless for common folk who are not AI experts with years of experience. As far as I can tell, the questions asks for existing and working tools to detect GPT. That is all. Not research or future tools.
    – Shadow Wizard Love Zelda
    Jan 30, 2023 at 8:43
  • 2
    @ShadowWizardChasingStars The question is "How can we identify ChatGPT-generated posts?" No idea why you exclude research. People have implemented AI bots for SE e.g. meta.stackexchange.com/q/291301/178179
    – Franck Dernoncourt
    Jan 30, 2023 at 8:47
2

In the beginning of 2023, a student created GPTZero as a personal project:

An app that can quickly and efficiently detect whether an essay is ChatGPT or human written

Looks like it can still be improved, but it should detect at least the obvious cases for now. It's meant for essays, but I believe this can also be used for answers across SE. (Didn't try it myself yet.)

Improve this answer
-23

Why do moderators need to discriminate against AI chatbots?

The upvote/downvote feature already works well to ensure answer quality.

The only relevant metric that we should be using to answer questions is, "does this answer the question correctly".

Whether or not it was written by a chatbot or a "human" is not relevant.

Regarding the concerns about the "accuracy/inaccuracy/quality" of ChatGPT (to quote @Mayken's criteria in their comment): Stack Exchange and Stack Overflow is open to the public. This means that anyone can answer questions and people frequently answer questions wrong. Unless we have an objective metric to show that ChatGPT is wrong more often than the median/average Stack Overflow contributor then we should not be discriminating against AI.

If we are worried about the volume of information, then anyone on SE or SO (not just moderators) can downvote it. This has the added benefit that the entire community can downvote bad answers so it scales better than just relying on moderators.

Most importantly, upvoting and downvoting will act like reinforcement learning to train ChatGPT to be even better.

Improve this answer
19
  • 14
    Yes, chatGPT is wrong more often than the median because it's almost always wrong It can't correct it's mistakes either, a human SME can if a typo in an answer pointed out in a comment.
    – Robert Longson
    Dec 10, 2022 at 15:44
  • 8
    "Most importantly, upvoting and downvoting will act like reinforcement learning to train ChatGPT to be even better." I don't see how. Once an answer is copy-pasted from ChatGPT into a Stack Overflow answer, it's no longer connected to the neural network and can't pass feedback back to it. AFAIK, for your last sentence to be true, SE would have to integrate ChatGPT into the site itself, and they have thus far refused to do that.
    – F1Krazy
    Dec 10, 2022 at 15:53
  • @RobertLongson the link you provided has no objective metric. The comments say. "Very often. It cannot know the correct answer." and the highest voted answer literally, says "I don't think it really matters how often it's correct or not". That answer also tells us that automation is allowed if it's better than humans. meta.stackexchange.com/questions/291301/…
    – Tomiwa
    Dec 10, 2022 at 16:29
  • @F1Krazy ChatGPT scrapes information from the internet including SE. It can pass the feedback into the Neural Network in the future. Think of it like a search engine crawler, it might not pass feedback in realtime but it does so periodically. Btw, even if SE refuses to "integrate" ChatGPT, OpenAI can still just scrape the site and use it to train their model. This isn't a statement of legality/ethics/"scraping is banned" but a statement of even if you don't allow it to give you answers. It will use your answers to make itself better.
    – Tomiwa
    Dec 10, 2022 at 16:35
  • 12
    The main problem was the answers were consistently wrong yet sometimes required a subject matter expert to figure that out. The manpower to check this was not available. Until the quality and accuracy of the answers rapidly improves, they are not acceptable on Q&A site. It is possible to use this tool for good, but people were recklessly pasting answers all over the place without care for validity. It would've ended the site if unchecked.
    – Mast
    Dec 10, 2022 at 20:38
  • 3
    I've seen hundreds of the answers and helped hunt down some of them. Others did way more than I possibly could've in this effort. You have no idea how big this problem was and we're unwilling to share the details to avoid people working around the limits. If you post a quality answer, it won't be removed. Simple. Violate the rules and risk removal of the post(s) with suspension if necessary.
    – Mast
    Dec 10, 2022 at 20:40
  • 5
    As said in the 'embedded.fm' podcast episode 437 at 01 h 02 min 20 secs: "These things are trained on a corpus of data from the Internet, which is basically a giant group of people talking about things they are not qualified to discuss.". And at 01 h 02 min 00 secs: "A finely-tuned BS machine". And at 01 h 00 min 52 secs: "The code it gave me was total garbage.". And at 00 h 49 min 40 secs: "...totally wrong about 90% of the things it talks about.". Direct download URL.
    – This_is_NOT_a_forum
    Dec 11, 2022 at 15:33
  • 10
    This discussion has some jarring similarities to other debates about disinformation and fake news online. The same basic asymmetry holds, in spades; posting low-quality and/or false content is vastly cheaper (and with AI, increasingly so, while removing many of the features of low-effort content produced by humans) than the effort to pinpoint the incorrect information and recover from any damage resulting from it. Merely pointing out that something is obviously wrong is rarely sufficient, and easily dismissed by those most in need of qualified advice.
    – tripleee
    Dec 11, 2022 at 15:56
  • 1
    @This_is_NOT_a_forum while you might have received "total garbage" I have had total amazement. I've been documenting my experience with ChatGPT. The results are amazing. Is it perfect? No. Anyone who sees the examples I've linked and is being intellectually honest with themselves will agree that while it's not perfect, it's good enough to be useful.
    – Tomiwa
    Dec 11, 2022 at 22:08
  • 2
    @Tomiwa: I haven't. I don't have any first-hand experience with ChatGPT. I was quoting (people who I very much trust (e.g., a podcast about 10 years old, by two (real) professionals)). I am not discounting ChatGPT. It can't be all bad. There must be some useful applications for it on Stack Overflow, for example, to tease out the most basic and required information from question askers (some OPs think other people are mind-readers). But several metric tonnes of salt might be useful. I am looking forward to your blog post.
    – This_is_NOT_a_forum
    Dec 12, 2022 at 15:53
  • 1
    E.g., the upcoming Staging Ground might employ a chat bot. There could be an option, "For the initial stages, do you prefer a chat bot or a real human? The chat bot interaction will only be visible to you.", as preferences differ (e.g., losing face vs. "cold").
    – This_is_NOT_a_forum
    Dec 12, 2022 at 16:03
  • 1
    @JonathanReez sure, and in the 60's of the 20th century, people were sure that by 2000 we'll be already living on Mars on vast colonies. Predictions are worthless in those things.
    – Shadow Wizard Love Zelda
    Dec 13, 2022 at 14:45
  • 2
    @This_is_NOT_a_forum please don't burden users with silly choices like that
    – user253751
    Dec 14, 2022 at 12:52
  • 1
    Because the AI output is garbage, @This_is_NOT_a_forum. New users aren't helped with incorrect answers that are written (reasonably) well.
    – Cerbrus
    Jan 4, 2023 at 8:14
  • 2
    ChatGPT is not "clever" enough to ask constructive questions like that. In fact, I've never even seen it ask the user something. It's reactive, not assertive. So no, that ain't gonna work.
    – Cerbrus
    Jan 9, 2023 at 8:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .