New flag for ChatGPT answers

Question

Answers generated using ChatGPT are banned on many Stack Exchange sites regardless of being credited or not. This is mainly because they aren't particularly accurate or useful, and are often misleading.

Personally, I've spotted a number of these; none of which were any good, and all of which have been deleted.

As I don't have the Deletion privilege, when I see a ChatGPT answer all I can do is to flag it (and downvote). But, none of the standard answer flags are a great match for this:

• Spam
• Rude or abusive
• Not an answer
• Very low quality

So, so far I've been using the custom flag:

• In need of moderator intervention

... with what is basically the same custom message, something along the lines of:

This looks like an absolute rubbish ChatGPT "answer".

But, custom flags can't be handled automatically, and so take up moderators' time which seems unnecessary.

---

I request that a new flag be created for ChatGPT based answers, something along the lines:

• ChatGPT
  This answer appears to be generated by ChatGPT (or similar AI) and provides inaccurate/misleading information.

Answer 1

My point is, that if 3 (or whatever number of) standard ChatGPT flags could get a bad answer removed (without needing detailed custom flag messages, or a moderator's attention), it'd be a less taxing system.

This seems to be the crux of your suggestion and unfortunately I think this is impossible (or a bad idea). Please continue to use "In need of moderator intervention" and give as much detail as possible as to why you think it's AI.

As to why the flag you describe won't work, the current process on most sites is as follows:

1. An "In need of moderator intervention" flag comes in describing why a user thinks an answer is AI-generated. (Flags can be cast by any user with 15 rep.)
2. A moderator investigates that answer and all of the user's other recent answers. This is where most of the work is, since so many users think they can pass off AI text as their own. Most moderators have chosen to keep the details of this process a secret so that troublemakers can't work around it.
3. If the suspicions were correct, the answer is deleted.
4. The moderator sends a moderator message. This will likely come with a suspension if this isn't the first infraction.

First of all, your suggestion prevents the user from giving any details about why the answer is AI generated (as in #1). These details are very important because they can considerably shorten the amount of time a moderator spends on step #2.

Step #2 is important. AI can be tricky to identify and it's easy to get things wrong if you only investigate superficially.

As for step #3, there are existing flags that delete answers directly, the "Spam" and "Rude or abusive" flags (known as red flags). These flags, however, are used on posts that are as obvious as slap in the face — there is no investigation to see if there could be a legitimate answer to a coding problem in a post that recommends you buy pills and supplements. Without moderator intervention, a post needs 6 red flags to be deleted but it's necessary for a group of users to coordinate in order to make sure spam and abusive posts are consistently deleted ASAP. The high number of flags needed makes it less likely that these flags will be used incorrectly. In my experience as a moderator on 3 sites, it's extremely rare to see someone other than myself flag an answer as AI. (Yes, I flag for my own attention. The current record on my sites is 4 other people flagging one answer as AI.)

As for #4, there is no flag that will formally warn the user or suspend them, though that is what's required in most cases. Deletion with no other consequences is too lenient, while red flag deletion is too harsh.

---

A note: The plagiarism flag on Stack Overflow is very similar in how it works to "In need of moderator intervention" except 1) it forces the user to give specific details and 2) it unlocks a special type of deletion (dissociation) that only moderators can use. It also creates a new category of flag that moderators can filter on, which is only a benefit on sites that have a lot of "In need of moderator intervention" flags (which is likely not very many outside of SO). I'm not sure that this gives enough benefits for an AI version to be worth a spot on the flag dialog; I'd rather keep that simple.

Answer 2

In need of moderator intervention is the best option here.

I've usually described the output of chatgpt as 'barely plausible garbage'. It's believable but often incorrect. In some cases unless you try the answer, are a subject matter expert, or have seen so many of these posts that some part of your brain immediately recognises it, it looks possible.

And while a flag type that automatically deletes it seems 'useful' - it also means that mods don't actually notice the deletion, and might miss a broader pattern of such posts. On some sites, there's also suspensions involved.

Taking care of the 'immediate' problem by automatic deletion also might mean that only way that this gets handled is a potential answer ban.

I'm also hopeful that once the initial excitement wears off, and as the technology matures that there'll be less of these posts, simply cause it's no longer 'interesting' to ask the bot a question and just paste it here.

With some exceptions - we're generally able to deal with it. I'd rather be writing meta posts and such of course, but I don't really see a major benefit over our current options

Answer 3

My points against the suggested implementation stand, but in the wake of the strike (in mid 2023), when AI flags piled up on many sites only to require a lot more effort to clear under the new policy (heuristics), my position towards a near-clone of the plagiarism flag has changed.

Between plagiarism and AI flags, the only significant change would be removing the URL field. Prompting for details and the ability to dissociate are helpful for both flag types. The description of the flag may also need some variables; a link to the AI policy would be very useful.

If this is created as an option, sites that get a lot of flags can enable it as needed (same as plagiarism flags?) and other sites can keep things simple.

I'll still say that we can manage without a new flag type, but it could be helpful.