This is a follow-up of Ban ChatGPT network-wide. How often does ChatGPT give an incorrect answer?
asked Dec 5, 2022 at 16:20
-
7ChatGPT loves the phrase "I hope this helps! Let me know if you have any other questions", searching for it on SO may be insightful. There are 9 results before December 2022, and 34 in December 2022, when ChatGPT became publicly available.– smitopDec 5, 2022 at 16:22
-
19Very often. It cannot know the correct answer. It just generates text that conforms to what an answer might look like. There was one user who posted two answers contradicting each other. And they were both still wrong. I've seen answers that contradict themselves even in the same paragraph.– VLAZDec 5, 2022 at 16:28
-
3ChatGPT is a nice game, a gimmick. Nothing more.– Shadow The Spring WizardDec 5, 2022 at 16:32
-
4Not SE questions, but this thread passed by my Twitter timeline and is a good illustration of "wrong answers": twitter.com/studentactivism/status/…– Tinkeringbell ModDec 5, 2022 at 16:54
-
@Tinkeringbell also see here twitter.com/vogon/status/1598334517647134720 and here is a paper on common problems with this approach to text generation: dl.acm.org/doi/10.1145/3442188.3445922– VLAZDec 5, 2022 at 17:26
-
6@Tink My favourite quote from that Twitter thread is "Because if ChatGPT is, as it seems to be, a consummate bullshitter, it's also—definitionally—a bullshitter who doesn't know when its bullshitting. And we all know that that's the most dangerous kind."– PM 2RingDec 5, 2022 at 17:27
-
This is unanswerable as written. Actually having an answer to this would require information which isn't available to anyone, because the complete answer would include every attempt that's ever been made of putting a question into ChatGPT. You'd need the logs for all input to ChatGPT (matched to SE questions) and all of its output. You'd then need to have humans evaluate every single ChatGPT response for accuracy. You could do a sampling using randomly selected questions, with each answer human evaluated for accuracy, but there's likely to be substantial variation across SE sites.– MakyenDec 10, 2022 at 19:25
-
Even the question you probably intended to ask: "Of all ChatGPT answers posted to SE sites, what percentage are incorrect/correct?" is unanswerable, assuming some semi-reasonable limit on the amount of human effort (note that posted answers are already a filtered subset of those created; presumably at least slightly filtered for accuracy by those posting). To estimate this, you'd have to look at a substantial number of randomly selected ChatGPT answers (including selecting from all deleted ones on all sites) and individually evaluate each for accuracy by humans. That's a huge amount of work.– MakyenDec 10, 2022 at 19:26
-
1@Makyen human evaluation on some prediction subset is something commonly done in my field (which turns out to include ChatGPT). 100 samples, 3 humans, 1h, done. The downvoters have no clue about NLP/AI research.– Franck DernoncourtDec 10, 2022 at 19:38
-
1I find it unlikely that you'd find 3 humans who, together, are experts on every subject covered on all SE sites. Yes, I know that you don't really mean "use 3 people", but I think your example underestimates the scope. If you don't have experts on all subjects being evaluated, then you don't really know if the answer is correct. These answers do fool people, particularly when the person isn't an expert and/or when not read critically. Eliminating possible Q&A because you don't have an expert for it introduces bias, which doesn't mean it wouldn't be data, just not what you asked for.– MakyenDec 10, 2022 at 19:46
-
1@Makyen subsample domains, analyze variance of the prediction quality across domains, etc.– Franck DernoncourtDec 11, 2022 at 4:19
-
Is that why this was closed, because "this is unanswerable as written"? A question should not be judged by its answers, though, and "this is unanswerable" is a perfectly fine answer to a perfectly fine question (I am not sure this Q is of the latter kind).– JoachimDec 12, 2022 at 15:38
Add a comment
|
3 Answers
As a language model, ChatGPT is not designed to provide answers to specific questions, especially those related to a specific topic or subject. Instead, it uses a large corpus of text to generate responses based on the input it receives. This means that the responses it generates may not always be accurate or relevant to the specific question being asked. Additionally, ChatGPT does not have access to external information, such as the internet, so it cannot provide answers to questions that require knowledge beyond what it has been trained on. In short, ChatGPT is not intended to be used as a source of information, and it is not able to provide accurate answers to all questions.
Emphasis mine - you may guess what the source of that text is.
I don't think it really matters how often it's correct or not; the main question right now is if the community can handle the times it's not. Automation is allowed on Stack Exchange if it performs significantly better than humans, but I trust the Stack Overflow community if they say it's beyond control. The score on this question speaks volumes.
answered Dec 5, 2022 at 16:29
-
Considering that Bing now gives access to the wider internet, it seems that this has to be asked again.– BraiamMar 16 at 15:41
From memory when looking through the chatGPT answers here when they existed. I think that was around 10-15 answers before they were deleted.
- Somewhere between a third and a half had comments indicating that they were problematical. I.e. the OP or a SME had commented on them raising issues. It's possible many more were wrong, but nobody had looked at them closely yet.
- One or two were accepted. That doesn't mean they were correct, merely that they were helpful to the question asker.
- One was downvoted. Of course downvoting costs rep and you need 125 rep to be able to do it at all so most question askers can't downvote.
The trouble is ChatGPT is prepared to make up an answer on any subject under the sun and so we'd likely need as many SMEs as answers to check them all properly. Neither "I don't know", or "I'm not sure" seem to be in its range of possible answers.
answered Dec 5, 2022 at 17:05
-
15Unfortunately, accepting doesn't even mean it was actually helpful to the question OP. It only means the question OP clicked the button. I recall one of these ChatGPT answers which I deleted, where the question OP had awarded a bounty to it, accepted it, and then left a comment to the effect of "this doesn't work". What I assume happened is that the question OP accepted and awarded the bounty because it looked good, but hadn't actually tried what the answer was saying was the solution, which is a good example of the primary problem with these ChatGPT generated answers: bad answers look good.– MakyenDec 5, 2022 at 19:05
-
@Makyen so, basically if we get a human that makes "looking good" but factually incorrect answers, then that's ok? I guess we are so tuned to look for signs of "bad quality" that we forgot the main one: being incorrect.– BraiamMar 16 at 15:43
-
No, @Braiam, incorrect answers by humans aren't desired. However, human generated poor answers tend to get downvoted. The difference is A) volume of posts and ease of creation, and B) human generated poor answers tend to be more easily identified as poor quality, because it takes most humans quite a bit of work to compose an answer that has good grammar, sounds confident, and presents in a manner that "looks good". AI generated content takes seconds, at most, and tends to be "eloquent bullshit", which looks and sounds good, even when hilariously wrong and self-contradictory.– MakyenMar 16 at 16:37
-
As to (A), the ease of creation and speed of AI vs human generation of content means that there are orders of magnitude more posts generated by the AI than a human doing the same. There are orders of magnitude more people posting them, often because those people think they are helping, when they are actually hurting by posting content they have no clue as to if it's correct, or not, which causes the person asking even more confusion.– MakyenMar 16 at 16:37
-
As to (B), the fact that the AI generated text is "eloquent bullshit" results in confusing a lot of people into thinking it's correct when it's not. It also tends to mean that evaluating it for correctness tends to require a higher level of subject matter expertise and, again, orders of magnitude more more time.– MakyenMar 16 at 16:37
-
Overall, these combine to mean that it requires orders of magnitude more curation resources to evaluate the aggregate volume of AI generated content on an "is this correct" basis. The sites, and SO in particular, are substantially below the level of curation resources they need to actually curate the already existing volume of human generated content. Adding the orders of magnitude additional curators needed to properly evaluate AI generated content, particularly individuals with a high level of subject matter expertise, over all subject areas, is something that's just not possible.– MakyenMar 16 at 16:38
-
Until an AI has inherent checks for correctness, it's just non-viable to allow the content. If it was only posted by people capable of checking for the correctness of that post, and who actually do so, it would be a different issue, but 99.9% (quite close to an accurate percentage) of the posted content isn't that. Unfortunately, even strong advocates for AI generation will routinely not care about manually checking for accuracy and just lie about doing so. We just aren't able to weed out that 99.9% in order to allow the reasonable 0.1%.– MakyenMar 16 at 16:38
-
Does AI get things right? Sometimes. Can it be helpful to some people? Sometimes. Can it be close enough or just feedback from something else allowing the human using it to break out of their thought patterns and get to an answer? Yes, sometimes. Is it often sufficiently wrong to cause the person using it to be even more confused or head in an even worse direction? Definitely. Currently, AI generation capability has absolutely no ability to even attempt to judge correctness. Doing so is just not part of the design. Until it is, AI generation will continue to be just "eloquent bullshit".– MakyenMar 16 at 16:38
-
@Braiam You're literally months late joining the discussion on AI generated content. While your position and opinion are valued, you're rehashing arguments that have been made many times over the last few months without appearing to bring anything new. If you're interested in the topic, please take the time to go through the existing discussions and move forward from them, rather than rehashing things that have already been covered in detail multiple times.– MakyenMar 16 at 16:38
-
@Makyen "you're rehashing arguments that have been made many times over the last few months without appearing to bring anything new", except that ChatGPT is new. As said elsewhere is a tool in constant development, now with access to the wider internet (Bing) and that funny incorrect answers are being corrected (ask what's the fastest marine bird, it's finally not the peregrine falcon). It's not that I bring something new, it's that we need to reassess our previous understanding about the tool. Like an answer on SE, it's not guaranteed to be correct from now to the heat death of the universe.– BraiamMar 16 at 19:24
-
If we are going to be so obtuse to not being able to reassess our previous point when the context changes, then why the heck we discuss things at all? Opinions need to be able to change @Makyen, if they do not, then I prefer not having them at all.– BraiamMar 16 at 19:25
Not a systematic study: but of 20 I read carefully (purported answers posted to Cross Validated, or responses to actual C.V. questions input to ChatGPT by me); I judged one to be useful, though not as useful as the relevant section of the obvious Wikipedia article, & about half of the rest to be not merely useless but positively misleading.
Extrapolating to other sites would be unwise. For example, I found ChatGPT to be terribly bad at answering questions with much mathematical content, so it wouldn't be surprising if useful answers from it to questions on Mathematics, Physics, &c. were vanishingly rare.
answered Dec 13, 2022 at 21:17
