This is a follow-up of Ban ChatGPT network-wide. How often does ChatGPT give an incorrect answer?
As a language model, ChatGPT is not designed to provide answers to specific questions, especially those related to a specific topic or subject. Instead, it uses a large corpus of text to generate responses based on the input it receives. This means that the responses it generates may not always be accurate or relevant to the specific question being asked. Additionally, ChatGPT does not have access to external information, such as the internet, so it cannot provide answers to questions that require knowledge beyond what it has been trained on. In short, ChatGPT is not intended to be used as a source of information, and it is not able to provide accurate answers to all questions.
Emphasis mine - you may guess what the source of that text is.
I don't think it really matters how often it's correct or not; the main question right now is if the community can handle the times it's not. Automation is allowed on Stack Exchange if it performs significantly better than humans, but I trust the Stack Overflow community if they say it's beyond control. The score on this question speaks volumes.
From memory when looking through the chatGPT answers here when they existed. I think that was around 10-15 answers before they were deleted.
- Somewhere between a third and a half had comments indicating that they were problematical. I.e. the OP or a SME had commented on them raising issues. It's possible many more were wrong, but nobody had looked at them closely yet.
- One or two were accepted. That doesn't mean they were correct, merely that they were helpful to the question asker.
- One was downvoted. Of course downvoting costs rep and you need 125 rep to be able to do it at all so most question askers can't downvote.
The trouble is ChatGPT is prepared to make up an answer on any subject under the sun and so we'd likely need as many SMEs as answers to check them all properly. Neither "I don't know", or "I'm not sure" seem to be in its range of possible answers.
Not a systematic study: but of 20 I read carefully (purported answers posted to Cross Validated, or responses to actual C.V. questions input to ChatGPT by me)†; I judged one to be useful, though not as useful as the relevant section of the obvious Wikipedia article, & about half of the rest to be not merely useless but positively misleading.
Extrapolating to other sites would be unwise. For example, I found ChatGPT to be terribly bad at answering questions with much mathematical content, so it wouldn't be surprising if useful answers from it to questions on Mathematics, Physics, &c. were vanishingly rare.
† I used the free version of ChatGPT available in Nov./Dec. 2022 (v.3?) & did not attempt any 'prompt engineering'; I'm pretty sure the users who posted the purported answers were following a similar process.
Some findings from the paper Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions. Samia Kabir, David N. Udo-Imeh, Bonan Kou, Tianyi Zhang. arXiv:2308.02312:
I copy/pasted 10 random questions from the Chinese Stack Exchange into https://poe.com/Sage which is described as:
Good in languages other than English, and also in programming-related tasks. Powered by gpt-3.5-turbo.
I give its answers a percentage value for how good I think the answer is (100% = about as good as an answer can get).
|Question||GPT 3.5 turbo, answer quality (%)|
|Difference between 必须得 and 一定要？||100%, significantly better than the current answers|
|的 in the “你们是坐飞机来北京的吗？”||20%, it got totally confused|
|what is the meaning of 都不甘于平庸 in the song 致你 by 苡慧?||70%, it struggled with song lyrics (it's hard for me to judge this one---I don't know myself)|
|Function of 了 Completion of Action or Change of State||90%, there were small imperfections if I'm being nit-picky|
|Can 人口 be interpreted as a number? [my question]||100%, clearly understood the question and my difficulties|
|What's the function/meaning of 就不要 in 过去的事情就不要说了?||100%, clearly understood the question, and concisely answered correctly, better than two (of three) human answers|
|How to say "a quarter past nine" in Chinese?||100%, easy question|
|What is the meaning of 教训 here？||100%, with an explanation significantly better than the current answers|
|Is it possible for 已 to make a sentence into the future tense?||80%, the explanation is a bit weird in places|
|What is the colloquial meaning of 小祖宗 ？||100%, it's a good answer, but so is the top human-submitted answer|
The answers it's writing are friendly, concise, almost always immaculately grammatical, and don't omit answering half the question (e.g., when someone asks a general question and gives examples). Two of the answers it wrote are substantially better than the current human-written answers. (What do I do with them?!) The first question contained a typo (which I left in---the question says 你必须的有咖啡? , but it should be 你必须得有咖啡? ), and the AI wasn't confused.
So in answer to the question:
How often does ChatGPT give an incorrect answer?
I'm not sure how GPT 3.5 Turbo and ChatGPT are different, but from the above experiment, GPT 3.5 Turbo was incorrect only once, so 10% of the time.
Update: I've been copy/pasting a lot of Chinese.SE questions into Sage (see: Examples of an AI (Sage) answering questions at Chinese.SE [reference post]). What I'm now understanding is that there's selection bias in my process...
Of the AI-generated answers I'm able to judge as (un)reasonable, almost all are reasonable. Much like the above, some have imperfections, some are great answers, and every now and then its answers are total rubbish. However, for more advanced Chinese questions, perhaps AI-generated answers are more likely to be nonsensical, and I'm unable to evaluate them.
The AI is really good at questions like "Is there a Chinese idiom meaning [something]?" and "What's the difference between [x] and [y]?" It's mediocre at questions like "How did the character [z] evolve?" I'm guessing it's poor at humor in Chinese, since it's poor at humor in English.
TL;DR: ChatGPT is capable of generating reasonable, and even excellent, answers to a subset of Chinese.SE questions (albeit with occasional nonsense). It's possible outside of this subset, ChatGPT produces nonsense more frequently.