Assessing the accuracy and consistency of answers by ChatGPT to questions regarding carbon monoxide poisoning

Jun Qiu, Youlian Zhou · 2024

Read Assessing the accuracy and consistency of answers by ChatGPT to questions regarding carbon monoxide poisoning by Jun Qiu, Youlian Zhou (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11578497.txt.

Description

Jun Qiu and Youlian Zhou grade ChatGPT's answers on carbon monoxide poisoning, in a 2024 study. Patients and students may ask chatbots medical questions, so the answers must be right and steady. The team drew 44 single-choice exam questions on the topic from a study app.

Each question went to the chatbot ten times in Chinese and ten times in English. About half the questions drew highly steady answers across repeats. About a third drew answers so varied they counted as unacceptably blurred. Accuracy was measured against the app's key as well. The mixed report warns that a fluent chatbot can still wobble on core medical facts, in either language.

Catalog record

CREATORJun Qiu, Youlian Zhou
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC11578497

Browse the full library · About the Tower of Babel