The Triage and Diagnostic Accuracy of Frontier Large Language Models: Updated Comparison to Physician Performance

Michael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi · 2024

Read The Triage and Diagnostic Accuracy of Frontier Large Language Models: Updated Comparison to Physician Performance by Michael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11662182.txt.

Description

Michael Joseph Sorich and co-authors compare new large language models with doctors in a 2024 study. They ask how well the models sort patients by urgency and name the right diagnosis. An earlier test with GPT-3 found it matched lay people on triage and trailed doctors on diagnosis.

The team gave 48 case stories to three top models: GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Each model had to reason step by step, then pick its top three diagnoses in order. The cases covered both common and severe illness. The study updates the older benchmark so readers can see whether newer models have closed the gap with doctors.

Catalog record

CREATORMichael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE7 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC11662182

Browse the full library · About the Tower of Babel