The Triage and Diagnostic Accuracy of Frontier Large Language Models: Updated Comparison to Physician Performance
Michael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi · 2024
Read The Triage and Diagnostic Accuracy of Frontier Large Language Models: Updated Comparison to Physician Performance by Michael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11662182.txt.
Description
Michael Joseph Sorich and co-authors compare new large language models with doctors in a 2024 study. They ask how well the models sort patients by urgency and name the right diagnosis. An earlier test with GPT-3 found it matched lay people on triage and trailed doctors on diagnosis.
The team gave 48 case stories to three top models: GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Each model had to reason step by step, then pick its top three diagnoses in order. The cases covered both common and severe illness. The study updates the older benchmark so readers can see whether newer models have closed the gap with doctors.
Catalog record
| CREATOR | Michael Joseph Sorich, Arduino Aleksander Mangoni, Stephen Bacchi, Bradley Douglas Menz, Ashley Mark Hopkins, Amaryllis Mavragani, Christopher Williams, Qiming Shi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 7 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |