A multi-agent large language model framework to automatically assess performance of a clinical AI Triage tool
Adam E Flanders, Yifan Peng, Luciano Prevedello, Robyn Ball, Errol Colak, Prahlad Menon, George Shih, Hui-Ming Lin, Paras Lakhani · 2026
Read A multi-agent large language model framework to automatically assess performance of a clinical AI Triage tool by Adam E Flanders, Yifan Peng, Luciano Prevedello, Robyn Ball, Errol Colak, Prahlad Menon, George Shih, Hui-Ming Lin, Paras Lakhani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-13233039.txt.
Description
Adam E Flanders and colleagues published a 2026 study on using teams of language models to grade a clinical artificial intelligence triage tool. The tool flags brain bleeding in scans, and checking its accuracy normally means humans re-reading many radiology reports. The researchers instead asked eight open-source language models plus an internal GPT-4o to judge whether reports described the bleeding.
Individual models varied in skill, with llama3.3:70b and GPT-4o reaching the highest accuracy scores. Groupings of models, including a top-three set and a full nine-model ensemble, performed about equally well, and no grouping clearly beat the others. Still, ensembles gave steadier judgments than any single model alone. The approach could make routine audits of hospital AI tools far cheaper.
Catalog record
| CREATOR | Adam E Flanders, Yifan Peng, Luciano Prevedello, Robyn Ball, Errol Colak, Prahlad Menon, George Shih, Hui-Ming Lin, Paras Lakhani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |