AI for evidence-based treatment recommendation in oncology: a blinded evaluation of large language models and agentic workflows
Guannan Zhai, Merav Bar, Andrew J. Cowan, Samuel Rubinstein, Qian Shi, Ningjie Zhang, En Xie, Will Ma · 2025
Read AI for evidence-based treatment recommendation in oncology: a blinded evaluation of large language models and agentic workflows by Guannan Zhai, Merav Bar, Andrew J. Cowan, Samuel Rubinstein, Qian Shi, Ningjie Zhang, En Xie, Will Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-12722510.txt.
Description
Cancer doctors face a flood of new evidence, and language models promise help keeping up. An eight-author team staged a blinded contest: five AI systems answered fifty clinical scenarios about multiple myeloma, a blood cancer, covering diagnosis, staging, and treatment. Three expert oncologists scored the answers for accuracy, relevance, completeness, invented facts, and readiness for clinical use.
A system built as an agentic workflow, coordinating several tools and steps, won clearly, reaching 82 percent accuracy. Raw chatbots scored far lower, and a retrieval-augmented system sat between. Invented facts were uncommon everywhere, at roughly 3 to 10 percent. Yet readiness for real clinical use was low for every system, topping out near 25 percent. The authors conclude the tools can support doctors but cannot yet be trusted unsupervised.
Catalog record
| CREATOR | Guannan Zhai, Merav Bar, Andrew J. Cowan, Samuel Rubinstein, Qian Shi, Ningjie Zhang, En Xie, Will Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |