Multimodal foundation models exploit text to make medical image predictions
Thomas A. Buckley, James A. Diao, Cam N. Srivastava, Peter G. Brodeur, Pranav Rajpurkar, Adam Rodman, Arjun K. Manrai · 2026
Read Multimodal foundation models exploit text to make medical image predictions by Thomas A. Buckley, James A. Diao, Cam N. Srivastava, Peter G. Brodeur, Pranav Rajpurkar, Adam Rodman, Arjun K. Manrai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/PMC13408344.txt.
Description
Thomas Buckley and co-authors put eight leading AI systems to a 2026 test across more than a thousand medical cases. Each case paired images with written case notes. Vendors claim these models reason over scans and text together. Yet nobody had pinned down which input truly drives their answers.
Accuracy rose steadily as the text grew more helpful. Even a mild hint toward a wrong diagnosis dragged scores down sharply. One model fell from strong to poor accuracy on cases it had read correctly from images alone. In long written cases judged by doctors, adding images sometimes lowered accuracy instead of raising it. The study warns hospitals that current systems lean heavily on wording, for better and worse.
Catalog record
| CREATOR | Thomas A. Buckley, James A. Diao, Cam N. Srivastava, Peter G. Brodeur, Pranav Rajpurkar, Adam Rodman, Arjun K. Manrai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38014 |
| LICENSE | https://creativecommons.org/licenses/by/4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |