Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration
Marc Leon, Ruibin Feng, Manuel Quiroz Flores, Glenn Pelletier, Daniel Bethencourt, Masafumi Shibata, Hao He, Chawannuch Ruaengsri · 2026
Read Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration by Marc Leon, Ruibin Feng, Manuel Quiroz Flores, Glenn Pelletier, Daniel Bethencourt, Masafumi Shibata, Hao He, Chawannuch Ruaengsri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-13260534.txt.
Description
Marc Leon and colleagues staged in 2026 a blinded test of five large language models on hard cardiac surgery decisions. Senior surgeons wrote fifteen realistic scenarios with reference answers and a ten-part scoring frame. In round one, a separate surgeon panel graded model answers without knowing their source.
The reasoning-focused model O1 earned the highest median score, followed by O3-mini-high and DeepSeek-R1. Across models, grasp of the scenario scored best, while patient safety, avoiding invented facts, and efficiency scored worst. In round two, judges saw the reference answers and could revise their grades. Most revisions moved downward, showing that first impressions had been too kind. The authors call this overacceptance the main teamwork risk, and they judge the models not yet safe for complex surgical use.
Catalog record
| CREATOR | Marc Leon, Ruibin Feng, Manuel Quiroz Flores, Glenn Pelletier, Daniel Bethencourt, Masafumi Shibata, Hao He, Chawannuch Ruaengsri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |