MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, Andrew Y. Ng, Jonathan H. Chen · 2025
Read MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents by Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, Andrew Y. Ng, Jonathan H. Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14654.txt.
Description
Yixing Jiang and co-authors wrote this 2025 paper introducing a test track for medical AI agents. Language models now act as agents that plan and use tools, but medicine lacked a standard way to grade them inside realistic record systems. The team built a virtual electronic health record world for that purpose.
MedAgentBench holds 300 physician-written tasks across ten categories and 100 patient profiles carrying over 700,000 data elements. Its environment speaks the same FHIR standard as real hospital systems, so the setup can migrate into live use. The best model tested, Claude 3.5 Sonnet v2, passed just under seventy percent of tasks. The authors stress the large headroom left, and the value of the benchmark as an unsaturated yardstick.
Catalog record
| CREATOR | Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, Andrew Y. Ng, Jonathan H. Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14654