Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications

Xiao Ye Jacob Dineen Zhaonan Li Zhikun Xu Weiyu Chen, Shijie Lu Yuxi Huang Ming Shen Phu Tran Ji–Eun Irene Yum, Muhammad Ali Khan Muhammad Umar Afzal Irbaz Bin Riaz Ben Zhou, [0.25em] Arizona State University · 2025

Read Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications by Xiao Ye Jacob Dineen Zhaonan Li Zhikun Xu Weiyu Chen, Shijie Lu Yuxi Huang Ming Shen Phu Tran Ji–Eun Irene Yum, Muhammad Ali Khan Muhammad Umar Afzal Irbaz Bin Riaz Ben Zhou, [0.25em] Arizona State University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17764.txt.

Description

Xiao Ye and thirteen co-authors survey how medical language models should be judged. Such models score well on exam-style benchmarks. Strong scores do not prove they work safely in real clinics. The team argues that judgment should depend on how much freedom a model has.

They sort medical tools into four levels of autonomy, numbered zero to three. Low levels only inform or reshape information for staff. Higher levels support decisions or act as supervised agents. Each level allows different actions and carries different risks. The survey maps benchmarks to levels and offers a reporting blueprint.

Catalog record

CREATORXiao Ye Jacob Dineen Zhaonan Li Zhikun Xu Weiyu Chen, Shijie Lu Yuxi Huang Ming Shen Phu Tran Ji–Eun Irene Yum, Muhammad Ali Khan Muhammad Umar Afzal Irbaz Bin Riaz Ben Zhou, [0.25em] Arizona State University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17764

Browse the full library · About the Tower of Babel