Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai Shu · 2025
Read Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs by Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai Shu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20001.txt.
Introduction
Yunpeng Xiao and four co-authors argue that medical question-answering benchmarks flatter language models, because real clinical decisions are far richer than picking an answer. They propose a unifying framework that grades clinical tasks along two dimensions: how complete the clinical background is, and how realistic the question is. As both approach the real clinic, difficulty rises.
Existing datasets and benchmarks are mapped onto these axes, and methods for improving clinical decision making, at training time and at test time, are reviewed with guidance on when each helps. The authors push evaluation beyond raw accuracy to efficiency and explainability. They close with open challenges, aiming to standardize comparisons and steer development toward models that are clinically meaningful rather than merely test-smart.
Catalog record
| CREATOR | Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai Shu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20001
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs Yunpeng Xiao, Carl Yang, Mark Mai, Xiao Hu, Kai Shu Abstract Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (Q&A) that under represents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs. 1 Introduction In the fast-evolving field of artificial intelligence, large language models (LLMs) have made remarkable progress, showcasing impressive capabilities across diverse applications and opening new possibilities for innovation Hurst et al. 2024 ; Achiam et al. 2023 ; Roumeliotis and Tselikas 2023 ; Liu et al. 2024a ; Guo et al. 2025 . The integration of large foundational models into real-world medical practice holds tremendous promise, offering the potential to transform clinical workflows and patient care Singhal et al. 2023b ; Singhal et al. 2025 ; Wu et al. 2024c . Nowadays, more and more medical fine-tuned large language models are being developed, enriching the practice of professionals and researchers Chen et al. 2023 ; Saab et al. 2024 ; Wang et al. 2023 ; Wu et al. 2024a ; Sellergren et al. 2025 . LLM as a tool needs to be tested before implementation. One aspect of interest to developers of LLM is the performance. Towards testing performance, the medical field has been focused on using datasets, most of which were sourced from standard tests. Among these datasets, the most common are question-answering datasets, including MedQA Jin et al. 2021 , MMLU Hendrycks et al. 2020 , MedmcQA Pal et al. 2022 , MedxpertQA Zuo et al. 2025 , Pubmed QA Jin et al. 2019 , and so on. Each case typically contains context (patient history, laboratory data, etc.), a question, and multiple answer options. The model needs to choose the most appropriate option. These datasets have many advantages, such as standardization and easy evaluation of indicators, and have therefore been widely used to evaluate the performance of large language models in the medical field Sellergren et al. 2025 ; Saab et al. 2024 ; Xie et al. 2024a . However, there are still many differences between the LLM-powered clinical use cases and the question-and-answer (Q&A). For example, during the patient’s first clinical encounter, the doctor may not be able to obtain the patient’s medical history, test results, and other information in advance; he or she must ask about the patient’s condition and ask for necessary laboratory tests while avoiding meaningless questions and laboratory tests. But medical Q&A can often provide this information directly and precisely. The process of making choices in a clinical environment is called clinical decision making, which is a multistep process that requires gathering and synthesizing data from diverse sources and continuously evaluating the facts to reach an evidence-based decision on a patient’s diagnosis and treatment Hager et al. 2024 . Therefore, medical Q&A can be understood as a clinical decision-making process that simplifies the data collection and analysis process and simplifies clinical solutions. Such simplification promotes the standardization of large language model evaluation in the medical field, but it also raises questions about whether large language models can be applied in actual clinical scenarios. Some existing research has begun to recognize the gap between medical Q&A and real-world clinical decision-making, and has begun to explore more realistic task settings. For example, MedQA can be transformed into a doctor-patient dialogue, where doctors must ask patients questions or request tests to gradually obtain information before ultimately making a decision. However, the settings for these tasks vary. For example, in addition to converting MedQA to a doctor-patient dialogue Schmidgall et al. 2024 ; Li et al. 2024b ; Li et al. 2025b , recent work directly adds irrelevant content or modifies options in MedQA Pan et al. 2025 . Whether these settings truly reflect the clinical decision-making capabilities of large models, and to what extent, is not addressed by the unified paradigm. For the reasons mentioned above, this survey aims to propose a paradigm for clinical decision-making tasks. Based on existing works, we evaluate the complexity of a clinical medical decision-making task from two dimensions: Clinical Background and Clinical Questions. We summarize our contributions as follows: • We define the clinical decision-making from the perspective of task setting, which provides a reference for other researchers in the future. • We propose a paradigm to describe the real-world clinical decision-making tasks and summarize the existing literature. • We propose several open issues for future clinical decision-making tasks. 2 Definition of Clinical Decision-Making Previous definitions of clinical decision-making were summarized in the clinical context, or healthcare Banning 2008 ; Thompson and Dowding 2002 ; Hager et al. 2024 . In the era of large language models, it is important to address the definition of clinical decision-making because it could facilitate the standardization of LLMs development, research, and evaluation in clinical decision-making. According to our observation and existing literature, different medical datasets and clinical decision-making tasks can be divided into two parts: Clinical Backgrounds . Before making a decision, doctors need to collect necessary information, such as clinical conditions and patient history, which is collectively called the clinical background. Different background settings exist in different tasks. We divide them into four categories: (1) No Background; (2) Precise Information Background; (3) Rich Information Background; (4) Incomplete Information Background. Figure 1: Three different clinical settings (No clinical background setting is not shown in the figure). Clinical Questions . In most cases, clinical questions represent decisions that need to be made in a clinical background. In many research works and surveys, “tasks” and “questions” are often used synonymously, and “questions” are categorized according to specific medical tasks, such as EHR summary questions, disease diagnosis questions, etc. In this paper, we classify questions based on the question types. We divide them into four categories: (1) True or false questions, (2) Multiple choice questions, (3) Short-answer questions, and (4) Open-ended questions. Figure 2: Four different clinical question settings. We show different backgrounds and question settings in Figures 1 and 2. The combination of background and questions constitutes the case for clinical decision-making. In clinical practice, rich information background and incomplete information background, especially incomplete information background, and the short-answer questions and open-ended questions, especially open-ended questions, are common. However, most current medical question-answering datasets still use precise information backgrounds and multiple-choice questions. We will introduce each background and question in detail in the next two sections. 3 Clinical Backgrounds Clinical backgrounds are part of clinical decision-making. In this section, we introduce various background settings for clinical decision-making. These tasks have different settings; they can be the original datasets or benchmarks formed by