MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs
Hsin-Ling Hsu Zizheng Wang Donghua Zhang Nai-Chia Chen Jerry Wang, Jun-En Ding Chia-Hsuan Hsu Guoan Wang Feng Liu Fang-Ming Hung, Chenwei Wu Liyue Shen · 2026
Read MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs by Hsin-Ling Hsu Zizheng Wang Donghua Zhang Nai-Chia Chen Jerry Wang, Jun-En Ding Chia-Hsuan Hsu Guoan Wang Feng Liu Fang-Ming Hung, Chenwei Wu Liyue Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07305.txt.
Introduction
Hsin-Ling Hsu and eleven co-authors train medical language models to diagnose actively, the way doctors work in practice. Most model testing is done with complete information given at once, unlike real practice, where doctors order tests and update their thinking over turns. The team finds three common failures: ungrounded test choices, weak updates, and fading coherence.
Their MedAction pipeline builds multi-turn case dialogues through model-environment interaction, filtered by two knowledge-graph metrics for consistency. The resulting dataset holds tens of thousands of such records from thousands of published cases. A fine-tuned open model reached leading open-source results on two testing benchmarks, pushing active testing forward.
Catalog record
| CREATOR | Hsin-Ling Hsu Zizheng Wang Donghua Zhang Nai-Chia Chen Jerry Wang, Jun-En Ding Chia-Hsuan Hsu Guoan Wang Feng Liu Fang-Ming Hung, Chenwei Wu Liyue Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07305
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs Hsin-Ling Hsu Zizheng Wang Donghua Zhang Nai-Chia Chen Jerry Wang, Jun-En Ding Chia-Hsuan Hsu Guoan Wang Feng Liu Fang-Ming Hung, Chenwei Wu Liyue Shen Abstract Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs --- ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence --- that together reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM--environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model’s hypothesis converges toward the correct diagnosis, and Reasoning–Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs. 1 1 1 These authors contributed equally. 2 2 footnotetext: Co-senior authorship. Contact: [email protected] , [email protected] 1 1 footnotetext: Code and data available at https://github.com/JustinHsu1019/MedAction 1 Introduction Large language models (LLMs) have shown promising performance on medical question answering (QA) benchmarks, suggesting great potential as clinical decision-support tools for diagnosis ( 27 ; 10 ) . However, most existing benchmarks formulate the medical diagnosis as a static, one-shot QA task: the model receives a complete and fully assembled patient record including laboratory results, imaging, and biomarkers, implicitly assuming such data are already available, and then is asked to predict the diagnosis via a well-posed question prompt. This setup is highly simplified and unrealistic and does not reflect the complexities of real-world clinical diagnosis. Instead, as shown in Figure 1 , medical diagnosis is an active, iterative process. In practice, clinicians always begin with incomplete information, then iteratively recommend new tests to resolve uncertainty, and progressively narrow the differential diagnosis based on new test results accordingly, until making a final definitive diagnosis with sufficient confidence. We call this new formulation active diagnosis with iterative test ordering . To build LLMs that can be genuinely useful in clinical settings, we must enable their capability for active diagnosis via evaluation in such multi-turn, information-gathering regime. While state-of-the-art LLMs perform well on static medical QA ( 10 ; 23 ; 11 ) with growing ability in multi-turn tasks in general domains ( 31 ; 16 ; 35 ) , a recent study ( 24 ) reveals a striking performance degradation when the same models are evaluated on active medical diagnosis task. This gap persists even for powerful agentic models such as GPT-o3 ( 19 ) and medical specialist models such as MedGemma ( 26 ) . To understand what makes this setting uniquely challenging, we investigate LLM behavior in active multi-turn diagnosis and identify three representative failure modes (Figure 2 ). Ungrounded test ordering. The clinical action space spans thousands of laboratory tests, imaging studies, and procedures, resulting in a highly complex decision landscape. We find that LLMs frequently order tests without a clear connection to their stated hypotheses, lacking case-specific clinical reasoning and context-grounded actions. Unreliable diagnostic update. Interpretation of clinical test results is non-trivial, requiring consideration of competing hypotheses, prior evidence, and patient context. Due to imprecise understanding of newly acquired test results, LLMs often adhere rigidly to an initial diagnosis despite contradictory evidence, or shift erratically between unrelated diagnoses. Degraded multi-turn coherence. As evidence accumulates across multi-turn interactions, LLMs often lose track of which tests have been ordered, what results were obtained, and what uncertainty remains unresolved, leading to inconsistencies in longer context. These failures reveal critical limitations of existing medical LLMs, which are usually finetuned with medical training data such as chain-of-thought (CoT) reasoning trajectories ( 30 ; 2 ) : models are trained to think — perform reasoning toward a diagnosis assuming complete information — but not to act — sequentially decide which new information to gather (“action”) and update the current hypothesis (“status”) in response to newly collected information. Addressing these challenges in active diagnosis requires the development of multi-turn interactive medical training data that can equip LLMs with such abilities to both reason and act correctly. Static Diagnosis Complete Patient Record CC Labs Exam Imaging Vitals Pathology Diagnosis vs. Active Diagnosis 6 DDx 4 DDx 2 DDx Turn 1 Turn 2 Turn 3 Final CC + Exam CC + Exam Labs CC + Exam Labs Imaging Order next test Diagnosis Figure 1: Static vs. Active Diagnosis. Left: Existing benchmarks provide a complete patient record and ask for a one-shot diagnosis. Right: In active diagnosis, the model begins with only the chief complaint (CC) and physical examination, then iteratively orders new tests to resolve uncertainty and progressively narrows the differential diagnosis (DDx) from six candidate conditions down to a definitive diagnosis through multiple turns. In this work, we introduce MedAction , a framework for automatically constructing high-quality multi-turn active diagnosis training data from publicly available medical literature. Unlike existing medical CoT data that teaches models to think from complete information, MedAction trajectories capture both thinking and acting under partial, evolving evidence. Starting from raw case reports, we first extract 2896 interactive clinical environments that simulate the partial-information setting of active diagnosis: the model observes only the initial patient information and needs to take proactive actions to request new test results turn by turn. We then construct structured multi-turn trajectories by having state-of-the-art LLMs interact with these environments, producing chains of reasoning-action data that mirror real clinicians’ diagnostic procedure: maintaining a differential, identifying diagnostic uncertainty, and selecting tests to resolve it. To distill high-quality training data and maximize data utilization, we propose two knowledge-graph-grounded metrics to enable clinically grounded rejection sampling. Disease Trajectory Consistency (DTC) measures the knowledge-graph distance between the model’s current top hypothesis and the ground-truth diagnosis on PrimeKG ( 1 ) , allowing us to retain complete trajectories that reach the correct answer as well as partially correct trajectories where the hypothesis was still converging. Reasoning–Action Consistency (RAC) then measures whether ordered tests drove meaningful changes in the differential on ClinGraph ( 13 ) , filtering individual turns where hypothesis shifts were unrelated to gathered evidence. We further investigate key design choices in trajectory construction, including teacher model selection, tree sampling strategy, and filtering thresholds, providing actionable guidance for building multi-turn medical reasoning-action data. Training on the resulting MedAction-32K dataset , our 8B-parameter model achieves state-of-the-art performance among open-source models under multi-turn active diagnosis settings, demonstrating that high-quality trajectory data, even at modest scale, drives larger gains than architecture scaling alone. In addition to the training set, we curate MedAction-300 Hard , a test set of 300 rare-disease cases that demand longer diagnostic trajectories and more extensive test ordering than existing benchmarks. Figure 2: Three