Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction
Jinrui Fang 1 , Runhan Chen 2 , Xu Yang 1 , Jian Yu 1 , Jiawei Xu 1 , Ashwin Vinod 1 , Wenqi Shi 3 , Tianlong Chen 4 , Heng Ji 5 , ChengXiang Zhai 5 , Ying Ding 1 , Yuji Zhang 5 1 The University of Texas at Austin 2 New York University 3 The University of Texas Southwestern Medical Center 4 University of North Carolina at Chapel Hill 5 University of Illinois Urbana-Champaign [email protected], [email protected], [email protected] · 2026
Read Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction by Jinrui Fang 1 , Runhan Chen 2 , Xu Yang 1 , Jian Yu 1 , Jiawei Xu 1 , Ashwin Vinod 1 , Wenqi Shi 3 , Tianlong Chen 4 , Heng Ji 5 , ChengXiang Zhai 5 , Ying Ding 1 , Yuji Zhang 5 1 The University of Texas at Austin 2 New York University 3 The University of Texas Southwestern Medical Center 4 University of North Carolina at Chapel Hill 5 University of Illinois Urbana-Champaign [email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04325.txt.
Introduction
Fang and colleagues introduce MINT, a multi-turn medical diagnosis benchmark of 1,035 cases with labelled evidence. Testing 11 language models reveals three habits. Models rush to answer early, with over 55 percent of answers locked in within two turns. Wrong-to-right revisions outnumber right-to-wrong flips by about 11 times, showing unused self-correction. Striking lab results trigger early answers even when models are told to wait. The authors advise delaying the diagnostic question and holding back striking evidence until later turns, which cuts early answers and lifts accuracy.
Catalog record
| CREATOR | Jinrui Fang 1 , Runhan Chen 2 , Xu Yang 1 , Jian Yu 1 , Jiawei Xu 1 , Ashwin Vinod 1 , Wenqi Shi 3 , Tianlong Chen 4 , Heng Ji 5 , ChengXiang Zhai 5 , Ying Ding 1 , Yuji Zhang 5 1 The University of Texas at Austin 2 New York University 3 The University of Texas Southwestern Medical Center 4 University of North Carolina at Chapel Hill 5 University of Illinois Urbana-Champaign [email protected], [email protected], [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58181 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction Jinrui Fang 1 , Runhan Chen 2 , Xu Yang 1 , Jian Yu 1 , Jiawei Xu 1 , Ashwin Vinod 1 , Wenqi Shi 3 , Tianlong Chen 4 , Heng Ji 5 , ChengXiang Zhai 5 , Ying Ding 1 , Yuji Zhang 5 1 The University of Texas at Austin 2 New York University 3 The University of Texas Southwestern Medical Center 4 University of North Carolina at Chapel Hill 5 University of Illinois Urbana-Champaign [email protected], [email protected], [email protected] Abstract Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn, yet how they behave under multi-turn evidence accumulation closer to real clinical reasoning remains unexplored. We introduce MINT 1 1 1 The data and code will be released upon publication. ( M edical I ncremental N - T urn Benchmark), a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases with clinically labeled evidence shards, controlled turn granularity, and information-preserving decomposition. Through systematic evaluation of 11 LLMs on MINT, we uncover three persistent behavioral patterns that significantly impact diagnostic decisions: (1) intent to answer , models rush to answer before sufficient evidence has been observed, with over 55% of answers committed within the first two turns; (2) self-correction , incorrect-to-correct answer revisions occur at up to 10.6 × \times the rate of correct-to-incorrect flips, revealing a latent capacity for self-correction that premature commitment forecloses; and (3) strong lures , clinically salient information such as laboratory results trigger premature answering even when models are explicitly instructed to wait. We translate these findings into clinically actionable guidance: deferring the diagnostic question to later turns reduces premature answering and improves accuracy at the first point of commitment by up to 62.6%, while reserving salient clinical evidence for later turns prevents a catastrophic accuracy drop of up to 23.3% caused by premature commitment. Our work provides both a controlled evaluation framework and concrete recommendations for improving the reliability of LLMs in multi-turn medical diagnosis. 1 Introduction Large language models (LLMs) have demonstrated strong performance in medical diagnosis under standard single-turn evaluation, where all clinical information is presented at once ( Chen et al. 2025 ; Wang et al. 2024 ) . However, real-world clinical reasoning rarely follows this static paradigm ( Liu et al. 2025 ; Fan et al. 2025 ; Li et al. 2024 ) . In practice, clinicians gather evidence sequentially through multi-turn interactions (e.g., history taking, iterative diagnostic workup), gradually refining their judgment as new information emerges ( Liu et al. 2025 ; Fan et al. 2025 ; Li et al. 2024 ) . This discrepancy raises a critical question: can LLMs maintain reliable diagnostic reasoning in sequential, multi-turn settings? Recent observations suggest that LLM performance degrades substantially when transitioning from single-turn to multi-turn settings ( Laban et al. 2026 ; Deshpande et al. 2025 ) , particularly in high-stakes domains such as medicine ( Liu et al. 2025 ; Fan et al. 2025 ; Li et al. 2024 ) . Several hypotheses have been proposed to explain this degradation. One line of reasoning attributes it to the model’s tendency to commit prematurely to an answer upon receiving partial information, without sufficient reasoning ( Laban et al. 2026 ; Li et al. 2024 ) . Another suggests that models lose track of relevant context over extended interactions ( Laban et al. 2026 ; Deshpande et al. 2025 ; Li et al. 2025 ) . Despite these accounts, a fundamental research gap remains: existing evaluations lack a controlled benchmark that can disentangle the effects of decision timing ( hold ), restoration ( self-correction ), and information exposure ( lure ) on model performance. Without such a benchmark, it remains unclear whether observed degradation stems from genuine model limitations or from artifacts introduced by uncontrolled data construction and evaluation protocols. To address this gap, we construct MINT, a high-fidelity, fine-grained, multi-turn medical diagnosis benchmark that enables systematic investigation of model behavior under sequential evidence accumulation. MINT is built around three design principles: information preservation, controlled turn granularity, and clinically labeled evidence shards. Building on MINT, we conduct a comprehensive analysis of the mechanisms underlying multi-turn performance degradation and identify three persistent behavioral patterns: (1) Intent to answer. By controlling when diagnostic questions are presented, we observe a large gap between initial and final answer accuracy when questions are presented early, indicating that premature answering significantly harms performance. Withholding the diagnostic question until all clinical information has been revealed largely restores accuracy to single-turn levels, demonstrating that models retain strong diagnostic potential when early answering is prevented. This result offers a key distinction from prior work, which has documented premature answering as a failure mode ( Li et al. 2024 ; Laban et al. 2026 ; Luo et al. 2025 ) but has not systematically isolated answer timing as a controlled variable: models retain strong diagnostic potential when commitment is deferred to later turns. (2) Self-correction. We further analyze how holding intent shapes model reasoning dynamics across turns. Notably, transitions from incorrect to correct answers occur at a significantly higher rate (10.6 \times ) than transitions from correct to incorrect. This finding suggests that holding intent does not merely delay answering, but actively unlocks the model’s capacity for self-correction during multi-turn reasoning. (3) Strong lures. Certain types of clinical information (e.g., lab results) act as strong lures that trigger immediate answering behavior. Models are significantly more likely to answer prematurely when laboratory results appear in early turns, an effect especially pronounced for lab-dependent diseases, leading to substantially higher error rates. These findings reveal that answering behavior is driven not only by internal intent, but also by the type and timing of incoming clinical evidence. Figure 1: Overview of MINT benchmark construction and evaluation settings. In summary, this work makes the following novel contributions (illustrated in Figure 1 ): • MINT Benchmark. We construct a high-fidelity, fine-grained multi-turn medical diagnosis benchmark, MINT, with controlled decomposition, verification, and structured labeling, enabling rigorous study of LLM behavior in multi-turn medical diagnosis. • Intent to Answer. By controlling when diagnostic questions are posed, we identify a universal pattern of premature commitment across all evaluated models and demonstrate that deferring the question to later turns substantially recovers diagnostic accuracy. • Self-Correction. By tracking reasoning dynamics across turns, we demonstrate a persistent pattern of strong self-correction across all models, with commercial LLMs demonstrating up to 10.6 \times higher rates of turning incorrect answers into correct ones. • Strong Lures. By presenting clinically salient information (e.g., lab results) in early turns, we uncover a consistent pattern in which strong lures reliably trigger premature diagnostic decisions, leading to substantially higher error rates. 2 Related Work Multi-turn Degradation in LLMs. Prior work has consistently shown that LLMs are less reliable in multi-turn than in single-turn settings. Multi-turn competence varies widely across tasks ( Deshpande et al. 2025 ) , instruction-following degrades across turns ( Sun et al. 2024 ) , and even frontier commercial models struggle with sustained contextual reasoning ( Deshpande et al. 2025 ; Laban et al. 2026 ) . These failures often stem not from longer context per se, but