CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine

Kevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, Bradley A. Malin · 2026

Read CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine by Kevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, Bradley A. Malin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01011.txt.

Introduction

Kevin H. Guo and seven co-authors introduce CLEAR, a framework that tests how noise and ambiguity degrade medical language models. Standard evaluations use simplified exam-style questions that rarely reflect real medical inquiries. CLEAR systematically changes three factors: how many plausible answer options are shown, whether a ground truth or abstention option is present, and how answer options are framed semantically.

Applied to three benchmarks across seventeen models, the tests reveal clear limits. More plausible answers weaken both correct selection and proper abstention. Caution drops further when abstention wording shifts from assertive rejection to admitting uncertainty, and merely including an I don't know option raises incorrect selections. A formal humility deficit, the gap between finding correct answers and abstaining on wrong ones, worsens with model scale.

Catalog record

CREATORKevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, Bradley A. Malin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01011

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine Kevin H. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, Bradley A. Malin Abstract Medical large language model (LLM) evaluations rely on simplified, exam-style benchmarks that rarely reflect the ambiguity of real-world medical inquiries. We introduce the CLinical Evaluation of Ambiguity and Reliability (CLEAR) framework, which assesses how decision-space presentation, ambiguity, and uncertainty affect LLMs’ reasoning on medical benchmarks. CLEAR systematically perturbs (1) the number of plausible answer options, (2) the presence of a ground truth or abstention option, and (3) the semantic framing of answer options. Applying CLEAR on three benchmarks evaluated across 17 LLMs reveals three notable limitations of existing evaluation methods. First, increasing the number of plausible answers degrades a model’s ability to identify the correct answer and abstain against incorrect ones. Second, this lack of caution intensifies as the framing of abstention shifts from assertive rejection like “None of the Above” to uncertainty admission like “I don’t know” (IDK). Notably, just including IDK in the answer space increases incorrect answer selections. Lastly, we formalize the performance gap between identifying the correct answer and abstaining from incorrect ones as the humility deficit , which worsens with model scale. Our findings reveal limitations in standard medical benchmarks and underscore that scaling alone does not resolve LLM reliability issues. keywords . Introduction The increasing digitization of healthcare data has introduced the need for tools capable of translating raw data into actionable insights [ 7 , 40 , 12 , 36 ] . Large language models (LLMs) have emerged as potential solutions to this translational bottleneck, demonstrating expert-level or superior performance on medical benchmarks [ 27 , 41 , 37 ] and witnessing early deployment across diverse clinical settings [ 24 , 1 , 22 , 29 , 43 , 13 ] . As LLMs move closer to the point of care, we must ensure their reliability in addition to diagnostic accuracy. To date, LLM evaluations have relied on benchmarking tasks designed to quantify the upper bounds of model performance [ 19 , 32 , 16 , 44 ] . For example, one of the most common medical benchmarks is MedQA, a dataset of multiple-choice questions derived from the United States Medical Licensing Exam (USMLE) [ 19 ] . While such exam-style benchmarks offer a standardized means of measuring medical fluency [ 15 ] , they are constrained by simplified and well-defined decision spaces that rarely reflect real-world ambiguity and complexity. Specifically, benchmarks typically operate under the idealized assumption that the correct answer and the information needed to reach it are present, which is often not the case in medicine [ 44 , 3 ] . Subsequently, LLMs that demonstrate expert level performance on exam-style benchmarks perform poorly in more practical and complex tasks such as differential diagnosis [ 34 ] . In practice, clinicians frequently navigate complex and incomplete patient data. This process involves filtering out extraneous, irrelevant information to discern the findings most relevant to making diagnoses [ 38 , 8 ] . Similarly, patients are increasingly turning to LLMs to address their own health concerns, presenting self-reported, often fragmented information for models to provide guidance [ 5 ] . However, recent evidence suggests that LLM performance, as measured by benchmarks, is fragile to variations in how a decision space is structured and presented [ 15 ] . For instance, one study found that simply paraphrasing common benchmark inputs caused evaluation performance to drop significantly across 34 state-of-the-art LLMs [ 25 ] . In another study focusing on MedQA, researchers discovered that replacing the correct answer with “None of the other answers” caused a reduction in LLM performance by up to 38% [ 6 ] , highlighting a lapse in both LLM decision-making and the benchmarks used to evaluate them. Yet another study found that LLMs are prone to complying with illogical medical requests such as differentiating between brand and generic name drugs, and frequently provide confident justifications for factually incorrect assumptions [ 11 ] . Whether it is clinicians or patients seeking medical advice, LLM reasoning must be robust to how a decision space is presented, and prioritize the humility to withhold diagnoses, admit uncertainty, or request additional information, when unsure or faced with insufficient clinical signal [ 9 , 33 ] . Existing evaluation methods lack the mechanisms to capture how LLMs navigate this decision-space noise, ambiguity, and uncertainty, leaving the reliability and humility of these systems for medical advice-seeking largely untested. To address this gap, we introduce the CL inical E valuation of A mbiguity and R eliability (CLEAR) framework, which systematically perturbs existing benchmarks in (1) the number of plausible, competing answer options, (2) the presence of the ground truth or abstention options (in replacement of the ground truth), and (3) the semantic framing of answer options. Together, these perturbations adapt existing question-answer benchmarks to provide insight into how noise, open-world ambiguity, and uncertainty influence LLM reliability and humility in medical advice seeking. We evaluate 17 popular open-source and proprietary models on three medical benchmarks, each perturbed using the CLEAR evaluation framework, totaling over 2.6 million unique inferences. Our analysis with CLEAR reveals several notable gaps in reliability that existing benchmarks fail to capture. Specifically, we find that LLMs become increasingly lost as a decision space saturates with competing answer options, both in identifying the clinical ground truth and in abstaining from incorrect decisions. In the latter case, beyond getting lost, models actively avoid abstaining, admitting uncertainty, and seeking assistance and instead, overwhelmingly select incorrect answers. Notably, just the presence of an option to admit uncertainty can cause incorrect selection rates to spike. While increasing model and reasoning complexity improve resilience against increasing decision space noise, they simultaneously exacerbate these humility-lacking behaviors, highlighting a potential scaling law failure where further scaling no longer yields proportional gains in calibrated reasoning or epistemic humility. Our analyses reveal that even within the medical domain, reliability and humility can vary across contexts, highlighting the limitations of current evaluation paradigms. Motivated by these findings, we conclude with best-practice recommendations for safely adapting LLMs to navigate real-world medical scenarios. Figure 1 : Overview of the CLEAR evaluation framework. a, MedMCQA, MedQA, and JAMA Clinical Challenges (JAMA-CC) benchmark datasets, spanning foundational biomedical knowledge questions to complex, real-world clinical scenario-based decision-making. b, CLEAR evaluates accuracy, measured by the model’s ability to select the correct answer, as the number of plausible but incorrect distractor options progressively increases. Models are evaluated under both direct-answer (base) and chain-of-thought (CoT) prompting strategies. c, CLEAR removes and replaces the clinical ground truth with one of three abstention options. Each represents a distinct semantic framing of the same functional objective, avoiding commitment to an incorrect answer when the clinical ground truth is unavailable. CLEAR also evaluates how LLMs interpret and decouple assertive abstention from uncertainty admission. System prompts for each strategy are included in Supplementary Figure S1 . Results Overview of CLEAR and experimental setup Building on existing medical benchmarks, CLEAR systematically introduces competing answer options, uncertainty, and ambiguity into the decision space to more closely resemble real-world medical advice-seeking scenarios. Specifically, we evaluate model reliability as a decision space is

Read the complete text