Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation

Dong Chen, Yanzhe Wei, Zonglin He, Guan-Ming Kuang, Canhua Ye, Meiru An, Huili Peng, Yong Hu, Huiren Tao, Kenneth MC Cheung · 2025

Read Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation by Dong Chen, Yanzhe Wei, Zonglin He, Guan-Ming Kuang, Canhua Ye, Meiru An, Huili Peng, Yong Hu, Huiren Tao, Kenneth MC Cheung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00588.txt.

Introduction

Dong Chen and nine co-authors put six leading language models through a safety check for spine surgery decision support. Hallucinations, outputs that sound plausible but clash with fact or context, could endanger patients if a model advises treatment. Their clinician-centered design scores diagnostic precision, recommendation quality, reasoning robustness, output coherence, and knowledge alignment across thirty expert-validated spinal cases.

One model, DeepSeek-R1, led overall, above all in trauma and infection cases. Reasoning-enhanced variants did not always beat their standard versions, a warning that longer chains of thought do not guarantee clinical reliability. Stress tests with amplified complexity lowered advice quality notably, even while surface clarity and coherence held or rose. The authors urge interpretability tools, such as visible reasoning chains, inside clinical workflows.

Catalog record

CREATORDong Chen, Yanzhe Wei, Zonglin He, Guan-Ming Kuang, Canhua Ye, Meiru An, Huili Peng, Yong Hu, Huiren Tao, Kenneth MC Cheung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation Dong Chen, Yanzhe Wei, Zonglin He, Guan-Ming Kuang, Canhua Ye, Meiru An, Huili Peng, Yong Hu, Huiren Tao, Kenneth MC Cheung Abstract Large language models (LLMs) hold transformative potential in clinical decision-support but introduce significant risks through hallucinations, which are factually inconsistent or contextually misaligned outputs that may compromise patient safety. This study introduces a novel, clinician-centered framework to quantify and mitigate hallucination risks in spine surgery by assessing diagnostic precision, recommendation quality, reasoning robustness, output coherence, and knowledge alignment. Our evaluation of six leading LLMs across 30 expert-validated Chinese spinal cases revealed robust inter-examiner reliability (mean r = 0.90 ± 0.014) among three independent surgeon-evaluators. DeepSeek-R1 demonstrated superior overall performance (total score: 86.03 ± 2.08), particularly in high-stakes domains such as trauma and infection. A critical finding emerged that reasoning-enhanced model variants did not uniformly outperform their standard counterparts. Notably, the extended thinking mode of Claude-3.7-Sonnet underperformed relative to its standard version (80.79 ± 1.83 vs. 81.56 ± 1.92), underscoring that extended chain-of-thought reasoning alone is insufficient for ensuring clinical reliability. Multidimensional stress-testing further exposed model-specific vulnerabilities in recommendation quality, which degraded by 7.4% under amplified informational complexity. This decline contrasted with marginal improvements in rationality (+2.0%), readability (+1.7%) and diagnosis (+4.7%) throughout the two-round decision-making workflow, highlighting a concerning divergence between perceived coherence and actionable clinical guidance. Our findings highlight the necessity of integrating interpretability mechanisms, such as reasoning chain visualization and real-time hallucination interception into high-risk clinical workflows. This work establishes the sequential validation framework for LLM reliability in dynamic surgical environments and advocates for a paradigm shift from accuracy-focused benchmarks to safety-aware, reasoning-grounded evaluation in medical AI. 1 Introduction Large language models (LLMs) are advanced artificial intelligence systems capable of parsing and generating human-like text, demonstrating unprecedented capabilities in healthcare [ 32 , 3 , 35 ] . LLMs offer significant potential to revolutionize healthcare and facilitate clinical workflows [ 16 , 46 , 37 ] , by automating data extraction [ 21 ] , synthesizing medical literature [ 7 ] , assisting differential diagnoses [ 9 , 42 , 24 , 18 ] , interpreting radiographic or pathologic reports [ 32 ] , personalizing patient care by identifying needs and preferences [ 22 , 15 ] , augmenting clinician training [ 36 ] , and streamlining administrative tasks [ 29 ] , among others. Despite their benefits, the use of LLMs still carries inherent risks [ 19 ] . Hallucinations, reported in numerous studies [ 1 , 33 , 10 , 40 , 27 , 4 ] , are a particular concern in clinical settings. These can manifest as interpretation errors, contextual misalignment, and embedded biases that may jeopardize patient safety and trust [ 1 , 33 , 43 ] . Studies have reported using several frameworks to assess the fidelity mismatch between LLM outputs and healthcare ground truth [ 1 , 48 ] . This tension between potential benefit and inherent risk highlights the critical need for rigorous, context-aware evaluation frameworks before widespread deployment. However, current hallucination evaluation frameworks mainly account for the severity of errors using taxonomies, the repeatability of LLM outputs, and the potential harms of the errors [ 1 , 48 ] . To tackle complex reasoning tasks, modern LLMs have employed chain-of-Thoughts (CoT) prompting strategies to enhance LLMs’ ability to parse, analyze, reflect upon, and ultimately solve complex problems [ 27 , 40 ] . Models incorporating CoT, such as GPT-o1 [ 17 ] and DeepSeek-r1 [ 11 ] , demonstrate capabilities surpassing conventional instruction-based LLMs, better mimicking human-like logical reasoning for intricate challenges [ 17 , 28 , 4 ] . Much research assessing LLMs in medicine has focused on the potential of serving as diagnostic aids within the physician-AI collaborative paradigm in clinical practice [ 1 ] . However, the reliance on the intermediary reasoning steps introduces novel risks, which render conventional benchmarking metrics inadequate for evaluating the clinical reliability of the outputs from thinking models with CoTs. On the one hand, current comprehensive benchmarking frameworks for CoT LLMs did not address the clinical healthcare setting [ 4 , 45 ] , such as peri-operative governance. For instance, benchmarking frameworks reported by Jiang et al. only focused on the LLM performances in math, science, optical character recognition (OCR), logic, space-time, and general scenes, neglecting healthcare setting [ 48 ] . On the other hand, currently reported evaluation metrics for LLM in healthcare were based on question banks to benchmark LLM performance against medical knowledge mastery within specific disease domains or clinical subspecialties [ 48 , 28 , 4 , 23 , 33 ] . Such approaches focused on static medical exams or datasets, hence failing to capture the dynamic, interdependent, and often ambiguous nature of real-world clinical reasoning [ 1 , 38 ] . For instance, Wang et al. only focused on the use of LLM in healthcare conversations, neglecting diagnostics and treatment planning [ 39 ] . However, in real-world clinical scenarios, patient symptoms are often non-specific, evidence can conflict, and disease progresses and evolves, while not all auxiliary tests are available. Therefore, to effectively evaluate the use of LLMs with CoT in real-world peri-operative governance settings, a novel benchmarking evaluation framework is needed to assess the diagnostic precision, quality of clinical action of the LLM-proposed examination and treatment suggestions, the reasoning robustness, output structure, and knowledge alignment [ 43 ] . Spinal surgeries present a uniquely compelling focus for such evaluation due to their exceptional clinical complexity and high-stakes decision-making requirements [ 47 , 44 , 6 , 5 , 12 , 25 ] . Spinal disorders represent a massive disease burden worldwide, spanning from common degenerative diseases like degenerative cervical myelopathy, the leading global cause of spinal cord impairment [ 31 , 41 ] , to rather rare and complex diseases like adolescent idiopathic scoliosis, the most common pediatric spinal disorder [ 30 , 26 ] . Moreover, low back pain alone is a primary driver of disability and the leading reason for medical consultations and abstinence from work [ 13 ] , with its prevalence and burden increasing with age [ 8 , 14 ] . This confluence of high clinical prevalence, procedural complexity, and severe consequence profiles establishes spinal surgery as an optimal domain for stress-testing LLMs’ clinical reasoning capabilities and safety alignment in high-risk scenarios. This study introduces a novel reasoning-oriented framework to assess LLMs in clinical settings, using spine surgery as a demonstration. Our primary goals are threefold: First, we establish a sequential framework to stratify the clinical workflow timeline, providing a nuanced evaluation that integrates three critical dimensions: diagnostic logic verification, therapeutic feasibility, and safety risk anticipation. This approach moves beyond performance benchmarks to capture the dynamic, multi-stage nature of clinical reasoning. Second, we introduce a pioneering methodology of targeted stress-testing at high-stakes surgical decision points. This is designed to rigorously probe LLM robustness under complexity, with performance quantified through a novel multi-metric assessment of diagnostic precision, clinical action quality, reasoning robustness, output coherence, and knowledge alignment. Finally, we create a transferable deployment foundation that prioritizes interpretability through reasoning chain visualization and safety through real-time hallucination interception.

Read the complete text