Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

Nisarg Patel · 2026

Read Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency by Nisarg Patel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29876.txt.

Introduction

Patel asks whether the reasoning of large language models is consistent, not just accurate. Models reach 60 to 70 percent accuracy on hard clinical case benchmarks, but a right answer can hide unstable pattern matching.

Patel converts free-text reasoning into clinical reasoning graphs, structured maps built with a medical scheme of five node types and seven edge types. The analysis covers 750 traces from five models on 50 published case-conference cases under three prompting styles. Graph similarity is nearly the same for similar and dissimilar cases, and no comparison survives statistical correction, so no stable diagnostic schemas appear. Structured reflection makes traces discuss distinguishing features more, but not more consistently.

Catalog record

CREATORNisarg Patel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency Nisarg Patel Abstract Modern large language models (LLMs) achieve 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching. We introduce clinical reasoning graphs , structured graph representations extracted from free-text LLM diagnostic traces using a domain-grounded ontology with 5 node types and 7 edge types. Applying this extraction pipeline to 750 traces from five LLMs across 50 New England Journal of Medicine Clinicopathological Conference cases and three prompt conditions, we test whether LLM diagnostic traces show stable structured reasoning patterns, i.e., “diagnostic schemas”, for clinically similar cases, determined by higher graph similarity among clinically similar cases than among clinically dissimilar cases. Across 15 model–condition comparisons, within-cluster and between-cluster composite similarity are nearly equal, with no comparison surviving multiple-testing correction; a component-level analysis finds any residual content signal to be far below schema scale. Graph similarity was also similar for pairs of models that were both correct versus both incorrect, suggesting that graph structure captures a dimension not reflected in diagnosis accuracy. Structured reflection prompting increased explicit discriminating-feature analysis within traces, but did not increase cross-case consistency. We release the ontology, extraction pipeline, validation protocol, and the extracted reasoning graphs and similarity artifacts as resources for structured evaluation of LLM clinical reasoning. 1 Introduction Recent large language models perform increasingly well on complex diagnostic benchmarks ( Eriksen et al. 2023 ; Arora et al. 2025 ) , motivating closer evaluation of not only their final answers but also their reasoning traces. Yet, clinical reasoning traces, the free-text explanations that models produce to support their clinical decisions, remain unstructured, difficult to audit and compare across cases, and difficult to evaluate with traditional accuracy metrics alone. Clinical reasoning science offers a richer framework for evaluating traces. Expert diagnosticians are thought to organize knowledge through diagnostic schemas , which are reusable mental representations that link clinical presentations, enabling conditions, key findings, and discriminating features ( Charlin et al. 2000 ; Bowen 2006 ) . These structures help physicians recognize familiar patterns while updating their reasoning when case-specific findings do not fit a particular framework; for example, a cardiologist encountering chest pain with ST elevation mentally activates a structured pattern of symptom (feature) recognition, hypothesis generation (e.g., myocardial infarction), and discriminating-feature analysis (e.g., which ECG leads show ST elevation?) that is broadly similar for each case. Here we ask: do LLM diagnostic traces exhibit anything analogous to diagnostic schemas? For example, if a model reasons about two nephrotic syndrome cases, do the resulting traces identify overlapping clinical features, differentials, semantic qualifiers, and discriminating features? Or do they follow unrelated reasoning paths that happen to converge on similar final answers? Answering this requires transforming free-text reasoning traces into structured, computable representations. We introduce clinical reasoning graphs : directed graphs extracted from LLM diagnostic traces using a domain-grounded ontology that captures the entities and relationships of clinical reasoning. Our contributions: 1. A clinical reasoning ontology with 5 node types and 7 edge types, grounded in established diagnostic reasoning models, designed to represent features, diagnoses, semantic qualifiers, discriminating features, evidence references, and reasoning-phase transitions 2. A scalable extraction pipeline converting unstructured traces into validated graphs (750/750 successful, mean 47.0 nodes, 58.7 edges per graph). 3. An empirical test of reasoning consistency : no evidence of schema-scale cross-case consistency in reasoning traces for clinically similar cases, despite strong diagnostic performance. 4. Evidence that accuracy and reasoning structure capture different evaluation dimensions : graph similarity is 0.488 when both models are correct versus 0.484 when both are incorrect. 2 Methods 2.1 Study Design We analyze 750 diagnostic reasoning traces generated under a preregistered protocol for diagnostic accuracy and confidence calibration ( Patel 2026 ) (generation details in Appendix A ). The traces span 50 New England Journal of Medicine (NEJM) Clinicopathological Conference (CPC) cases stratified by physician-assigned difficulty into 15 easy, 20 moderate, and 15 hard cases, and grouped into 20 clinical clusters for the consistency analysis. Five contemporary LLMs were evaluated: GPT-5.4, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 3 Pro. Each case was presented under three prompting conditions. The baseline prompt requested a differential diagnosis and probability estimates directly. The adversarial prompt used a phased protocol and asked the model to argue against its leading diagnosis before offering a revised answer. The structured reflection prompt used a phased protocol requiring a problem representation, balanced stress test, and explicit defend-or-update decision for each diagnosis ( Dhaliwal 2017 ) . Each trace was generated from a single prompt in one autoregressive pass at temperature 1.0 ; the phased conditions sequence their protocol phases within that prompt rather than across separate model turns. Final diagnoses were scored against the CPC reference diagnosis by an LLM judge (Claude Opus 4.6). 2.2 Clinical Reasoning Ontology We define a directed graph schema grounded in clinical reasoning theory ( Charlin et al. 2000 ; Bowen 2006 ; Dhaliwal 2017 ) : The ontology contains five node types: (1) clinical features, (2) diagnoses, (3) semantic qualifiers, (4) discriminating features, and (5) evidence references. Clinical features represent signs, symptoms, laboratory values, and imaging findings; diagnoses represent candidate conditions with probability estimates; semantic qualifiers represent abstracted descriptors such as “acute-onset”; discriminating features represent findings used to distinguish between diagnoses; and evidence references represent cited clinical knowledge, such as guidelines, clinical rules, and studies. The seven edge types capture (1) evidential support, (2) evidence against a diagnosis, (3) discrimination between diagnoses, (4) triggered reflection, and post-reflection diagnosis updates: (5) promoted, (6) demoted, or (7) unchanged. Figure 1 : A clinical reasoning graph extracted from a single LLM trace, read top-down through phases. Features (boxes) support or argue against diagnoses (circles; numbers are model confidence). Idiopathic pulmonary fibrosis (IPF) leads in Phase 1 ( 60\% ); in Phase 2, discriminating features (nodules, lymphadenopathy, lack of basal predominance) argue against IPF, support sarcoidosis, and trigger a reflection that promotes fibrotic sarcoidosis to the revised diagnosis ( 65\% ). Abbreviations: HP, hypersensitivity pneumonitis; IPF, idiopathic pulmonary fibrosis; CHP, chronic hypersensitivity pneumonitis; CTD-ILD, connective tissue disease–associated ILD; NSIP, nonspecific interstitial pneumonia. 2.3 Graph Extraction Pipeline Each trace was processed by GPT-5.4 using a structured extraction prompt. The prompt specified the ontology, required schema-valid JSON output, enforced connected graphs, required phase tags for all edges, and constrained reflection-related edges in multi-phase traces. The full constraint list appears in Appendix B . All 750 traces yielded valid graphs (100% extraction rate). Mean statistics: 47.0\pm 12.4 nodes, 58.7\pm 15.2 edges, mean edge-to-node ratio 1.26 , discriminating edges in 76% of graphs. Because GPT-5.4 was both an evaluated model and the extraction model, we performed additional checks for source-model bias. Extraction summary statistics did not differ across source models and a subset of traces was independently re-extracted using Claude Opus 4.7. To assess extraction fidelity, the author, a physician, reviewed all 30 sampled graphs against their source traces. In this sample, edge precision was 98.4% (1,754/1,783 edges correct; range 95–100%). Clinically salient recall, assessed by identifying all explicit reasoning relationships in each source text, was 94.8% (range 78-97%; baseline 92.8%, adversarial 95.5%, structured 96.2%). The author did not identify hallucinated edges or

Read the complete text