Radiologist-Guided Causal Concept Bottleneck Models for Chest X-Ray Interpretation

Amy Rafferty, Rishi Ramaesh, Ajitha Rajan · 2026

Read Radiologist-Guided Causal Concept Bottleneck Models for Chest X-Ray Interpretation by Amy Rafferty, Rishi Ramaesh, Ajitha Rajan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07785.txt.

Introduction

Amy Rafferty and two co-authors make chest X-ray models reason the way radiologists do. Concept bottleneck models predict clinical concepts before a diagnosis, which aids reading, but most treat concepts as mere clues. Real clinical reasoning runs the other way: diseases produce the visible findings on the film.

Their model, XpertCausal, encodes that direction with a noisy-OR structure linking diseases to concepts, then runs it backward, Bayesian-style, to score diseases from predicted concepts. Expert-curated links constrain the model to plausible clinical pathways. On the MIMIC-CXR dataset it beat a non-causal baseline and a freer causal ablation on accuracy, calibration, and explanation quality. Its learned links also aligned more closely with expert knowledge.

Catalog record

CREATORAmy Rafferty, Rishi Ramaesh, Ajitha Rajan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07785

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Radiologist-Guided Causal Concept Bottleneck Models for Chest X-Ray Interpretation Amy Rafferty, Rishi Ramaesh, Ajitha Rajan Abstract Concept Bottleneck Models (CBMs) in medical imaging aim to improve model interpretability by predicting intermediate clinical concepts before final diagnoses. However, most existing CBMs treat concepts as discriminative predictors of pathology labels, without explicitly modelling the underlying clinical generative process where diseases produce observable radiographic findings. We propose XpertCausal, a radiologist-guided causal CBM for chest X-ray interpretation which models pathology-to-concept relationships using a probabilistic noisy-OR framework. This generative model is then inverted via Bayesian inference to estimate pathology probabilities from predicted concepts. Radiologist-curated concept-pathology associations are used to constrain model structure to radiologist-defined clinically plausible reasoning pathways. We evaluate XpertCausal on MIMIC-CXR across pathology classification performance, calibration, explanation quality, and alignment with radiologist-defined reasoning pathways. Compared with both a non-causal CBM baseline and a causal ablation with unconstrained learned associations, XpertCausal achieves improved AUROC, calibration, and clinically relevant explanation quality, while learning concept–pathology relationships that more closely align with expert knowledge. These results demonstrate that incorporating clinically motivated causal structure and expert domain knowledge into CBMs can lead to more accurate, interpretable, and clinically aligned models for CXR interpretation. 1 Introduction Chest X-rays (CXRs) are one of the most commonly performed medical imaging examinations worldwide, and serve as a first-line diagnostic tool in the diagnosis of thoracic diseases such as pneumonia and lung cancer. Recent advances in deep learning have led to significant improvements in automated CXR interpretation, with convolutional neural networks achieving expert-level performance on a range of pathology classification tasks Singh et al. 2024 ; Rodman et al. 2023 . However, despite strong predictive performance, most existing approaches operate as black-box models, providing limited insight into the reasoning processes underlying their predictions. This lack of transparency remains a major barrier to clinical deployment, particularly in high-stakes clinical settings where interpretability and trust are essential, and incorrect predictions can lead to serious patient harm Wadden 2022 ; Yan et al. 2023 . Concept Bottleneck Models (CBMs) have emerged as a promising approach for improving interpretability in deep learning systems Koh et al. 2020 . Rather than directly predicting labels from images, CBMs first predict a set of human-interpretable concepts before using these concepts to produce final predictions. In the context of CXR interpretation, these concepts are typically observable image features, such as areas of opacity or heart enlargement. This intermediate concept layer provides explanations of model predictions in clinically meaningful terms, allowing them to be inspected and validated by end users (i.e., clinicians). CBMs have increasingly been explored for medical imaging tasks Mpinda et al. 2026 ; Xu et al. 2026 . However, their application is often constrained by the limited availability of large-scale datasets with ground truth concept-level annotations. Obtaining these annotations from radiologists is expensive, time-consuming, and difficult to scale, particularly for complex medical imaging datasets. As a result, many recent approaches instead rely on weakly supervised, automatically generated, or latent concepts Oikarinen et al. 2023 ; Yuksekgonul et al. 2023 ; Hu et al. 2025 , including attention-based concept discovery methods Alukaev et al. 2023 and large language model (LLM)-based semantic representations Yang et al. 2023 . While such approaches can improve interpretability compared with fully black-box models, their learned concepts are often not explicitly clinically grounded, semantically stable, or aligned with meaningful diagnostic reasoning processes Mahinpei et al. 2021 ; Rafferty et al. 2025 . In response to this, XpertXAI Rafferty et al. 2026 recently demonstrated that clinically grounded concepts can instead be extracted directly from radiology reports using radiologist-defined phrase mappings. By leveraging expert-defined findings rather than learned latent representations, XpertXAI provided a concept bottleneck framework whose intermediate reasoning process aligns more closely with clinically meaningful radiographic observations. Despite these advantages, most existing CBMs are fundamentally discriminative, where concepts are typically treated as predictive features used to infer pathology labels Koh et al. 2020 ; Zarlenga et al. 2022 . In real-world clinical reasoning, however, the underlying disease generation process proceeds in the opposite direction: pathologies instead cause radiographic findings, which clinicians then interpret to infer diagnoses Shwe et al. 1990 . Ignoring this generative structure may encourage models to learn spurious associations between concepts and labels, particularly in biased or noisy medical datasets Geirhos et al. 2020 ; Prosperi et al. 2020 . Another limitation is that many concept-based models treat concepts as independent predictive variables, overlooking the structured dependencies that naturally exist between diseases and their observable manifestations. Recent work has explored incorporating causal structure into concept-based learning through causal and structured CBMs Dominici et al. 2025 ; Felice et al. 2026 . Unlike standard CBMs, which typically treat concepts as independent features, these approaches model dependencies between concepts and downstream predictions using directed graphical structures, structural causal models (SCMs), or probabilistic generative frameworks Schölkopf et al. 2021 in order to improve explanation faithfulness, robustness, and alignment between model reasoning and human-understandable concepts Reinhold et al. 2021 ; Bing et al. 2026 ; Ismail et al. 2024 . However, many existing approaches either infer these relationships directly from observational data or evaluate primarily on simplified benchmark settings with synthetic or non-clinical concepts. In medical imaging, these learned relationships may instead capture spurious correlations or clinically implausible dependencies rather than real diagnostic reasoning pathways. Probabilistic graphical models have a long history in computer-assisted medical diagnosis. Early systems such as INTERNIST-1 and Quick Medical Reference (QMR) explored structured representations of disease–finding relationships for diagnostic reasoning Miller et al. 1982 ; Miller and Masarie 1989 , while probabilistic alternatives such as QMR-DT modelled diseases and clinical findings using Bayesian networks and noisy-OR relationships Shwe et al. 1990 . In these systems, latent diseases generated observable findings, and posterior disease probabilities were inferred using Bayesian reasoning. Although highly interpretable, these approaches predated modern deep learning and therefore lacked the ability to learn visual representations directly from medical images. In this work, we propose XpertCausal , a radiologist-guided causal CBM for chest X-ray interpretation that combines interpretable concept bottlenecks with probabilistic causal disease modelling. XpertCausal models pathology-to-concept relationships using a noisy-OR generative framework constrained by radiologist-defined concept-pathology associations. Rather than treating concepts solely as predictive features (Concept \rightarrow Pathology), the model explicitly represents how latent diseases generate observable radiographic findings (Pathology \rightarrow Concept). Final pathology predictions are then obtained through Bayesian inference over this generative framework. By incorporating expert-defined structural constraints, XpertCausal encourages clinically grounded reasoning while reducing implausible concept-pathology associations which may be learned from data alone. We evaluate XpertCausal on the MIMIC-CXR dataset Johnson et al. 2019a ; Goldberger et al. 2000 ; Johnson et al. 2019b across pathology classification performance, calibration, explanation quality, and alignment with radiologist-defined reasoning pathways. We compare against both a non-causal expert-driven CBM baseline (XpertXAI Rafferty et al. 2026 ) and a causal ablation model with unconstrained learned associations. Our results suggest that incorporating both causal structure and expert-guided constraints improves predictive performance, calibration, and the clinical relevance of model explanations. The main contributions of this work are as follows: • We propose XpertCausal , a radiologist-guided causal CBM for chest X-ray interpretation which combines interpretable concept bottlenecks with probabilistic causal disease modelling using a noisy-OR generative framework and Bayesian inference. • We incorporate

Read the complete text