CF-Seg: Counterfactuals meet Segmentation

Raghav Mehta, Fabio De Sousa Ribeiro, Tian Xia, Mélanie Roschewitz, Ainkaran Santhirasekaram, Dominic C. Marshall, Ben Glocker · 2025

Read CF-Seg: Counterfactuals meet Segmentation by Raghav Mehta, Fabio De Sousa Ribeiro, Tian Xia, Mélanie Roschewitz, Ainkaran Santhirasekaram, Dominic C. Marshall, Ben Glocker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16213.txt.

Introduction

Raghav Mehta and six co-authors use made-up healthy versions of chest scans to improve anatomy mapping in a paper from 2025. Software that outlines organs in medical images often fails when disease distorts the tissue. Their idea is to first generate a counterfactual image: the same chest as it would look with no disease, keeping the true anatomy in place.

The existing mapping software then runs on the cleaned-up image instead of the raw scan. Tests on two real chest X-ray data sets showed better outlines of anatomical structures. Better outlines can feed safer measures and choices in later care, the authors argue, with no change to the mapping model itself.

Catalog record

CREATORRaghav Mehta, Fabio De Sousa Ribeiro, Tian Xia, Mélanie Roschewitz, Ainkaran Santhirasekaram, Dominic C. Marshall, Ben Glocker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16213

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CF-Seg: Counterfactuals meet Segmentation Raghav Mehta, Fabio De Sousa Ribeiro, Tian Xia, Mélanie Roschewitz, Ainkaran Santhirasekaram, Dominic C. Marshall, Ben Glocker Abstract Segmenting anatomical structures in medical images plays an important role in the quantitative assessment of various diseases. However, accurate segmentation becomes significantly more challenging in the presence of disease. Disease patterns can alter the appearance of surrounding healthy tissues, introduce ambiguous boundaries, or even obscure critical anatomical structures. As such, segmentation models trained on real-world datasets may struggle to provide good anatomical segmentation, leading to potential misdiagnosis. In this paper, we generate counterfactual (CF) images to simulate how the same anatomy would appear in the absence of disease without altering the underlying structure. We then use these CF images to segment structures of interest, without requiring any changes to the underlying segmentation model. Our experiments on two real-world clinical chest X-ray datasets show that the use of counterfactual images improves anatomical segmentation, thereby aiding downstream clinical decision-making. 1 Introduction Anatomical segmentation plays an important role in medical imaging, as it enables precise identification and delineation of organs and tissues, which in turn plays a pivotal role in treatment planning. For example, accurate lung segmentation in chest X-ray (CXR) helps in identifying the progression of lung disease and monitoring response to therapy [ 11 , 14 ] . Recent advances in deep learning have enabled precise and accurate segmentation of anatomical structures [ 27 , 26 , 18 ] . However, these methods often struggle in the presence of disease, where abnormalities can obscure or alter these structures [ 26 ] . For example, pathologic states, such as pleural effusion, edema, tumour or pneumonia, can alter the appearance of the lungs, increasing their opacity and making lung segmentation more difficult. The ability to remove disease patterns from medical images while preserving anatomical structures is crucial for improving anatomical segmentation accuracy. In this work, we explore this possibility using recent advancements in counterfactual image generation models. Fundamentally, counterfactual (CF) images represent ‘what-if’ scenarios, such as: What would the patient’s chest X-ray look like if there was no pleural effusion? In this case, the underlying lung structure should remain unchanged, while the effusion would be removed. This would lead to a clearer anatomical representation, making lung segmentation easier for machine learning models (see Fig. 1 ). Motivated by this, we propose a framework that leverages counterfactual image generation models to produce pseudo-healthy images from diseased images and then apply pre-trained segmentors to these counterfactual images. Notably, our approach does not require re-training the segmentation model, making it directly compatible with any pretrained segmentator. Our main contributions can be summarized as: (i) We propose CF-Seg, a novel framework leveraging healthy CF images, to improve anatomical segmentation in the presence of disease during inference, without requiring any modifications to the underlying segmentation model. (ii) We conduct a user study with two expert radiologists – reviewing 300 images each from MIMIC-CXR and PadChest [ 13 , 3 ] – finding that experts prefer lung segmentation generated with CF-Seg in comparison to publicly available (silver standard) segmentation masks [ 7 ] . (iii) We collect “ground-truth” expert segmentation for 140 images from healthy subjects and subjects with pleural effusion and find that utilizing counterfactual images improves the lung segmentation performance substantially in the presence of effusion. Figure 1 : Effect of disease on lung segmentation (red - right lung, green - left lung). ( a-b) Healthy example: segmentation is relatively easy with high similarity between (a) expert segmentation and (b) automatic segmentation. (c-d) For a subject with pleural effusion, lungs are partially obscured, and segmentation is difficult with major differences between (c) expert segmentation and (d) automatic segmentation. Generating the counterfactual ‘pseudo-healthy’ image removes effusion without altering the underlying anatomy, on which (e) automatic segmentation becomes more similar to (c) expert segmentation. 2 Related Work To improve anatomical segmentation, one popular research avenue consists of leveraging anatomical knowledge by incorporating shape priors [ 21 , 17 ] or atlas-based segmentation [ 20 , 9 , 33 ] . However, these studies mainly focus on improving anatomical segmentation for healthy subjects without considering the effect of disease pathology on segmentation performance. On the other hand, in the context of pathology segmentation, methods have been proposed to either directly incorporate anatomical priors in neural networks [ 12 , 19 ] ; or use generative models to synthesize normal pseudo-healthy anatomy and use difference maps to localize and segment anomalies or pathology [ 31 , 35 , 2 ] . The later methods are closest to our work, as we also utilize pseudo-healthy images in our work. But in contrast to prior works, we specifically focus on anatomy structure segmentation in the presence of disease rather than on the segmentation of pathology . Various approaches have been proposed for pseudo-healthy image generation, including GANs [ 15 , 31 ] , VAEs [ 32 , 8 ] , and diffusion models [ 29 , 1 ] . However, they do not explicitly model the underlying causal structure of the generative process. In this work, we utilize deep structural causal models (DSCMs) that integrate causal structures with deep generative models [ 22 , 5 ] . Specifically, we use a hierarchical variational auto-encoder (HVAE) based generative model proposed by Riberio et al. [ 5 ] to generate pseudo-healthy counterfactual images, which we then utilize for downstream segmentation of anatomical structure. Note that in this paper, we do not focus on proposing a new generative model, but rather focus on using existing generative models to improve anatomical segmentation. Counterfactual images have been employed for various medical image analysis tasks and applications, such as data augmentation [ 10 , 34 ] , contrastive learning [ 28 ] , bias mitigation [ 16 ] , explainability [ 6 , 4 ] , and disease progression modeling [ 24 , 25 ] . However, their application for anatomical segmentation in the presence of disease pathology remains unexplored. 3 Methodology 3.1 Background on Deep Structural Causal Models A Structural Causal Model (SCM) [ 23 ] consists of a set of endogenous variables \mathbf{X}=\{X_{i}\}_{i=1}^{n} (e.g. medical scan, patient age, disease status etc), exogenous variables \mathbf{U}=\{U_{i}\}_{i=1}^{n} (unobserved influences), and a set of functions \mathbf{F}=\{f_{i}\}_{i=1}^{n} which define causal relationships (e.g. mechanism of disease). Each X_{i} is determined by its parents \mathbf{pa}_{i} (direct causes) and an exogenous variable U_{i} via a structural equation: X_{i}\coloneqq f_{i}(\mathbf{pa}_{i},U_{i}) . SCMs enable the estimation of counterfactuals , which represent hypothetical scenarios given observed evidence. For example, one may query “What would this patient’s scan look like if there was no disease?” . Counterfactual inference involves three steps: (i) Abduction: infer the posterior exogenous noise distribution given observed evidence P_{\mathbf{U}|\mathbf{X}} ; (ii) Action: intervene on one or more of the endogenous variables do(X_{i}\coloneqq x) , such as disease status, to obtain a modified SCM \mathcal{M}_{x} ; (iii) Prediction: use \mathcal{M}_{x} and P_{\mathbf{U}|\mathbf{X}} to generate a counterfactual. Deep SCMs [ 22 , 5 ] (DSCMs) and Neural Causal Models (NCMs) [ 30 ] provide a principled framework for using deep learning components in SCMs, thereby enabling tractable counterfactual inference of high-dimensional variables such as images. To estimate counterfactuals of chest X-rays, we here use the

Read the complete text