Towards Interpretable Counterfactual Generation via Multimodal Autoregression

Chenglong Ma † \dagger, Yuanfeng Ji † \dagger, Jin Ye, Lu Zhang, Ying Chen, Tianbin Li, Mingjie Li, Junjun He *, Hongming Shan * · 2025

Read Towards Interpretable Counterfactual Generation via Multimodal Autoregression by Chenglong Ma † \dagger, Yuanfeng Ji † \dagger, Jin Ye, Lu Zhang, Ying Chen, Tianbin Li, Mingjie Li, Junjun He *, Hongming Shan * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23149.txt.

Introduction

Chenglong Ma and co-authors set a new task for medical AI: imagine how a chest scan would look if a disease progressed, and explain the change in words. Silent image predictions leave doctors unable to check the reasoning. Their dataset pairs repeat scans with what-if prompts and written interpretations of the expected changes.

Their model, ProgEmu, generates the future image and its explanation in one autoregressive stream. It beat rivals at matching the stated progression. The authors see uses in decision support and in training clinicians to foresee disease courses.

Catalog record

CREATORChenglong Ma † \dagger, Yuanfeng Ji † \dagger, Jin Ye, Lu Zhang, Ying Chen, Tianbin Li, Mingjie Li, Junjun He *, Hongming Shan *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Interpretable Counterfactual Generation via Multimodal Autoregression Chenglong Ma † \dagger, Yuanfeng Ji † \dagger, Jin Ye, Lu Zhang, Ying Chen, Tianbin Li, Mingjie Li, Junjun He *, Hongming Shan * Abstract Counterfactual medical image generation enables clinicians to explore clinical hypotheses, such as predicting disease progression, facilitating their decision-making. While existing methods can generate visually plausible images from disease progression prompts, they produce silent predictions that lack interpretation to verify how the generation reflects the hypothesized progression—a critical gap for medical applications that require traceable reasoning. In this paper, we propose I nterpretable C ounterfactual G eneration (ICG), a novel task requiring the joint generation of counterfactual images that reflect the clinical hypothesis and interpretation texts that outline the visual changes induced by the hypothesis. To enable ICG, we present ICG-CXR, the first dataset pairing longitudinal medical images with hypothetical progression prompts and textual interpretations. We further introduce ProgEmu, an autoregressive model that unifies the generation of counterfactual images and textual interpretations. We demonstrate the superiority of ProgEmu in generating progression-aligned counterfactuals and interpretations, showing significant potential in enhancing clinical decision support and medical education. Project page: https://progemu.github.io . † † footnotetext: † Equal contribution. ∗ Co-corresponding authors. 1 Introduction Counterfactual generation, which aims to synthesize hypothetical “what-if” scenarios based on real data, has emerged as a transformative tool for medical image analysis [ 4 , 15 ] . By generating realistic counterfactual images from a patient’s scan under various clinical scenarios ( e.g. , disease progression and regression), clinicians can visualize potential outcomes and customize patient-specific interventions, thereby enhancing decision-making and medical education. Recent advances in generative models have shown promising results in counterfactual medical image generation, particularly in chest X-ray (CXR) analysis [ 4 , 11 , 18 ] . Although these models can generate visually plausible CXR images from input images and disease progression prompts ( e.g. , “New small left pleural effusion”), their clinical application is limited by what we call silent predictions. In other words, the models produce counterfactual images without providing accompanying texts that explain the specific radiological changes associated with the disease progression ( e.g. , “The left hemidiaphragm shows new blunting in the posterior pleural sulcus, indicating a new small left pleural effusion”). Consequently, these models fail to establish a transparent, causal chain to validate the counterfactual generation, undermining their utility in educational or decision-support applications that require traceable reasoning. To this end, we introduce I nterpretable C ounterfactual G eneration (ICG), a novel task that requires the joint generation of a counterfactual image reflecting the input hypothesis and an interpretation text justifying the generated changes, thereby ensuring causal alignment between the produced images and texts. However, ICG presents two key challenges, which can be summarized as follows: (a) data scarcity –publicly available datasets rarely include longitudinal studies with paired interpretation for the changes observed between prior and subsequent images, and (b) disjointed modeling –existing solutions typically isolate image and text generation [ 11 , 14 , 17 , 24 ] , hindering effective modeling of vision-language dependencies. In this paper, we tackle ICG through two key innovations. First, we focus on CXR modality and curate the ICG-CXR dataset using a GPT4-based pipeline (Fig. 1 ). To our knowledge, ICG-CXR is the first large-scale longitudinal CXR dataset designed for interpretable counterfactual generation. It contains 11,439 quadruples from 7,388 unique patients, each including a prior image, a subsequent image, a progression prompt, and a corresponding textual interpretation of the observed changes. Second, we develop ProgEmu, a multimodal autoregressive model that jointly generates counterfactual images and their corresponding textual interpretations (Fig. 2 ). By unifying these modalities, ProgEmu ensures that the generated image aligns with the hypothesized progression while the text outlines the underlying radiological changes. Our contributions can be summarized as follows: • We formalize ICG as a novel task requiring the joint generation of counterfactual images and interpretation texts. • We curate the ICG-CXR dataset, addressing the critical data scarcity for the ICG task. • We introduce ProgEmu, a multimodal autoregressive model that unifies image and text generation for interpretable counterfactual generation. • Quantitative and qualitative results show that ProgEmu achieves state-of-the-art generation of progression-aligned counterfactual images and interpretation texts, highlighting its potential for counterfactual medical analysis. 2 Related Work Counterfactual Medical Image Generation. Counterfactual medical image generation shows significant potential for simulating hypothetical disease scenarios [ 1 , 11 , 14 , 18 , 24 ] and improving the interpretability of AI diagnostic models [ 7 , 9 , 10 , 20 ] . BiomedJourney [ 11 ] frames the counterfactual image generation problem as an image-editing task and adapts an image-editing framework [ 3 ] to patient journey data for generating counterfactual images. PIE [ 18 ] uses diffusion-based [ 23 ] progressive image editing to simulate realistic disease progression, although it relies on external region masks and does not explain the generated images. CXR-IRGen [ 24 ] combines a diffusion model with a large language model [ 16 ] to produce paired CXR images and radiology reports simultaneously, yet the text only describes the counterfactual image without highlighting differences between images. In contrast, our ProgEmu unifies the generation of counterfactual images and interpretation texts within a single model. Unified Multimodal Generative Models. Recent breakthroughs in generative AI have enabled single model to generate various data types ( e.g. , image and text) simultaneously [ 6 , 26 , 27 ] , achieving superior performance on multiple tasks. Chameleon [ 26 ] discretizes visual and textual data into tokens and trains a generic multimodal transformer from scratch in an autoregressive fashion, enabling seamless generation of images and texts. Emu3 [ 27 ] extends this approach and enhances the generation of images, texts, and videos by incorporating a more powerful vision tokenizer and an improved post-training strategy. Following this line of work, we propose an autoregressive model, ProgEmu, that leverages the strengths of unified vision-language processing and autoregressive modeling, making it naturally suitable for the proposed interpretable counterfactual generation task. 3 Methodology For a consecutive study pair, given a prior image and a disease progression prompt, the ICG task aims to generate a progression-aligned counterfactual image and an interpretation text describing the progression-induced radiological changes between prior and counterfactual images. We address this task by first curating the ICG-CXR dataset and then developing an autoregressive multimodal generative model, ProgEmu, detailed below. 3.1 ICG-CXR Dataset Curation CXR image-report datasets like MIMIC-CXR [ 13 ] (377,110 pairs from 227,827 studies) and CheXpertPlus [ 5 ] (223,228 pairs from 187,711 studies) are widely used in medical imaging research. However, these datasets are not directly applicable for the interpretable counterfactual generation task, because they lack comparative descriptions between two CXR studies, and a non-trivial number of studies are of low quality, requiring further processing. To this end, we curate ICG-CXR, a longitudinal dataset for the interpretable counterfactual generation task, derived from MIMIC-CXR and CheXpertPlus. We start the curation by keeping only those with posterior-anterior view images and discarding any with empty “Impression” or “Findings” sections. For each patient, we form study pairs by selecting a prior CXR study and a subsequent one conducted within 100 days to minimize unrelated confounding factors, and then resize

Read the complete text