CoCa-CXR: Co ntrastive Ca ptioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
Yixiong Chen 1, Shawn Xu 2, Andrew Sellergren 2, Yossi Matias 2, Avinatan Hassidim 2, Shravya Shetty 2, Daniel Golden 2, Alan Yuille 1, Lin Yang 2 · 2025
Read CoCa-CXR: Co ntrastive Ca ptioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding by Yixiong Chen 1, Shawn Xu 2, Andrew Sellergren 2, Yossi Matias 2, Avinatan Hassidim 2, Shravya Shetty 2, Daniel Golden 2, Alan Yuille 1, Lin Yang 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20509.txt.
Introduction
Yixiong Chen and co-authors teach an AI system to read chest X-rays across time in a 2025 medical imaging paper. Radiologists constantly compare a new scan with the patient's last one, and their reports say so. Most vision-language models, though, train on single images. The team's pipeline first splits reports into description and comparison parts with a language model, mining fine labels about change.
Their model, CoCa-CXR, then learns both to caption a scan and to describe how it changed, using a regional attention module that spotlights where paired images differ. Tests showed strong gains in temporal understanding. Reading change over time is where chest imaging earns its keep, so the focus is well aimed.
Catalog record
| CREATOR | Yixiong Chen 1, Shawn Xu 2, Andrew Sellergren 2, Yossi Matias 2, Avinatan Hassidim 2, Shravya Shetty 2, Daniel Golden 2, Alan Yuille 1, Lin Yang 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoCa-CXR: Co ntrastive Ca ptioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding Yixiong Chen 1, Shawn Xu 2, Andrew Sellergren 2, Yossi Matias 2, Avinatan Hassidim 2, Shravya Shetty 2, Daniel Golden 2, Alan Yuille 1, Lin Yang 2 Abstract Vision-language models have proven to be of great benefit for medical image analysis since they learn rich semantics from both images and reports. Prior efforts have focused on better alignment of image and text representations to enhance image understanding. However, though explicit reference to a prior image is common in Chest X-Ray (CXR) reports, aligning progression descriptions with the semantics differences in image pairs remains under-explored. In this work, we propose two components to address this issue. (1) A CXR report processing pipeline to extract temporal structure. It processes reports with a large language model (LLM) to separate the description and comparison contexts, and extracts fine-grained annotations from reports. (2) A contrastive captioner model for CXR, namely CoCa-CXR, to learn how to both describe images and their temporal progressions. CoCa-CXR incorporates a novel regional cross-attention module to identify local differences between paired CXR images. Extensive experiments show the superiority of CoCa-CXR on both progression analysis and report generation compared to previous methods. Notably, on MS-CXR-T progression classification, CoCa-CXR obtains 65.0% average testing accuracy on five pulmonary conditions, outperforming the previous state-of-the-art (SOTA) model BioViL-T by 4.8%. It also achieves a RadGraph F1 of 24.2% on MIMIC-CXR, which is comparable to the Med-Gemini foundation model. 1 Introduction Recent advances in vision-language (VL) pre-training have significantly enhanced the development of flexible and powerful models for chest X-ray (CXR) analysis. Training on both images and reports allows models to capture rich semantics, aligning medical concepts with image representations. However, CXR reports frequently include comparisons between multiple examinations [ 13 ] , a crucial aspect often overlooked by existing approaches [ 11 , 4 , 19 , 21 ] . Most prior work focuses on predicting findings from a single image without explicitly modeling temporal progression, limiting their ability to understand disease evolution and restricting their clinical applicability. Recent literature [ 2 , 7 , 1 , 23 , 26 ] has began to utilize the temporal structure of the reports to train VL models that can make predictions based on multiple CXR images. The common practice is to fuse the representations of images as the joint representation to generate reports. While straightforward, this strategy does not explicitly guide models to learn temporal differences. The key challenges remain: (1) the lack of datasets with aligned image pairs and corresponding comparative descriptions, and (2) the need for model architectures that can effectively capture subtle regional changes over time. Figure 1: CoCa-CXR adds a regional cross-attention module to CoCa and is trained with three stages. We utilize an LLM, Gemini, to parse MIMIC-CXR reports and get image description and pair comparison. Chest ImaGenome scene graphs provide us with the local comparison between CXR image pairs. To leverage the locality of disease conditions, we apply cross-attention to emphasize the correlation between neighboring tokens of two images. We propose a systematic framework (Fig. 1 ) to address the above problems. For (1), we build a CXR report processing pipeline to curate a new dataset, CXR-4, with four sub-datasets, detailed in Tab. 1 . It uses a large language model (LLM) [ 20 ] to separate reports into descriptions and comparison components, allowing the model to learn them sequentially. In addition, we also leverage the comparison and localization from scene graphs [ 25 ] which contain the description of the abnormal organs, condition progressions, and the corresponding bounding boxes, enabling the models to learn regional differences. For (2), to leverage the regional comparison in the CXR-4 dataset, we propose a regional cross-attention module, inspired by [ 6 ] , but specifically designed for cross-attention between current and prior images. This module refines traditional cross-attention [ 22 ] by restricting each token in an image to attend only to its spatially corresponding tokens in the prior image. We choose Contrastive Captioner (CoCa) [ 28 ] as our baseline model. It has a minimalist architecture to perform VL contrastive and generative learning together, which are essential for aligned representation and downstream multitasking. In summary, our work presents three major contributions to temporal CXR understanding. 1) We introduce the CXR-4 dataset. It provides not only the alignment between CXR images and text descriptions, but also explicit comparison. 2) We propose CoCa-CXR, which is a CoCa-based model that can generate reports from image pairs, predicting condition progressions, and localizing abnormal organs. 3) Experiments on both progression prediction and report generation tasks show the superior performance of CoCa-CXR to previous SOTA CXR temporal models. Table 1: Statistics of the sub-datasets used for training CoCa-CXR. Sub-Dataset Stage Description Data Source #Samples 1. Clean image-report pair 1&2&3 Image-report pairs without any image pair comparison content. MIMIC-CXR 224,487 2. Image pair & filtered report 2&3 Image pair with corresponding report. All reports must contain comparison info. MIMIC-CXR 132,320 3. Image pair & comparison info 2&3 Image pair with corresponding comparison sentences from the report. MIMIC-CXR 259,562 4. Image pair & abnormal organs 2&3 Image pair with abnormal organ’s condition, coordinates, and progression. MIMIC-CXR & Chest ImaGenome 758,344 2 CXR-4 Dataset We introduce CXR-4, a new CXR dataset comprising four sub-datasets (Tab. 1 ), built from MIMIC-CXR [ 10 , 13 , 12 ] images, reports, and Chest ImaGenome scene graphs [ 25 , 24 ] . The dataset follows the official MIMIC-CXR split, excluding the MS-CXR-T [ 2 , 3 ] test set. We develop a report processing pipeline (Fig. 1 ) to extract structured information. Example prompts for Gemini [ 20 ] in-context learning and sample illustrations are provided in Fig. 4 (supplementary material). Below, we detail the sub-datasets. 1. Clean image-report pairs. To align VL representations, we pair MIMIC-CXR images with their radiological reports. We retain only AP/PA view scans and use Gemini to filter reports, keeping only view information, FINDINGS, and IMPRESSION sections while removing comparative descriptions. 2. Image pairs & filtered reports. This subset contains samples with explicit comparison. Each sample consists of an image, its most recent prior image, and its FINDINGS and IMPRESSION sections as textual descriptions. 3. Image pairs & comparative descriptions. Building on (2), this subset includes sentences explicitly describing progression, extracted via Gemini. To augment the data, we reverse image pairs and modify text descriptions accordingly (e.g., “improved pneumonia” \rightarrow “worsened pneumonia”). 4. Image pair & abnormal organs. Using Chest ImaGenome, we extract structured comparative descriptions, providing localized abnormality progression. The format follows: "[condition] [progress] at [organ], coordinates for current image is [x_{cur,1},x_{cur,2} , y_{cur,1},y_{cur,2}] , coordinates for previous image is [x_{prior,1},x_{prior,2} , y_{prior,1},y_{prior,2}] ." As additional augmentation, we reverse image pairs and adjust descriptions accordingly. 3 CoCa-CXR 3.1 Contrastive Captioners for CXR Image Pair Understanding CoCa [ 28 ] (top left in Fig. 1 ) encodes images and text into latent representations using a Vision Transformer (ViT) [ 8 ] and a unimodal text encoder. A transformer text decoder then cross-attends to image features to generate captions. The model is trained with a multi-modal contrastive loss: \mathcal{L}_{\text{Con}}=-\frac{1}{N}\left(\sum_{i=1}^{N}\log\frac{\exp(x_{i}^{\top}y_{i}/\tau)}{\sum_{j=1}^{N}\exp(x_{i}^{\top}y_{j}/\tau)}+\sum_{i=1}^{N}\log\frac{\exp(y_{i}^{\top}x_{i}/\tau)}{\sum_{j=1}^{N}\exp(y_{i}^{\top}x_{j}/\tau)}\right) where x_{i} and y_{i} are normalized image and