Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPA
Zhusi Zhong, Yuli Wang, Lulu Bi, Zhuoqi Ma, Sun Ho Ahn, Christopher J. Mullin, Colin F. Greineder, Michael K. Atalay, Scott Collins, Grayson L. Baird, Cheng Ting Lin, Webster Stayman, Todd M. Kolb, Ihab Kamel, Harrison X. Bai, Zhicheng Jiao · 2025
Read Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPA by Zhusi Zhong, Yuli Wang, Lulu Bi, Zhuoqi Ma, Sun Ho Ahn, Christopher J. Mullin, Colin F. Greineder, Michael K. Atalay, Scott Collins, Grayson L. Baird, Cheng Ting Lin, Webster Stayman, Todd M. Kolb, Ihab Kamel, Harrison X. Bai, Zhicheng Jiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02034.txt.
Introduction
Zhusi Zhong and a large team build a model that reads chest scans for lung artery clots and drafts the written report, in a paper from 2025. Such scans are a key test for that clot, yet reading them and writing full reports is hard and slow.
Their model, Abn-BLIP, aligns image finds with words for each flaw it spots, guided by learned queries and cross links between image and text. In tests it beat top medical image text models and three dimensional report tools on both find rates and report worth. The authors share the code in the open.
Catalog record
| CREATOR | Zhusi Zhong, Yuli Wang, Lulu Bi, Zhuoqi Ma, Sun Ho Ahn, Christopher J. Mullin, Colin F. Greineder, Michael K. Atalay, Scott Collins, Grayson L. Baird, Cheng Ting Lin, Webster Stayman, Todd M. Kolb, Ihab Kamel, Harrison X. Bai, Zhicheng Jiao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPA Zhusi Zhong, Yuli Wang, Lulu Bi, Zhuoqi Ma, Sun Ho Ahn, Christopher J. Mullin, Colin F. Greineder, Michael K. Atalay, Scott Collins, Grayson L. Baird, Cheng Ting Lin, Webster Stayman, Todd M. Kolb, Ihab Kamel, Harrison X. Bai, Zhicheng Jiao Abstract Medical imaging plays a pivotal role in modern healthcare, with computed tomography pulmonary angiography (CTPA) being a critical tool for diagnosing pulmonary embolism and other thoracic conditions. However, the complexity of interpreting CTPA scans and generating accurate radiology reports remains a significant challenge. This paper introduces Abn-BLIP (Abnormality-aligned Bootstrapping Language-Image Pretraining), an advanced diagnosis model designed to align abnormal findings to generate the accuracy and comprehensiveness of radiology reports. By leveraging learnable queries and cross-modal attention mechanisms, our model demonstrates superior performance in detecting abnormalities, reducing missed findings, and generating structured reports compared to existing methods. Our experiments show that Abn-BLIP outperforms state-of-the-art medical vision-language models and 3D report generation methods in both accuracy and clinical relevance. These results highlight the potential of integrating multimodal learning strategies for improving radiology reporting. The source code is available at https://github.com/zzs95/abn-blip. Journal: Medical Image Analysis Fig. 1: Abn-BLIP inference pipeline for CTPA abnormality identification and structured report generation. The Abn-IDed image encoder detects 32 CTPA abnormalities and extracts abnormality-identified features. The learned visual queries interrogate CTPA scans by Abn-QFormer to extract the corresponding abnormal findings. These queries help generate a structured CTPA report, categorizing abnormalities under relevant organ-specific sections, such as pulmonary arteries and the heart. 1 Introduction Pulmonary embolism (PE) is a life-threatening condition caused by thromboembolic obstruction of the pulmonary arteries, often leading to severe complications, long-term morbidity, and high mortality risk ( Bĕlohlávek et al., 2013 ) . Timely and accurate diagnosis is essential for effective treatment and improved outcomes ( Alonso-Martínez et al., 2010 ; Hendriksen et al., 2017 ; Cahan et al., 2023 ) . Computed tomography pulmonary angiography (CTPA) remains the reference standard owing to its high sensitivity and specificity ( Stein et al., 2006 ) . However, its interpretation is labor-intensive, reader-dependent, and prone to delays in high-volume clinical environments ( Singh et al., 2011 ) . Recent advances in medical image AI have demonstrated considerable promise in enhancing PE diagnosis on CTPA ( Soffer et al., 2021 ) . Deep learning–based multimodal approaches have been developed to automate embolus identification, quantify clot burden, and stratify patient risk ( Huang et al., 2020a ; Liu et al., 2020 ; Zhong et al., 2025a ) , thereby improving efficiency and reducing inter-reader variability. Nonetheless, most existing systems primarily generate probabilistic predictions with limited interpretability, constraining their clinical reliability ( Huang et al., 2020a ; Huang et al., 2020b ; Lindenmeyer et al., 2024 ) . Efforts to incorporate vascular spatial structure ( Tajbakhsh et al., 2019 ) or artery segmentation with threshold-based analysis ( Pu et al., 2023 ) have improved embolus characterization, yet current solutions remain largely restricted to embolism detection rather than providing comprehensive CTPA assessment, including cardiac function, clot distribution, and ancillary thoracic findings. Vision–language models (VLMs) represent a promising direction for comprehensive CTPA-based PE assessment by integrating imaging with textual descriptions, thereby enhancing interpretability and decision support ( Wu et al., 2025 ; Zhong et al., 2025b ) . Medical VLMs bridge AI-generated outputs with radiologists’ workflows, facilitate automated structured report generation, and reduce inter-observer variability ( Nazi and Peng, 2024 ; Hartsock and Rasool, 2024 ; Tanno et al., 2024 ; Jin et al., 2024 ) . By incorporating multimodal information, such as clinical scores and patient history, VLMs enable holistic evaluation for improved patient management and risk stratification ( Zhong et al., 2024 ) . Unlike conventional models limited to classification or segmentation, VLMs generate comprehensive, human-readable reports directly from imaging data, thereby enhancing transparency and clinical adoption ( Wu et al., 2023 ; Bai et al., 2024 ; Huang et al., 2023 ) . Despite these advantages, general-purpose medical VLMs remain suboptimal for CTPA-based PE assessment ( Hager et al., 2024 ; Zhong et al., 2025b ) . Trained on heterogeneous datasets spanning diverse modalities, these models often lack domain-specific expertise, resulting in reduced sensitivity to subtle radiological findings critical for PE diagnosis. Their ability to address complex reports and multi-abnormality queries is also limited, constraining integration of visual, textual, and clinical information at a level comparable to expert radiologists ( Hartsock and Rasool, 2024 ) . The key challenge is to develop a PE-specific VLM that combines high diagnostic accuracy with interpretability, alignment with radiologists’ reporting conventions, and effective multimodal integration for comprehensive decision support. To address this gap, we propose Abnormality-aligned Bootstrapping Language–Image Pretraining (Abn-BLIP), a PE-specific VLM that integrates abnormality recognition with structured descriptions for CTPA report generation (Fig. 1 ). Abn-BLIP structures abnormality-specific visual queries into organized diagnostic findings, enabling a multi-stage workflow that enhances interpretability, systematically organizes assessments, and improves the clinical utility of AI-generated radiology reports. The key contributions are presented as follows: 1. We propose a multi-label abnormality recognition module to enhance diagnostic accuracy in CTPA report generation, with a particular focus on hierarchical analysis of the pulmonary artery region. 2. We introduce Abn-QFormer, which leverages abnormality-driven queries to aggregate image–text features at the abnormality level, dynamically refining cross-modal retrieval and enabling clinician-like examination of individual findings. 3. We develop Abnormality-aligned Contrastive Learning (ACL) to achieve fine-grained alignment between radiological features and textual findings, thereby reinforcing abnormality-level correspondence. 4. Guided by medical diagnostic principles, our framework explicitly models hierarchical relationships between anatomical regions and abnormalities, ensuring comprehensive, structured, and clinically meaningful CTPA reporting. Fig. 2: The figure illustrates the population distribution of 32 CTPA abnormalities across two datasets (BUH and INSPECT), categorized into 7 anatomical regions: Pulmonary Arteries, Lungs and Airways, Pleura, Heart, Mediastinum and Hila, Chest Wall and Lower Neck, and Bones. This hierarchical framework facilitates comprehensive abnormality detection and enhances the generation of clinically meaningful CTPA reports. The abnormality labels were extracted from radiology reports using a large language model (LLM), enabling a multi-dimensional assessment of inter-regional variations across the datasets. 2 Related work Image Captioning. Early image captioning methods predominantly adopted an encoder–decoder framework, in which convolutional neural networks (CNNs) extracted global visual features and recurrent neural networks (RNNs) generated sequential text ( Vinyals et al., 2015 ; Chen et al., 2015 ) . The introduction of attention mechanisms substantially improved performance by adaptively weighting salient image regions, thereby overcoming the limitations of fixed-length representations ( Xu, 2015 ; Lu et al., 2017 ) . Transformer-based architectures ( Vaswani, 2017 ) have been employed to capture long-range dependencies and support parallel decoding, resulting in more fluent and contextually consistent descriptions. Vision-Language Pre-training. Contrastive learning has become a foundational technique in Vision-Language Pre-training, aligning paired images and text within a shared embedding space to enable generalization across tasks. Models such as CLIP ( Radford et al., 2021 ) and SimCLR ( Khosla et al., 2020 ) optimize InfoNCE losses ( Oord et al., 2018 ) to learn discriminative representations. Despite impressive zero-shot capabilities, these approaches often neglect intra-modal structures and lack mechanisms for aligning localized visual semantics with fine-grained textual descriptions—an essential requirement in clinical scenarios