Cross-Modal Clinical Knowledge Integration for Mammography Report Generation

Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen · 2026

Read Cross-Modal Clinical Knowledge Integration for Mammography Report Generation by Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31093.txt.

Introduction

Zhu and colleagues build MammoRG, a breast report framework for the report workflow. Screening creates a workload for readers, making exact reports a key challenge. Old methods miss medical thought. MammoRG follows the BI-RADS rules and uses prior knowledge for reports. In a first stage, it mixes prior knowledge from scans through sorting guidance. In a second stage, fine-tuning models breast terms as units for reports. They also develop MammoRGTool, a tool for free-text reports. Tests show MammoRG beats methods across medical metrics, mainly BI-RADS scores.

Catalog record

CREATORJiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58223
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31093

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang, Zhixuan Chen, Yuan Guo, Qingcong Kong, Zhenhui Li, Qiong Luo, Hao Chen Abstract Breast cancer is a major global health concern, and mammography screening plays a central role in early detection. The large volume of screening examinations creates a substantial workload for radiologists, making accurate and consistent report generation a critical clinical challenge. Existing automated mammography report generation methods primarily focus on direct visual-to-text mapping, while overlooking the structured clinical reasoning process followed by radiologists in real-world practice. To address this limitation, we propose MammoRG, a mammography report generation framework that explicitly simulates the clinical reporting workflow by following the BI-RADS guideline and incorporating prior clinical knowledge to produce diagnostic reports. Specifically, MammoRG adopts a two-stage training framework. In the first stage, the model learns to integrate clinically relevant prior knowledge from a patient’s four-view mammograms through classification-based supervision. In the second stage, a terminology-aware supervised fine-tuning strategy is introduced to model mammography-specific clinical terms as atomic semantic units, enabling the generation of high-quality reports with improved clinical consistency. To facilitate clinical efficacy evaluation of generated reports, we further develop MammoRGTool, a dedicated mammography report parsing tool that extracts structured clinical information from free-text reports. Extensive experiments demonstrate that MammoRG consistently outperforms existing methods across multiple clinical efficacy metrics, particularly in diagnosis-related BI-RADS F1, where it surpasses the second-best model by 2.73%, 2.04%, 1.90%, and 3.27% on the internal, external 1, external 2, and VinDr-Mammo datasets, respectively. keywords Mammography,Report generation,Medical image analysis, † † credit: Conceptualized and designed the study, assisted with data processing, including data cleaning and post-processing, designed and developed MammoRGTool and MammoRG, designed and conducted all experiments, and wrote the manuscript † † credit: Assisted in conceptualizing and designing the study, and collected and processed the data † † credit: Collected and annotated the datasets from a hospital, and provided clinical assessment recommendations † † credit: Provided critical feedback on the manuscript and experimental design † † credit: Provided critical feedback on the manuscript and experimental design † † credit: Collected and annotated the datasets from a hospital † † credit: Collected and annotated the datasets from a hospital † † credit: Collected and annotated the datasets from a hospital † † credit: Provided critical feedback on the manuscript and experimental design, and supervised the research † † credit: Provided critical feedback on the manuscript and experimental design, and supervised the research † † corresponding: Corresponding author † † These authors contributed equally to this work. 1 Introduction Breast cancer is one of the most significant health challenges globally and a leading cause of cancer-related deaths among women, with millions of new cases diagnosed each year ( 3 ; 21 ) . Mammography stands as the most effective screening method for the early detection of breast cancer, and its widespread application has been proven to significantly reduce mortality rates ( 12 ; 19 ) . With the widespread implementation of breast cancer screening worldwide, the number of images that radiologists need to interpret has increased dramatically, imposing a substantial workload ( 1 ) . Radiology report generation is the task of automatically producing free-text descriptions of radiological images, aiming to provide comprehensive and structured textual summaries for given patient scans. Due to its great potential in alleviating radiologists’ workload, improving the efficiency of diagnostic workflows, and reducing human error, this field has attracted widespread attention in recent years, leading to the emergence of numerous report generation methods across various imaging modalities, including X‑ray ( 41 ) , CT ( 15 ) , MRI ( 14 ) , and histopathology ( 33 ) . However, despite the rapid advancement of report generation technology, existing general-purpose radiology report generation models cannot be directly adapted to mammography. Most existing methods are primarily designed for generic single-image visual-to-text generation and often fail to model the mammography-specific diagnostic reasoning workflow required in clinical practice. In particular, mammography report generation is inherently a multi-view diagnostic task that requires joint reasoning across bilateral breasts and multiple standardized views, including right craniocaudal (R-CC), right mediolateral oblique (R-MLO), left craniocaudal (L-CC), and left mediolateral oblique (L-MLO) views, to accurately characterize fine-grained findings and ensure consistent diagnostic assessment. Even powerful general models such as GPT-5 ( 30 ) are unable to effectively address this challenge (Fig. 1 (a)). Beyond describing findings, mammography reporting must further assess malignancy risk and assign appropriate Breast Imaging Reporting and Data System (BI-RADS) categories, which directly guide clinical decision-making ( 16 ) . Despite the clinical importance of mammography report generation, prior work remains limited, with few publicly reproducible models and benchmarks, hindering systematic validation and comparison ( 37 ; 6 ; 4 ) . Moreover, existing approaches—including general-purpose report generation models—primarily focus on direct visual-to-text mapping and often underutilize mammography-specific clinical priors (Fig. 1 (b)), such as BI-RADS–grounded assessment criteria and established reporting conventions. However, the clinical value of a mammography report depends not only on linguistic fluency, but more importantly on the correctness of clinically relevant findings and diagnostic assessments. Therefore, developing a robust and open-resourced mammography report generation framework that can effectively incorporate mammography-specific clinical knowledge is crucial for advancing this field toward clinical utility. Inspired by these challenges, we propose MammoRG, a two-stage training framework for mammography report generation that leverages cross-modal clinical knowledge integration and term-aware supervised fine-tuning to improve both linguistic quality and clinical diagnostic accuracy. Specifically, the proposed cross-modal clinical knowledge integration module incorporates clinically relevant prior knowledge from a report database and a mammography knowledge graph into the report generation process (Fig. 1 (c)), enabling better alignment between visual features and diagnostic semantic information. To address the inherent limitations of pretrained language models in tokenizing and semantically understanding professional terminology, we further expand the language model vocabulary with a mammography lexicon during the term-aware supervised fine-tuning stage. To facilitate clinical efficacy (CE) evaluation of generated reports, we additionally develop MammoRGTool, a dedicated mammography report parsing tool that extracts structured clinical information from free-text reports, including BI-RADS categories, tissue composition types, abnormal findings, and their associated anatomical locations, descriptive terms, and diagnostic suggestions. Experiments on an internal test set, two independent external test sets, and a publicly available external dataset demonstrate that MammoRG achieves state-of-the-art (SOTA) performance on both natural language generation (NLG) metrics and CE metrics. We summarize our contributions as follows: Figure 1 : Examples demonstrating limitations of general models in mammography reporting and the comparison between traditional uni-modal methods and our cross-modal method. • We propose MammoRG, a two-stage training framework that leverages cross-modal clinical knowledge integration and term-aware supervised fine-tuning, effectively enhancing both the linguistic fluency and diagnostic accuracy of generated reports. • We develop MammoRGTool, a specialized report parsing tool for evaluating mammography reports, which achieves a high macro-F1 score of over 99% on an external validation set and establishes a reliable benchmark for the field. • Extensive evaluation on the internal and three external test datasets demonstrates that MammoRG achieves SOTA performance on both NLG and CE metrics. 2 Related Work 2.1 Radiology Report Generation Radiology report generation has evolved from early image-to-text frameworks toward multimodal reasoning systems that emphasize diagnostic accuracy and

Read the complete text