Medical Report Generation: A Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework

Yucheng Song, Yifan Ge, Junhao Li, Zhining Liao, Zhifang Liao · 2025

Read Medical Report Generation: A Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework by Yucheng Song, Yifan Ge, Junhao Li, Zhining Liao, Zhifang Liao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02271.txt.

Introduction

Yucheng Song, Yifan Ge, Junhao Li, Zhining Liao, and Zhifang Liao proposed a framework for automatically writing radiology reports from medical images, but the authors withdrew the submission from arXiv. In their withdrawal notice they stated that problems with the training epochs and training strategy had caused numerical errors in the results comparison table. They had decided to withdraw the preprint because of those errors. Its reported performance figures should therefore not be relied on.

As submitted, the work had described a system called HTSC-CIF that split report generation into three levels: aligning medical entities with image locations, tightening text-image alignment through masked modelling. It also applied a causal intervention to remove spurious correlations between image and text. The authors presented experiments claiming it beat current methods, claims that the withdrawal leaves unestablished.

Catalog record

CREATORYucheng Song, Yifan Ge, Junhao Li, Zhining Liao, Zhifang Liao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02271

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Medical Report Generation: A Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework Yucheng Song, Yifan Ge, Junhao Li, Zhining Liao, Zhifang Liao Abstract Medical Report Generation (MRG) is an indispensable component of modern medical diagnostics. It can automatically generate corresponding medical reports based on given radiological images, alleviating the burden on radiologists. However, designing a MRG model capable of reliably describing lesion areas still presents the following challenges: 1) insufficient understanding of domain-specific knowledge; 2) poor embedding space alignment between entities in the report text and visual signals; and 3) spurious correlations arising from visual and linguistic biases. Previous attempts have mostly focused on addressing individual challenges without comprehensively considering the issues faced by MRG. In this paper, we aim to tackle the above three formidable challenges through a novel hierarchical task decomposition perspective. To this end, we propose a Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework (HTSC-CIF) for MRG. This framework is the first to classify the three challenges of MRG into different levels. In the low-level task, we adopt a novel method for aligning entity features specific to medical tasks with spatial locations to help the upper-level tasks’ visual encoder better understand domain knowledge in terms of spatial positions. In the mid-level task, we employ Prefix Language Modeling for text and Masked Image Modeling for images, enhancing cross-modal understanding and alignment through mutually guided generation. Finally, to mitigate the spurious correlations caused by cross-modal data biases, we design a cross-modal causal intervention module. This module aims to reduce cross-model confounders through causal front-door intervention, thereby achieving enhanced interpretability in the high-level task. Extensive experimental results demonstrate that our hierarchical task framework is reasonable and effective, significantly outperforming other state-of-the-art methods in the MRG task. We make code public on the acceptance of the paper. Figure 1 : Multi-level task design of HTSC-CIF. 1 Introduction In the realm of medical imaging technology, the identification of latent pathological changes and the formulation of lucid diagnostic reports represent a time-consuming and technically demanding task. This imposes a significant burden on radiologists, particularly compromising the quality of reports and elevating the potential for errors Delrue et al. 2011 . This challenge necessitates the exploration of automated Medical Report Generation (MRG) systems. However, current MRG also faces several challenges: 1) How to incorporate rich domain knowledge into the model to improve the accuracy and reliability of the report. Medical images contain a significant amount of specialized information, which often requires a deep medical background to fully comprehend; 2) How to effectively optimize visual encoding and ensure the spatial alignment between entity descriptions in the report text and visual signals. In medical image analysis, visual encoding must not only capture the visual features in the image but also ensure that these features accurately correspond to the text descriptions in the report; 3) Spurious correlations caused by visual and linguistic biases. This bias may arise from inconsistencies between the image data and the statistical descriptions in the text, leading to misleading associations during model training. Specifically, the model may incorrectly associate certain visual features with specific statistical characteristics of text descriptions, despite these associations lacking any medical basis. This phenomenon not only reduces the accuracy of the report but also significantly impacts the interpretability of the model. To address the above challenges, researchers have made significant efforts to advance MRG methods Hartsock and Rasool 2024 . Firstly, existing studies integrate domain knowledge to strengthen visual encoding and contextual understanding, thereby laying the foundation for cross-modal alignment Huang et al. 2023 ; Li et al. 2023a ; Wu et al. 2023 ; Liu et al. 2021a . Simultaneously, researchers explore cross-modal alignment between visual signals and entity descriptions to achieve semantic consistency Liu et al. 2021b ; Wang et al. 2018 ; Qin and Song 2022 ; Boecking et al. 2022 . However, due to spurious correlations in image-text data arising from visual and linguistic biases, generating accurate and reliable reports that describe lesion regions remains highly challenging. To resolve this, many studies incorporate causal intervention methods into MRG to enhance report accuracy and interpretability Chen et al. 2023 ; Chen et al. 2021a . It is evident that these works are interconnected, progressively deepening, and mutually reinforcing. Therefore, an effective approach is needed to combine domain knowledge integration, cross-modal alignment of visual and textual features, and causal intervention, thereby comprehensively improving the accuracy, reliability, and interpretability of medical imaging diagnostic report generation. In this paper, we propose a Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework (HTSC-CIF) for medical imaging report generation. This framework primarily consists of three hierarchical tasks: domain knowledge enhancement, cross-modal alignment, and interpretability enhancement under causal intervention. These tasks can be regarded as low-level, mid-level, and high-level tasks in MRG. For the low-level task, we design a domain knowledge enhancement module. By employing a novel entity feature description and spatial alignment method tailored to specific medical tasks, this module assists the visual encoder in better understanding domain knowledge at the spatial level for higher-level tasks. For the mid-level task, we design a cross-modal alignment module. We employ Prefix Language Modeling (PLM) for text and Masked Image Modeling (MIM) for images, enhancing cross-modal alignment through guided generation. This approach enables pretraining a network capable of generating preliminary report results. Finally, to mitigate spurious correlations caused by cross-modal data bias, we transfer the weights of the mid-level pre-trained network and design a cross-modal causal intervention module. This module aims to reduce cross-modal confounders through causal front-door intervention, thereby achieving the high-level task of interpretability enhancement. To the best of our knowledge, our work is the first to address the three challenges of medical image report generation. The main contributions of this paper are summarized as follows: • We are the first to simultaneously perform low-level domain knowledge enhancement, mid-level cross-modal information alignment, and high-level causal intervention tasks in MRG. These three tasks mutually guide and progressively promote each other from low to high levels. • We achieve the injection enhancement of low-level domain knowledge into upper-level tasks, and the upper-level tasks can obtain rich domain knowledge without additional the network architecture. • We experimentally demonstrate the superiority of HTSC-CIF and SOTA performance on two public benchmark datasets. 2 Related Work 2.1 Domain Knowledge Enhancement Enhancing domain knowledge in medical image report generation improves model understanding of image data, leading to more accurate, clinically relevant, and interpretable reports. Researchers have developed various injection methods for MRG Xie et al. 2021 , including using knowledge graphs Zhang et al. 2020 ; Yan et al. 2023 to reduce language biases and fuse visual features Xu et al. 2023 . Other approaches extract medical concepts from reports Yuan et al. 2019 to enrich semantics. As a crucial foundational task, domain knowledge provides medical context, and we optimize the visual encoder by injecting it to better capture image features, enhancing the ability to parse data for report generation. Figure 2 : The overall structure of HTSC-CIF. (a) Domain Knowledge Enhancement Module. (b) Cross-Modal Alignment Module. (c) Cross-Modal Causal Intervention Module. 2.2 Cross-Modal Alignment Cross-modal alignment in medical image report generation ensures semantic consistency between visual and textual data, improving report accuracy by addressing modality differences. Key methods include alignment modules

Read the complete text