HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation
Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, Baocai Yin · 2024
Read HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation by Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, Baocai Yin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11070.txt.
Introduction
Tengfei Liu and co-authors teach a language model to write radiology reports that remember the patient's past scans. A single chest X-ray read in isolation misses the story: what changed since last year is often the diagnosis. Their framework, HC-LLM, feeds the model a series of images and reports, and pulls out features shared across time and features unique to each visit.
Constraints keep the story straight. Similarity rules bind each report to its own image, while contrastive and structural rules align words and pictures across the timeline. The model then writes reports that track progression rather than redescribe each film. On the Longitudinal-MIMIC benchmark the system set the best scores, and it still worked when a test case arrived with no history attached.
Catalog record
| CREATOR | Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, Baocai Yin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, Baocai Yin Abstract Radiology report generation (RRG) models typically focus on individual exams, often overlooking the integration of historical visual or textual data, which is crucial for patient follow-ups. Traditional methods usually struggle with long sequence dependencies when incorporating historical information, but large language models (LLMs) excel at in-context learning, making them well-suited for analyzing longitudinal medical data. In light of this, we propose a novel Historical-Constrained Large Language Models (HC-LLM) framework for RRG, empowering LLMs with longitudinal report generation capabilities by constraining the consistency and differences between longitudinal images and their corresponding reports. Specifically, our approach extracts both time-shared and time-specific features from longitudinal chest X-rays and diagnostic reports to capture disease progression. Then, we ensure consistent representation by applying intra-modality similarity constraints and aligning various features across modalities with multimodal contrastive and structural constraints. These combined constraints effectively guide the LLMs in generating diagnostic reports that accurately reflect the progression of the disease, achieving state-of-the-art results on the Longitudinal-MIMIC dataset. Notably, our approach performs well even without historical data during testing and can be easily adapted to other multimodal large models, enhancing its versatility. Code is available at: https://github.com/TengfeiLiu966/HC-LLM . Introduction Figure 1: Illustration of the longitudinal report generation process. Unlike traditional methods that rely solely on the current chest X-ray, our approach emphasizes the effective utilization of historical diagnostic information to significantly enhance the accuracy of LLMs in RRG. Radiology report generation (RRG) is a crucial research area in medical AI, with numerous studies focused on reducing the heavy workload of radiologists. In clinical practice, an essential function of these reports is to document pathological changes in patients, aiding doctors in recalling and diagnosing disease progression. As a result, ground truth reports often include descriptions of historical information. However, most existing models ( Chen et al. 2020 ; Liu et al. 2021a ; Chen et al. 2021 ; Li et al. 2023a ; Li et al. 2023c ; Tanida et al. 2023 ; Liu, Tian, and Song 2023 ; Huang, Zhang, and Zhang 2023 ; Wang et al. 2023b ; Wang et al. 2023a ; Liu et al. 2024b ; Jin et al. 2024 ; Liu et al. 2024a ; Shen et al. 2024 ; Li et al. 2024 ) rely on a single image as input, preventing them from accurately generating descriptions of prior references and thereby impacting their performance. This limitation is evident as current models struggle to achieve high scores on natural language generation (NLG) metrics. Therefore, as illustrated in Figure 1 , this paper focuses on a more practical research problem: how to generate radiology reports from longitudinal data. Zhu et al. 2023b have made preliminary attempts in this direction by utilizing a memory mechanism to incorporate historical information for enhanced chest X-ray report generation. However, their approach still relies on traditional cross-attention mechanisms and requires the presence of historical data during testing, limiting its practicality. Recently, large language models (LLMs) ( Wang et al. 2024b ) have been successfully applied to traditional radiology report generation tasks ( Jin et al. 2024 ; Liu et al. 2024a ; Liu et al. 2024b ) , and their inherent in-context learning abilities make them well-suited for analyzing longitudinal medical data. However, despite this potential, directly inputting longitudinal medical data into LLMs often struggles to produce reports that accurately capture the progression of diseases over time. Therefore, our work focuses on a key challenge: How can historical diagnostic information be effectively utilized to enhance the radiology report generation capabilities of LLMs? To address this challenge, we propose a Historical-Constrained Large Language Models (HC-LLM) framework for RRG. Specifically, considering changes in disease progression, which may involve the disappearance, stability, and emergence of conditions, we first extract time-shared (stability) and time-specific (disappearance, emergence) features from chest X-rays and diagnostic reports at two-time points. We then employ two types of constraints: intra-modality and inter-modality constraints, which indirectly guide LLM generation by aligning the shared and specific features of the generated and historical reports with those of the corresponding longitudinal images. This alignment ensures that the generated reports accurately capture disease progression, thereby enhancing overall effectiveness. For intra-modality constraints, we apply similarity constraints to ensure the consistency of time-shared features within each modality, preserving the integrity of chest X-ray and diagnostic report characteristics over time. For inter-modality constraints, we implement multimodal contrastive constraint and multimodal structural constraint, respectively. The former is responsible for aligning time-shared and time-specific features of corresponding chest X-rays and reports while distancing non-matching pairs. This constraint also indirectly enhances the separation of features within the same modality, making the representation of time-shared and time-specific features more precise. The latter further regulates the spatial distribution of features by forming triangular structures, ensuring that the geometric relationships (e.g., distances and angles) within the triangles of image features correspond to those within text features. The combined effect of these constraints ensures that the generated reports more accurately reflect the progression of diseases, thereby enhancing their accuracy. Experimental results on the Longitudinal-MIMIC dataset demonstrate that our method achieves state-of-the-art performance on most NLG metrics, validating its effectiveness. Additionally, our method achieves superior results compared to other approaches without using historical information during testing and can be adapted to various multimodal large model frameworks, demonstrating strong applicability. We summarize contributions as follows: • We propose an innovative HC-LLM framework that leverages historical diagnostic data to improve the adaptability and performance of LLMs in RRG. • We propose tri-consistency constraints that can effectively enhance the consistency and specificity of generated reports with historical data, ensuring alignment with disease progression observed in sequential chest X-rays. • The proposed framework achieves superior performance without relying on historical data during testing and can be easily integrated with various multimodal large models, demonstrating its strong applicability. • Extensive evaluations on the Longitudinal-MIMIC dataset demonstrate that our method achieves state-of-the-art performance, underscoring its effectiveness and robustness in leveraging historical data to enhance radiology report generation with LLMs. Figure 2: Overview of the proposed framework: First, the current chest X-ray is processed to generate a diagnostic report using a visual encoder and LLM. The framework then extracts time-shared and time-specific features from the current and prior chest X-rays, along with the generated and prior diagnostic reports. Then, similarity constraints are first applied to ensure consistent time-shared representation over time. Finally, multimodal contrastive and structural constraints are employed to align shared and specific features across modalities, ensuring the generated report accurately reflects disease progression. Related Works Radiology Report Generation : Radiology report generation methodologies have evolved significantly from early CNN-RNN frameworks ( Vinyals et al. 2015 ; Lu et al. 2017 ; Wang et al. 2024a ; Wang et al. 2024c ) to the integration of advanced Transformer architectures ( Li et al. 2022 ; Wang et al. 2023a ; Huang, Zhang, and Zhang 2023 ) . Initial approaches focused on cross-modal alignment, utilizing CNNs for image features and RNNs for text generation. The advent of Transformers brought enhanced cross-modal interactions and long-range dependency modeling. Researchers introduced memory modules ( Chen et al. 2020 ; Chen