A CNN-Transformer for Classification of Longitudinal 3D MRI Images – A Case Study on Hepatocellular Carcinoma Prediction
Jakob Nolte, Maureen M. J. Guichelaar, Donald E. Bouman, Stéphanie M. van den Berg, Maryam Amir Haeri · 2025
Read A CNN-Transformer for Classification of Longitudinal 3D MRI Images – A Case Study on Hepatocellular Carcinoma Prediction by Jakob Nolte, Maureen M. J. Guichelaar, Donald E. Bouman, Stéphanie M. van den Berg, Maryam Amir Haeri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10733.txt.
Introduction
Jakob Nolte and co-authors build HCCNet, a model that reads a patient's series of liver MRIs over time to predict liver cancer before it shows. Their paper appeared in 2025. Earlier tools look at one scan at a time, yet cirrhosis patients are scanned on irregular schedules and changes are subtle. The model pairs a three-dimensional convolutional network for each scan's detail with a transformer that reads the sequence, learning how the disease evolves.
Both halves were pre-trained with self-made tasks suited to medical scans and their ordering. On a surveillance cohort, the model predicted cancer better than single-scan approaches. The authors say using the film, not the snapshot, is the honest way to screen a slow disease.
Catalog record
| CREATOR | Jakob Nolte, Maureen M. J. Guichelaar, Donald E. Bouman, Stéphanie M. van den Berg, Maryam Amir Haeri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-10733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A CNN-Transformer for Classification of Longitudinal 3D MRI Images – A Case Study on Hepatocellular Carcinoma Prediction Jakob Nolte, Maureen M. J. Guichelaar, Donald E. Bouman, Stéphanie M. van den Berg, Maryam Amir Haeri Abstract Longitudinal MRI analysis is crucial for predicting disease outcomes, particularly in chronic conditions like hepatocellular carcinoma (HCC), where early detection can significantly influence treatment strategies and patient prognosis. Yet, due to challenges like limited data availability, subtle parenchymal changes, and the irregular timing of medical screenings, current approaches have so far focused on cross-sectional imaging data. To address this, we propose HCCNet, a novel model architecture that integrates a 3D adaptation of the ConvNeXt CNN architecture with a Transformer encoder, capturing both the intricate spatial features of 3D MRIs and the complex temporal dependencies across different time points. HCCNet utilizes a two-stage pre-training process tailored for longitudinal MRI data. The CNN backbone is pre-trained using a self-supervised learning framework adapted for 3D MRIs, while the Transformer encoder is pre-trained with a sequence-order-prediction task to enhance its understanding of disease progression over time. We demonstrate the effectiveness of HCCNet by applying it to a cohort of liver cirrhosis patients undergoing regular MRI screenings for HCC surveillance. Our results show that HCCNet significantly improves predictive accuracy and reliability over baseline models, providing a robust tool for personalized HCC surveillance. The methodological approach presented in this paper is versatile and can be adapted to various longitudinal MRI screening applications. Its ability to handle varying patient record lengths and irregular screening intervals establishes it as an invaluable framework for monitoring chronic diseases, where timely and accurate disease prognosis is critical for effective treatment planning. Journal: Biomedical Signal Processing and Control 1 Introduction Predicting disease outcomes based on longitudinal MRI scans is crucial for improving patient management. This is especially important in conditions like hepatocellular carcinoma (HCC), where early detection of tissue changes and subsequent monitoring can significantly impact treatment decisions and prognosis ( Parikh et al., 2020 ) . Longitudinal MRI analysis enables the capture of disease progression over time, providing a comprehensive view of how pathological changes evolve ( Gong et al., 2024 ) . However, despite its potential, the development of effective predictive models that can fully leverage the temporal information in longitudinal MRI data remains a significant challenge ( Jin et al., 2021b ) . One of the major challenges lies in accurately detecting early signs of disease progression from MRI scans, particularly in identifying subtle changes in the underlying parenchyma that may eventually lead to a malignancy. These challenges are further compounded by the need to accurately capture complex temporal dependencies across multiple time points while preserving the spatial integrity of the 3D MRI data ( Tang et al., 2021 ) . Traditional approaches often simplify this task by extracting 2D slices from 3D MRI scans and applying convolutional neural networks (CNNs) pre-trained on large-scale natural image datasets ( Kumar and Marttinen, 2024 ) . However, this method has limitations, particularly in the medical domain where datasets are often small, and the differences between natural and medical images—such as texture, contrast, and noise—significantly impact the transferability of pre-trained models ( Zeng et al., 2021 ) . The scarcity of large, labeled datasets in medical imaging exacerbates these challenges, making it difficult to train models that generalize well across different patient populations. As a result, pre-training approaches have become crucial in the medical domain, helping to improve model performance by enabling the network to learn robust features from limited data ( Tang et al., 2021 ) . However, many existing approaches have not fully explored the potential of these techniques in the context of longitudinal MRI analysis, particularly for diseases that require monitoring over extended periods ( Parikh et al., 2020 ) . Likewise, most deep learning applications are aimed at detecting a malignancy, resulting in a lack of studies focusing on detecting tissue changes before the tumor actually develops. In response to these limitations, we propose HCCNet, a novel model architecture designed specifically for longitudinal 3D MRI analysis. HCCNet integrates a 3D adaptation of the ConvNeXt architecture with a Transformer encoder, allowing it to capture both the spatial features of 3D MRIs and the temporal dependencies between different scans ( Gong et al., 2024 ) . Unlike previous approaches that rely on 2D slices or disregard the irregular intervals between scans, HCCNet is designed to handle the complexities of longitudinal data, making it better suited for tasks like predicting disease progression ( Jin et al., 2021b ) . To address the challenges of early detection and small datasets, HCCNet incorporates a robust pre-training strategy. The CNN backbone is pre-trained using an extended version of the DINO framework, adapted to 3D MRI, which treats different MRI sequences as natural augmentations of the same image ( Tang et al., 2021 ) . This self-supervised learning approach allows the model to learn from the inherent variability in MRI data, even when labeled examples are scarce ( Kumar and Marttinen, 2024 ) . Additionally, the Transformer encoder is pre-trained with a sequence-order-prediction task inspired by techniques used in natural language processing, which helps the model to understand the progression of disease over time and to make accurate predictions despite the irregular timing of medical screenings ( Zeng et al., 2021 ) . Our study specifically applies HCCNet to the task of predicting HCC development in liver cirrhosis patients, based on longitudinal MRI scans. By fine-tuning the pre-trained model on this dataset, we demonstrate the model’s ability to predict whether a patient will develop HCC at the next examination ( Parikh et al., 2020 ) . This application not only serves as a showcase of HCCNet’s capabilities but also addresses a critical need in personalized HCC surveillance of cirrhotic patients, where timely and accurate predictions can significantly influence clinical outcomes. Furthermore, the methodological approach presented in this paper can also be considered for predicting other conditions that, similar to HCC, benefit from monitoring changes over time. Thus, HCCNet provides a versatile framework that can be adapted to various longitudinal medical imaging tasks, potentially improving predictive accuracy and patient outcomes across a range of diseases. All code for the proposed method is available at: github.com/jmnolte/HCCNet . 2 Related Work 2.1 Modeling Approaches in Longitudinal Medical Image Analysis To date, deep learning applications to medical imaging still predominately consider cross-sectional instead of repeated imaging data. However, physicians usually assess the progression of disease or the effect of treatment by comparing patients’ current to previous imaging records. As a result, longitudinal imaging has long been established as the gold standard in many branches of oncology, enticing recent studies to extend the analysis of medical imaging to the analysis of imaging data across time. Three main methodological frameworks have emerged from these studies. Among them, fully convolutional neural networks (CNNs) model the within person change across time by stacking the image screenings from multiple time points into a multidimensional image and feeding the single joint image through the network ( Gao et al., 2021 ; Dammu et al., 2023 ) . While the approach is relatively simple to implement, it has been shown to not fully explore the relationship between time points ( Dammu