TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis from Imaging, Clinical, and Radiomic Data Fusion
Linglong Wu 1,† , Xuhao Shan 1,† , Ruiquan Ge 1,∗ , Ruoyu Liang 2 , Chi Zhang 2 , Yonghong Li 3 , Ahmed Elazab 4 , Huoling Luo 3,7 , Yunbi Liu 5 , Changmiao Wang 6,∗ · 2025
Read TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis from Imaging, Clinical, and Radiomic Data Fusion by Linglong Wu 1,† , Xuhao Shan 1,† , Ruiquan Ge 1,∗ , Ruoyu Liang 2 , Chi Zhang 2 , Yonghong Li 3 , Ahmed Elazab 4 , Huoling Luo 3,7 , Yunbi Liu 5 , Changmiao Wang 6,∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00695.txt.
Introduction
Linglong Wu and co-authors (2025) build a network that forecasts the course of chronic liver disease by fusing three kinds of patient data. CT scans show the organ. Radiomics extracts hundreds of numerical features from those scans. Clinical variables add the patient's history and labs. Each source describes the disease differently, and naive fusion can blur rather than sharpen. The authors' network, TMI-CLNet, is designed around that tension.
One module first consolidates each modality on its own terms. A second module then runs cross-attention across all three, letting each source query the others for what it lacks. The design is meant to scale as richer medical data types arrive, since new modalities can join the same exchange. For a disease where prognosis decides transplant timing and treatment intensity, squeezing more signal from data hospitals already collect is the practical prize.
Catalog record
| CREATOR | Linglong Wu 1,† , Xuhao Shan 1,† , Ruiquan Ge 1,∗ , Ruoyu Liang 2 , Chi Zhang 2 , Yonghong Li 3 , Ahmed Elazab 4 , Huoling Luo 3,7 , Yunbi Liu 5 , Changmiao Wang 6,∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00695
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TMI-CLNet: Triple-Modal Interaction Network for Chronic Liver Disease Prognosis from Imaging, Clinical, and Radiomic Data Fusion Linglong Wu 1,† , Xuhao Shan 1,† , Ruiquan Ge 1,∗ , Ruoyu Liang 2 , Chi Zhang 2 , Yonghong Li 3 , Ahmed Elazab 4 , Huoling Luo 3,7 , Yunbi Liu 5 , Changmiao Wang 6,∗ Abstract Chronic liver disease represents a significant health challenge worldwide and accurate prognostic evaluations are essential for personalized treatment plans. Recent evidence suggests that integrating multimodal data, such as computed tomography imaging, radiomic features, and clinical information, can provide more comprehensive prognostic information. However, modalities have an inherent heterogeneity, and incorporating additional modalities may exacerbate the challenges of heterogeneous data fusion. Moreover, existing multimodal fusion methods often struggle to adapt to richer medical modalities, making it difficult to capture inter-modal relationships. To overcome these limitations, We present the Triple-Modal Interaction Chronic Liver Network (TMI-CLNet). Specifically, we develop an Intra-Modality Aggregation module and a Triple-Modal Cross-Attention Fusion module, which are designed to eliminate intra-modality redundancy and extract cross-modal information, respectively. Furthermore, we design a Triple-Modal Feature Fusion loss function to align feature representations across modalities. Extensive experiments on the liver prognosis dataset demonstrate that our approach significantly outperforms existing state-of-the-art unimodal models and other multi-modal techniques. Our code is available at https://github.com/Mysterwll/liver.git . † † address: 1 Hangzhou Dianzi University, China 2 The Chinese University of Hong Kong, Shenzhen, Shenzhen, China 3 Shenzhen Institute of Information Technology, China 4 Shenzhen University, China 5 Nanjing University of Posts and Telecommunications , China 6 Shenzhen Research Institute of Big Data, China 7 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China 1 Introduction Chronic liver disease poses a significant threat to human health and safety. Chronic hepatitis caused by viral infections of hepatitis B virus (HBV) and hepatitis C virus (HCV) may progress to cirrhosis and hepatocellular carcinoma (HCC) [ 1 ] . Therefore, providing clinicians with timely and accurate prognostic results to guide early interventions and treatment decisions is of critical importance in mitigating the health burden of chronic liver diseases. In clinical practice, patients present various non-imaging data including demographic information, complete blood count reports, and lipid analysis reports alongside routine imaging examinations such as Computed Tomography (CT). Meanwhile, radiomics can extract quantitative features from medical imaging data to describe disease characteristics. By integrating comprehensive domain knowledge with auxiliary information, we can develop a holistic and precise prognostic model for chronic liver disease, thereby offering robust support for clinical practice. Numerous studies have explored multimodal approaches across various fields [ 2 ] , often merging different modalities through simple concatenation or attention mechanisms [ 3 , 4 ] . However, these conventional methods fail to accurately capture the intricate relationships between different modalities and struggle when dealing with a larger number of data types, thereby falling short of the demands of more complex datasets. To address the challenges, we pioneeringly introduce a multimodal approach for prognosis assessment in chronic liver disease. Our work makes three notable contributions: (1) To our knowledge, this is the pioneering study to simultaneously integrate CT imaging, radiomic features, and clinical information into a unified multi-modal learning framework. (2) We design a unique TCAF module to address the heterogeneity among different modalities. This module effectively extracts cross-modal information to generate comprehensive global feature representations. (3) We introduce a TMFF loss to align feature representations across these three modalities during training, ensuring consistent semantic matching. Figure 1: Overview of our proposed TMI-CLNet. Given the CT scans, radiomic features, and clinical text, we feed them into an independent modal encoder to extract high-dimensional features. Subsequently, these features are fused through multi-modal interaction to incorporate global characteristics for prognosis classification. Simultaneously, the TMFF loss is employed to align multi-modal features. 2 Methodology The overall architecture of our network, as illustrated in Figure 1 , consists of three main components: a feature extraction module, a multi-modal interaction module, and a classification head module. In the feature extraction module, we employed a pre-trained 3D ResNet-50 as the visual encoder to extract deep visual features [ 5 ] . The radiomics data were tensorized and then processed using a Multilayer Perceptron (MLP) as the radiomics encoder to achieve more abstract feature representations. For clinical text, we utilized a pre-trained BioBERT model as the text encoder, selecting the output of the last hidden layer as the text feature representation [ 6 ] . In the classification head module, a pre-trained 1D DenseNet-121 is employed as the classification head. 2.1 Multi-modal Interaction Module In this module, we employed linear layers from the outset to map the high-dimensional features obtained from the feature extraction module to a uniform size. Following this, the normalized high-dimensional features are processed by the Intra-Modal Aggregation (IMA) module to enhance their quality. At the core of the IMA is a multi-head self-attention mechanism with 16 attention heads. The primary role of the IMA is to consolidate intra-modal information before feature fusion, thereby improving the effectiveness of the fusion process and enhancing the overall performance of the model. Figure 2: Overview of our proposed TCAF module. The highlighting the process by the independent features from Modality 1 are integrated to produce cross-modal features. 2.2 Triple-modal Cross-Attention Fusion Module The architecture of this module is illustrated in Figure 2 . After processing through the IMA module, the enhanced features from different modalities pass through linear layers to calculate their respective query \mathit{Q} , key \mathit{K} , and value \mathit{V} components for attention computation. The \mathit{Q} of each modality is multiplied by the transposed \mathit{K} of the adjacent modality, followed by a softmax layer. This result is then multiplied by the \mathit{V} of the adjacent modality to compute the attention scores for the given modality. Formally, this can be expressed as: F_{hidden}^{i}=softmax(\frac{Q_{i}K_{j}}{\sqrt{d_{k}}})V_{j},j=i\bmod 3+1, (1) where \mathit{i} and \mathit{j} represent the index of the modality, \mathit{d_{k}} is the dimensions of the vector \mathit{K} , Moreover, \mathit{a\bmod n} represents the remainder of \mathit{a} divided by \mathit{n} . This is because the third modality interacts with the first modality through computational operations. Subsequently, to retain the original independent feature information, the attention scores ( \mathit{F_{hidden}} ) are combined with the initial features ( \mathit{F_{modality}} ) through concatenation. This combined output is then multiplied by a learnable weight matrix ( \mathit{W^{o}} ). This procedure is applied consistently to inputs from the three different modalities. Ultimately, the results are concatenated to produce the global feature output ( \mathit{F_{global}} ). Mathematically, this can be represented as: F_{global}=\bigoplus_{i=1}^{n}W_{i}^{o}(F_{hidden}^{i}\oplus F_{modality}^{i}). (2) This ensures that the connected features effectively serve as the comprehensive global feature representation. 2.3 Loss Function TMFF Loss. To align features across different modalities, we drew inspiration from the Similarity Distribution Matching (SDM) loss [ 7 ] and developed a TMFF loss specifically for clinical texts, radiomics, and deep image features. The SDM loss was originally designed for the global alignment of visual and textual embeddings and is defined as: \mathscr{L}_{\mathrm{v2t}}=KL\left(\mathbf{p}_{\mathbf{i}}\|\mathbf{q}_{\mathbf{i}}\right)=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n}p_{i,j}\log\left(\frac{p_{i,j}}{q_{i,j}}\right), (3) where q denotes the genuine matching probability, and p represents the fraction of a particular cosine similarity score relative to the total sum. The bidirectional SDM loss aggregates both v2t and t2v losses. Thus, the SDM loss between visual and textual