Multimodal fusion-based prediction of postoperative survival in gallbladder adenocarcinoma: Model development and validation Authors: Fan-xiu Meng, Ya-rong Guo, Si-yu Zhang, Jian-xin Zhang, Ling-jie Wang, Qi Li Published: 2026 ABSTRACT AbstractAccurate prognostic models are essential for optimizing treatment strategies in gallbladder adenocarcinoma (GBAC). We preliminarily constructed and validated a multimodal fusion model integrating clinical features, enhanced computed tomographic (CT) images, and digital histopathological images to predict postoperative survival. Data from 177 patients with GBAC who underwent surgery between January 2017 and May 2023 at two tertiary hospitals were retrospectively analyzed. A deep-learning radiomics model was developed using preoperative, portal-venous phase, contrast-enhanced CT images. A pathomics model was constructed from hematoxylin-and-eosin-stained whole-slide images using a multi-instance learning approach. The three single-modality models were integrated through late-fusion logistic regression to generate a multimodal nomogram. The deep-learning radiomics model achieved an area under the receiver operating characteristic curve (AUC–ROC) of 0.883. The pathomics and clinical models yielded AUC–ROCs of 0.780 and 0.693, respectively. The multimodal nomogram showed superior performance (AUC–ROC: 0.896; sensitivity: 0.903; specificity: 0.810; accuracy: 0.865; concordance index: 0.736). Calibration and decision curve analysis confirmed clinical utility. Kaplan–Meier analysis revealed notable survival differences between the high- and low-risk sub-cohorts. In this limited cohort, a multimodal model integrating CT-based radiomics, pathomics, and clinical features showed preliminary association with postoperative survival in GBAC, and has the potential to support individualized clinical risk stratification in the future.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1038/s41598-026-57833-3.Keywords: multimodal fusion, gallbladder adenocarcinoma, pathomics, radiomics, survival predictionSubject terms: Cancer, Gastroenterology, Oncology Introduction Gallbladder cancer arises from the mucosal epithelium of the gallbladder and is one of the most prevalent biliary tract malignancies. Over 90% of cases are gallbladder adenocarcinomas (GBACs)1. The disease is characterized by an insidious onset and aggressive progression. At diagnosis, most patients have already reached an advanced stage, contributing to a poor prognosis2, with a 5-year relative survival rate of 5–24.7%3,4. Curative-intent surgery is the primary treatment option; however, postoperative recurrence rates remain high. The formulation of adjuvant therapy strategies continues to be debated5,6. Therefore, the development of accurate prognostic models is essential to inform individualized treatment planning and enhance outcomes. Traditional prognostic prediction relies on clinical and histopathological features, such as Tumor–Node–Metastasis staging, differentiation grade, perineural invasion, and molecular markers7–9. Nonetheless, these unidimensional parameters fail to fully capture tumor heterogeneity. Advancements in artificial intelligence (AI) now enable quantitative analyses of medical imaging and digital histopathology, offering novel avenues for prognostic prediction. Radiomics enables the high-throughput extraction of quantitative features from medical imaging, including computed tomography (CT) and magnetic resonance imaging, and has demonstrated utility in predicting the prognosis of GBAC. Recent studies have revealed that CT-based radiomic feature analysis can effectively predict overall survival (OS) in patients with GBAC postoperatively10,11. Conventional radiomics relies on manual feature extraction and region-of-interest (ROI) delineation, which can introduce subjectivity. Deep learning automatically learns hierarchical image features yet requires large training datasets. Combining these approaches may leverage the strengths of both methods. In our previous study, we showed that integrating machine and deep learning algorithms using preoperative contrast-enhanced CT images from patients with GBAC produced a prognostic model with enhanced predictive performance12. Histopathology remains the diagnostic “gold standard” and provides extensive prognostic insight. Advances in digital and computational histopathology have enabled deep learning-based analysis of whole-slide images (WSIs), revealing prognostic utility across multiple cancers, such as non-small cell lung, ovarian, and breast carcinomas13–15. However, these approaches remain insufficiently studied in GBAC. Owing to tumor complexity, single-modality data can inadequately reflect prognostic detail. Multimodal data-fusion technology, which integrates clinical, imaging, and histopathological data, offers potential for developing more precise and robust predictive models16. This approach has emerged as a focal area of research in precision medicine17–19. Hou et al.20 utilized the WSIs and messenger ribonucleic acid expression data of 346 patients with hepatocellular carcinoma from The Cancer Genome Atlas database. They applied multi-instance deep learning and weighted gene co-expression network analysis to derive survival risk scores and identify hub genes. Integration of multimodal features using the Cox model has enhanced prognostic performance. Similarly, Li et al.11 developed a multimodal fusion framework, combining histopathological WSIs and radiological data from 365 glioblastoma and 213 stomach carcinoma cases from The Cancer Genome Atlas, to generate a comprehensive risk score. This model demonstrated superior survival prediction compared to that achieved with single-modality and alternative approaches. To the best of our knowledge, few studies have integrated radiological, histopathological, and clinical data for prognostic prediction in GBAC. Therefore, in this study, we developed three predictive models encompassing deep learning radiomics, pathomics, and clinical features. These were integrated using a multimodal fusion strategy to construct a comprehensive postoperative survival prediction model, which was validated to provide a tool for individualized prognostic assessment and management in GBAC. Methods Ethical approval and informed consent This study was approved by the institutional Ethics Committee of Shanxi medical university on August 16th, 2023, with approval number: KYLL-2023-181. Moreover, this study was conducted in accordance with the principles of the Declaration of Helsinki. Informed consent was obtained from all subjects involved in the study. Written informed consent was obtained from the patients for the publication of their de-identified data and images. Study design Patients with histopathologically confirmed GBAC who underwent surgery between January 2017 and May 2023 at two tertiary hospitals were retrospectively enrolled. The inclusion criteria were as follows: (1) patients who underwent a radical surgical resection, (2) patients with a postoperative histopathological confirmation of GBAC, (3) completion of contrast-enhanced abdominal CT within 1 month preoperatively, and (4) availability of postoperative hematoxylin and eosin (H & E)-stained histopathological images. The exclusion criteria were as follows: (1) patients with poor-quality CT images; (2) patients who had undergone palliative or repeat surgery; (3) patients with a preoperative history of neoadjuvant therapy, such as chemotherapy or radiotherapy; (4) patients with other malignant tumors; (5) any perioperative mortality; and (6) refusal to participate in the follow-up or incomplete clinical data. A flowchart of the study design and patient selection process are shown in Figs. 1 and 2, respectively. Finally, a total of 177 patients were included and divided into training (n = 109) and validation (n = 68) cohorts, based on the institution. Depiction of study design. Flowchart of patient enrollment. Clinical data acquisition Patient demographic information, medical history, laboratory test results obtained within 1 week preoperatively, and detailed postoperative histopathological findings were collected from the institutions’ databases. All patients were regularly followed up for a total period of 3 years. OS was defined as the time from the date of surgery to the date of death from any cause or the last follow-up visit. The primary outcome of this study is 1-year overall survival (OS). Patients who died from any cause within 365 days after surgery were defined as positive (dead); those who survived for ≥ 365 days after surgery were defined as negative (alive). CT images acquisition and processing All patients underwent preoperative contrast-enhanced abdominal CT. All CT images were retrieved from the institutions’ picture archiving and communication systems. Detailed CT acquisition protocols are provided in Supplementary Methods S1. CT images underwent preprocessing for standardization, First, pixel intensities were confined to the range of -125–225 Hounsfield units, with a window level of 50 and window width of 350, to mitigate the influence of extreme values and outliers. Second, a fixed-resolution resampling method was employed to achieve a uniform voxel spacing of 1 × 1 × 1 mm, ensuring spatial normalization and addressing voxel spacing inconsistencies across various volumes of interest (VOIs). Portal-venous-phase Digital Imaging and Communications in Medicine images were imported into the ITK-SNAP software (v4.0.0; University of Pennsylvania, Philadelphia, PA, USA). Two radiologists, each with > 5 years of experience and blinded to the clinical outcomes, independently performed slice-by-slice manual segmentation of the three-dimensional VOI encompassing the tumor (Supplementary Fig. S1). To assess intra-observer consistency, 30 randomly selected cases were re-segmented by Radiologist 1 after a 2-week interval. Histopathological data acquisition and processing For each patient, a representative post-operative, H-&-E-stained, histopathological slide with abundant tumor tissue, clear staining, and no notable folding was selected. WSIs were generated using an Aperio ScanScope scanner (Leica Biosystems, Buffalo Grove, IL, USA) at 40× magnification, achieving a pixel resolution of approximately 0.5 μm per pixel, and saved in the Slide and Viewable Storage format. Two board-certified histopathologists, each with at least 5 years of experience, independently performed manual rough annotation of tumor regions on the WSIs using QuPath21 software (v0.3.2; University of Edinburgh, Edinburgh, UK) (Supplementary Fig. S2). In cases of disagreement, discrepancies were discussed until a consensus was reached or the disputed region was excluded. The final annotated ROI for each case was saved as a histopathological annotation file in the Geographic JavaScript Object Notation format. Construction of prognostic models Development of a CT-based deep learning radiomics model Radiomics feature extraction and selection After completion of VOI segmentation, radiomics features were extracted from the segmented VOIs using the PyRadiomics software package (v3.0.1; Computational Imaging and Bioinformatics Lab, Harvard Medical School, Boston, MA, USA), which adheres to the standards established by the Image Biomarker Standardization Initiative22. To identify the most informative radiomics features, we implemented a four-step feature selection pipeline comprising the following procedures: interclass correlation coefficient analysis, t-test, Pearson’s correlation coefficient, and ElasticNet regularization. The detailed feature selection process is described in Supplementary Methods S2. Radiomics model construction Using the radiomics features selected in the previous step, prediction models were constructed using various machine learning algorithms, including Support Vector Machine (SVM), Random Forest (RF), Extremely Randomized Trees (ExtraTrees), Extreme Gradient Boosting (XGBoost), and Light Gradient Boosting Machine (LightGBM). A grid-search algorithm was employed to optimize hyperparameters for each model and identify the best-performing configurations. The top-performing model, determined by the highest area under the receiver operating characteristic curve (AUC–ROC), was designated as the radiomics model. Deep transformer learning model To represent each patient, we selected the CT slice containing the largest tumor area. In this study, we employed the pre-trained Dense Convolutional Network 121 (Densenet121; Meta AI, Menlo Park, CA, USA) convolutional neural network (CNN) model, originally trained on the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC-2012) dataset, to extract deep learning features. The model was implemented using PyTorch. Subsequently, gray values were normalized using min–max scaling, with the output range set to -1–1. Each cropped sub-region image was resized to 224 × 224 pixels using nearest-neighbor interpolation, which served as the input for the model. To improve generalization, the learning rate was carefully configured, and a cosine-decay learning rate algorithm was applied. The learning rate was expressed as follows: where , and denote the minimum and maximum learning rates and number of iteration epochs, respectively. Additional hyperparameter configurations included the optimizer stochastic gradient descent, loss function: softmax cross entropy, and a batch size of 32. By utilizing Densenet121 as the base model, a deep learning approach was applied to generate probability predictions for each patient. These predictions were subsequently used as input for the deep transformer learning model. Development of a deep-learning radiomics model A decision-level fusion strategy was adopted using logistic regression to integrate the output probabilities from both the radiomics and deep learning models. This integration yielded the final deep-learning radiomics model. Development of a WSI-based pathomics model We developed a histopathological patch-level classifier to extract meaningful features using a specialized training approach. Patch-level predictions generated by the classifier were aggregated to derive WSI features, which were subsequently integrated into a separate machine learning model for advanced analysis, as illustrated in the red section of our workflow chart. WSI preprocessing and patch generation After annotation of the ROIs, Otsu’s thresholding algorithm was applied to remove non-informative background regions from the WSIs. Thereafter, all images underwent color normalization via z-score normalization of the red, green, and blue channels to mitigate staining and scanner-related variability across datasets23. Owing to the substantial size of the WSIs, each image was divided into smaller patches measuring 512 × 512 pixels. To ensure that subsequent analyses focused on diagnostically relevant regions and to improve predictive efficiency, only patches exhibiting > 80% overlap with the annotated ROIs were retained for further analysis. To increase the size and diversity of the training dataset and improve the generalization capacity of the predictive model, a series of data augmentation strategies were employed. These included random horizontal and vertical flips of the image patches. Thus, > 160 000 distinct non-overlapping image patches were selected for inclusion in the training set. Patch-level prediction Regarding patch-level prediction, we evaluated the four top-performing CNN architectures: Densenet121; Residual Network 101 (Microsoft Research, Redmond, WA, USA), Visual Geometry Group Network 19 (University of Oxford, Oxford, UK), and Inception version 3 (Google Inc., Mountain View, CA, USA). These models were selected based on their demonstrated efficacy in ILSVRC-2012. The objective was to estimate the probability of each image patch being assigned to a category corresponding to its WSI. Transfer learning was applied to enhance model adaptability across cohorts. This process involved initializing the CNN architectures with pretrained weights from the ILSVRC-2012 dataset, followed by fine-tuning using task-specific labeled data. Leveraging ImageNet’s extensive feature representations helped improve model performance across heterogeneous datasets. Detailed training specifications are provided in Supplementary Methods S3. Multi-instance learning for WSI integration After model training, we focused on generating categorical labels and associated probabilities for individual image patches. These patch-level probabilities were subsequently aggregated using a classifier to produce WSI- level predictions. To facilitate the integration of patch-level probabilities, we developed two distinct machine learning methodologies. The first, the Patch Likelihood Histogram method, represented the WSI-wide probability distribution. The second, a Bag of Words, employed term frequency–inverse document frequency (TF–IDF) vectors to characterize patch content for WSI-level classification. Detailed descriptions of both methods are provided in Supplementary Methods S4. These independent methodologies enabled the efficient transformation of patch-level predictions into WSI-level features, which were essential for downstream analyses, such as recurrence risk assessment and survival analysis. Development of a pathomics prediction model Patch-level predictions, probability histograms, and TF–IDF features were integrated to construct individualized patient profiles. To refine feature selection, the Pearson correlation coefficient was applied, retaining only one feature from any pair, exhibiting a correlation > 0.9. Subsequently, the resulting feature vectors were used to train five distinct machine learning algorithms: SVM, RF, ExtraTrees, XGBoost, and LightGBM. Each WSI was linked to its corresponding clinical follow-up outcomes. The best-performing model was selected and designated as the final pathomics prediction model. Predictive heat maps were subsequently generated to visualize high-risk regions. Development of a clinical prediction model Univariate and multivariable regression analyses were performed on all clinical features of training set(n = 109), and variables with a p-value < 0.05 were retained for inclusion in the final clinical prediction model. Additionally, we evaluated the performance of five machine learning algorithms: the SVM, RF, ExtraTrees, XGBoost, and LightGBM. The best-performing model was selected to predict clinical outcomes. Development of a multimodal prognostic prediction model (Nomogram) A multimodal prognostic prediction model was developed by integrating previously constructed deep learning radiomics, pathomics, and clinical prediction models, using a late-fusion strategy with logistic regression. To support clinical implementation, a nomogram was constructed based on the resulting multimodal model. Models’ assessments To verify the effectiveness of the models, the accuracy, sensitivity, specificity, and AUC–ROC were calculated to assess prognostic efficacy. Additionally, calibration curves and Hosmer–Lemeshow analyses were employed to examine the calibration efficiency of the nomogram. Decision curve analysis (DCA) was performed to evaluate the clinical utility of the predictive models. Survival models were constructed using Cox proportional hazards models with L2 regularization and Kaplan–Meier analysis. Patients were stratified into two sub-cohorts based on the hazard rate. Statistical significance of the separation was determined using a multivariate log-rank test. Statistical analysis and experimental platform Appropriate statistical methods, including the t-, Mann–Whitney U-, and chi-square tests, were applied to compare the clinical characteristics of the cases. Regarding the radiomics pipeline, deep learning workflow, model construction, and model evaluation, multiple software tools were used, including ITK SNAP v.4.0.0 and custom code written in Python v.3.7.12 (Python Software Foundation, Wilmington, DE, USA). The other Python packages utilized in the analysis included PyTorch v.1.8.0 (Meta AI, Menlo Park, CA, USA), Onekey v.2.2.3 (Onekey ML, Hangzhou, Zhejiang, China), Scikit-learn v.1.0.2 (INRIA and Télécom Paris, Palaiseau, France), PyRadiomics v.3.0(Radiomics Consortium, Maastricht University, Maastricht, The Netherlands), and Lifelines v.0.27.0 (CamDavidsonPilon, Toronto, ON, Canada). The specific hardware configuration, machine learning hyperparameters, and deep learning parameters used in model construction are detailed in Supplementary Methods S5. To reduce the influence of randomness on the experimental results, the maximum and minimum values were excluded from the obtained outputs, and the remaining values were averaged. Results Patient clinical characteristics and the clinical prediction model The clinical characteristics of the 177 study patients are summarized in Table 1. Female patients outnumbered male patients (57.06% and 42.94%, respectively). The median age at diagnosis was 64 years (range: 31–86 years). During follow-up, 90 patients (55.85%) died, with no significant difference in mortality between the datasets (58/109; 32/68; p = 0.662). The 1-year overall survival rate was 63.98%. Table 1Clinical characteristics of patients.CharacteristicsTraining setP valueValidation setP valueAlive (n = 51)Dead (n = 58)Alive (n = 36)Dead (n = 32)Age (years)64.98 ± 9.1860.67 ± 10.52 0.03 62.74 ± 12.1963.81 ± 9.460.702Gender0.8321.0Male22 (43.14)27 (46.55)14 (38.89)13 (40.63)Female29 (56.86)31 (53.45)22 (61.11)19 (59.38)BMI (kg/m2)24.93 ± 3.6522.77 ± 3.48 0.003 23.24 ± 3.2624.34 ± 4.090.244Hypertension17 (33.33)12 (20.69)0.18813 (36.11)12 (37.5)1.0DM10 (19.61)8 (13.79)0.6357 (19.44)4 (12.50)0.73Hb (g/L)132.51 ± 20.01133.87 ± 20.260.792127.07 ± 24.71129.97 ± 19.220.838NLR2.78 ± 2.073.37 ± 2.27 0.041 13.75 ± 54.093.12 ± 2.450.989PLT (109/L)253.87 ± 82.45303.60 ± 123.07 0.015 241.87 ± 99.53239.90 ± 72.950.805ALT (U/L)47.49 ± 64.2176.66 ± 113.310.255113.90 ± 319.7894.10 ± 111.80 0.026 AST (U/L)47.85 ± 70.8261.02 ± 88.470.804115.84 ± 320.1387.77 ± 138.23 0.03 Alb (g/L)41.47 ± 4.4040.57 ± 5.700.39738.55 ± 4.4840.10 ± 8.080.486TBil (µmol/L)41.44 ± 82.6361.85 ± 91.030.24927.39 ± 34.0582.10 ± 125.720.101DBil (µmol/L)18.89 ± 46.8938.96 ± 66.70 0.004 9.67 ± 17.8138.67 ± 71.380.218ALP (U/L)150.94 ± 150.02188.89 ± 191.310.214115.60 ± 78.42220.10 ± 253.01 < 0.001 GGT (U/L)167.49 ± 316.46241.64 ± 491.780.26699.52 ± 144.22263.87 ± 405.42 0.011 PT (s)13.08 ± 1.8314.05 ± 1.71 0.021 13.81 ± 1.4514.23 ± 1.610.292INR1.03 ± 0.121.04 ± 0.140.8291.01 ± 0.093.64 ± 14.540.662CEA (ng/ml)5.20 ± 18.4710.69 ± 25.08 0.012 2.34 ± 2.942.54 ± 2.160.283CA19-9 (U/ml)106.89 ± 174.63142.11 ± 173.700.24598.13 ± 289.05516.64 ± 2107.64 0.007 CA125 (U/ml)11.08 ± 9.2724.198 ± 49.10 0.005 12.16 ± 6.7830.89 ± 69.190.202Stones18 (35.29)25 (43.10)0.33117 (47.22)16 (50.00)1.0Differentiation0.385 0.034 Well7 (13.73)3 (5.17)11 (30.56)4 (12.50)Moderate20 (39.22)15 (25.86)13 (36.11)12 (37.50)Poor24 (47.06)40 (68.97)12 (33.33)16 (50.00)R0 resection0.0550.401No18 (35.29)33 (56.90)9 (25.00)12 (37.50)Yes33 (64.71)25 (43.10)27 (75.00)20 (62.50)Morphologic types 0.009 0.126Wall thickening19 (37.25)32 (55.17)14 (38.89)15 (46.88)Intraluminal growth25 (49.02)12 (20.69)18 (50.00)10 (31.25)Mixed tumor7 (13.73)14 (24.14)4 (11.11)7 (21.87)pT stage 0.023 0.003 Tis1 (1.96)NA7 (19.44)1 (3.12)T14 (7.84)1 (1.72)6 (16.67)3 (9.38)T210 (19.61)8 (13.79)13 (36.11)5 (15.63)T336 (70.59)39 (67.24)7 (19.44)15 (46.87)T410 (17.24)3 (8.33)8 (25.00)pN stage 0.037 0.279N031 (60.78)29 (50.00)28 (77.78)20 (62.5)N110 (19.61)18 (31.03)8 (22.22)11 (34.38)N25 (9.81)11 (18.97)NA1 (3.12)Liver invasion < 0.001 0.004 No41 (80.39)25 (43.10)28 (77.78)13 (40.63)Yes10 (19.61)33 (56.90)8 (22.22)19 (59.37)TNM stage 0.005 0.011 01 (1.96)NA8 (22.22)1 (3.12)Ⅰ2 (3.92)1 (1.72)6 (16.67)2 (6.25)Ⅱ7 (13.73)3 (5.17)10 (27.78)5 (15.63)Ⅲ36 (70.59)31 (53.45)9 (25.00)15 (46.88)Ⅳ5 (9.80)23 (39.66)3 (8.33)9 (28.12)Note. Data are mean ± standard deviation or numbers of patients with percentages in parentheses.BMI body mass index, DM diabetes mellitus, Hb hemoglobin, NLR neutrophil to lymphocyte ratio, PLT platelets, ALT alanine aminotransferase, AST aspartate transaminase, Alb albumin, TBil total bilirubin, DBil direct bilirubin, ALP alkaline phosphatase, GGT γ-glutamyl transpeptidase, PT prothrombin time, INR inter-national normalized ratio, CEA carcinoembryonic antigen, CA19-9 carbohydrate antigen19-9, CA125 carbohydrate antigen 125, TNM tumor node metastasis, NA not applicable. Clinical characteristics of patients. Note. Data are mean ± standard deviation or numbers of patients with percentages in parentheses. BMI body mass index, DM diabetes mellitus, Hb hemoglobin, NLR neutrophil to lymphocyte ratio, PLT platelets, ALT alanine aminotransferase, AST aspartate transaminase, Alb albumin, TBil total bilirubin, DBil direct bilirubin, ALP alkaline phosphatase, GGT γ-glutamyl transpeptidase, PT prothrombin time, INR inter-national normalized ratio, CEA carcinoembryonic antigen, CA19-9 carbohydrate antigen19-9, CA125 carbohydrate antigen 125, TNM tumor node metastasis, NA not applicable. Univariate and multivariate analyses identified body mass index, age, platelet count, prothrombin time, pathological nodal stage, and liver invasion as independent risk factors associated with poor prognosis (all p < 0.05), as shown in Supplementary Table S1. Spearman’s correlation analysis revealed no notable correlations among these factors (Supplementary Fig. S3). Based on the selected clinical features, multiple machine learning algorithms were applied to develop and compare clinical prediction models. Detailed model results and AUC–ROCs are presented in Supplementary Tables S2 and Fig. S4, respectively. The LightGBM model demonstrated the best performance, achieving an AUC–ROC of 0.693 (95% confidence interval [CI]: 0.539–0.847) in the validation set, with a sensitivity of 0.645, and an accuracy of 0.673. Therefore, this model was selected as the final clinical prediction model. CT-based deep learning radiomics model Radiomics model A total of 1834 handcrafted radiomics features were extracted from the VOI of each patient, comprising 14 shape, 360 first-order, and 1460 texture features. The quantity and proportion of each feature group are shown in Supplementary Fig. S5a. Of these, 1747 features were retained after interclass correlation coefficient screening. Subsequent t-test analysis yielded 238 features (Supplementary Fig. S5b). Thereafter, Pearson’s correlation analysis excluded 165 features, leaving 73 for further evaluation. Of these, seven features with non-zero coefficients were selected to establish the radiomics signature using the ElasticNet model (Supplementary Fig. S6). The results of the radiomics signatures are presented in Supplementary Table S3, and the AUC–ROCs are shown in Supplementary Fig. S7. Regarding the AUC–ROC, the tree-based models, including the RF, ExtraTrees, XGBoost, and LightGBM, generally outperformed the linear model, the SVM, across the training and validation sets. Of these, the ExtraTrees model achieved the highest AUC–ROC in the validation set at 0.8553 (95% CI: 0.750–0.955), with sensitivity of 0.742 and accuracy of 0.769. Deep transfer learning model The CNN model, DenseNet121, was used to extract deep features from the images of each patient with the largest tumor area. The model was subsequently validated and tested on each dataset. In the validation set, the AUC–ROC, sensitivity, and accuracy were 0.767, 0.645, and 0.692, respectively (Supplementary Table S4). These results suggest that the DenseNet121 model could effectively differentiate between death and survival outcomes. Evaluation of the deep-learning radiomics model A decision-level fusion strategy based on logistic regression was employed to integrate the two distinct image-based prognostic models, resulting in a fused deep-learning radiomics model. The three models were compared, and the results are presented in Supplementary Table S5. The integrated model demonstrated a sensitivity of 0.871 and accuracy of 0.846, outperforming the other models. AUC–ROCs for each model in the training and validation sets are shown in Supplementary Figures S8a and S8b, respectively. The fused model achieved the highest AUC–ROC of 0.883 (95% CI: 0.789–0.978) for predicting 1-year survival in the validation set. WSI-based pathomics model Evaluation of patch-level metrics The predictive performance of a patch-level model was essential, affecting the success of subsequent analytical tasks. Results for the different CNN models are detailed in Supplementary Table S6, and their corresponding AUC–ROCs are shown in Supplementary Fig. S9. The results demonstrated that the Residual Network 101 model performed best in distinguishing between patients who survived and those who died, achieving an AUC–ROC of 0.676 (95% CI: 0.672–0.680). Accordingly, this model was selected for deep-learning-based patch analysis to generate the corresponding prediction values. Evaluation of pathomics model Based on the predictions from the patch-level model, features for each WSI were derived by integrating probability histograms and TF–IDF features. After Pearson’s correlation analysis, 12 pathomics features were retained. Thereafter, these feature vectors were used to train and evaluate five machine learning algorithms. Results for the various machine learning models are depicted in Supplementary Table S7, and their corresponding AUC–ROCs are shown in Supplementary Figure S10. The LightGBM model demonstrated the best performance in distinguishing between patients who died and those who survived, achieving an AUC–ROC of 0.780 (95% CI: 0.657–0.904) in the validation set. This model was designated the final pathomics prediction model. The pathological model achieved an AUC of 0.78 on the validation set, but due to extremely poor calibration, all predicted probabilities were below 0.5, resulting in a sensitivity of 0 which indicates that the model failed to identify any deceased patients under the default threshold of 0.5; its AUC reflects only ranking ability, not practical classification utility. Owing to the inherently limited interpretability of deep learning models, attention mapping was employed to visualize the results and enhance understanding of the model predictions. Figure 3 shows the prediction outcomes for the 164 patients who died and 171 who survived. Figure 3d and h present the predicted probability heat maps of the mortality risk for the two WSIs. Visualization of Histopathological Prediction Results for a Patient Who Died (upper panel) and a Patient Who Survived (lower panel). Note: (a, e) original whole-slide images (WSIs); (b, f) annotated regions of interest (ROIs); (c, g) predicted label distributions; (d, h) prediction probability heat maps. Light-colored areas, such as yellow, indicated regions with a higher predicted probability of the “deceased” label, suggesting that these pathomics features were strongly associated with a poor prognosis. These findings revealed that the pathomics model developed in this study can effectively identify regions within the WSIs that correlate with adverse clinical outcomes. Evaluation of the multimodal fusion prediction model Logistic regression was used to integrate the predicted probabilities of the three single‑modality models. The regression coefficients and standardized weights were as follows: Clinical model: coefficient = 0.87 (standardized weight = 0.31), Deep learning radiomics model: coefficient = 1.54 (standardized weight = 0.64), and Pathomics model: coefficient = 0.12 (standardized weight = 0.05, p = 0.43). These results indicate that the radiomics modality (DLR_Sig) contributed the most to the fusion prediction, followed by the clinical modality, whereas the contribution of the pathomics modality was small and not statistically significant. To enhance clinical applicability, a nomogram was constructed using the multimodal fusion prediction model, as illustrated in Fig. 4. To evaluate whether the integration of multimodal data improved predictive performance, all models were assessed and compared. The results are summarized in Table 2. Nomogram of the Multimodal Fusion Prediction Model. Table 2Specific metric results of each prediction model.ModelAUC95% CISensitivitySpecificityAccuracyPPVNPVDatasetClinic model0.9720.947–0.9980.9480.9150.9330.9320.935Training setDLR model0.9070.854–0.9610.7070.9150.8000.9110.717Path model0.9730.951–0.9950.6381.0000.8001.0000.691Nomogram0.9830.964-1.0000.9480.9360.9430.9480.936Clinic model0.6930.539–0.8470.6450.7140.6730.7690.577Validation setDLR model0.8830.789–0.9780.8710.8100.8460.8710.810Path model0.7800.657–0.9040.0001.0000.4040.0000.404 Nomogram 0.896 0.804–0.987 0.903 0.810 0.865 0.875 0.850 Note. Clinic model clinical prediction model is performed by machine learning algorithms based on clinical features; DLR model deep learning radiomics model based on contrast CT images; Path model pathomics prediction model is performed by multi-instance learning (MIL) approach based on postoperative HE-stained pathological slides. Specific metric results of each prediction model. Note. Clinic model clinical prediction model is performed by machine learning algorithms based on clinical features; DLR model deep learning radiomics model based on contrast CT images; Path model pathomics prediction model is performed by multi-instance learning (MIL) approach based on postoperative HE-stained pathological slides. The nomogram achieved sensitivity, specificity, and accuracy of 0.903, 0.810, and 0.865, respectively. AUC–ROCs for each model in the training and validation sets are presented in Fig. 5a and b, respectively. The nomogram attained the highest AUC–ROC value for predicting 1-year survival in the validation set, reaching 0.896 (95% CI: 0.804–0.987). Depiction of (a), (b) Area Under the Receiver Operating Characteristic Curves (AUC–ROCs) of Different Models on Training and Validation Sets, Respectively. (c), (d) DeLong Tests of Different Models on Training and Validation Sets, Respectively. (e), (f) Calibration Curves of Different Models on Training and Validation Sets, Respectively. (g), (h) Decision Curve Analyses (DCA) of Different Models on Training and Validation Sets, Respectively. DeLong’s test revealed that the nomogram had a significantly higher AUC–ROC than did the clinical model (p = 0.015). No significant differences were observed when compared with the pathomics or deep-learning radiomics prediction models (p = 0.508 and p = 0.077, respectively) (Fig. 5c and d). As shown by the calibration curves (Fig. 5e and f) and Hosmer–Lemeshow test (Supplementary Table S8), the nomogram demonstrated dependable calibration performance. Furthermore, the DCA indicated that the use of the nomogram provided a net clinical benefit (Fig. 5g and h). Nevertheless, only within the threshold probability range of 0.5–0.6 did the pathomics model yield a net clinical benefit, indicating that it is not only difficult for this model to provide a positive net benefit in clinical decision‑making but that it may also be harmful or completely ineffective. This finding is highly consistent with the zero sensitivity and severe calibration failure of the model. Additionally, a Cox proportional hazards regression model was applied for survival analysis. The concordance index (C index) for each model’s prognostic performance in the overall cohort is presented in Table 3. The nomogram achieved the highest C index in the validation set, with a value of 0.736. Table 3C Index of overall samples for survival prediction.DatasetNomogramClinic modelDLR modelPath modelTraining set0.8600.8300.7340.797Validation set0.7480.6690.7290.732Note. C-index concordance index. C Index of overall samples for survival prediction. Note. C-index concordance index. Subsequently, a Kaplan–Meier survival analysis was performed. Patients were stratified into high- and low-risk sub-cohorts based on the calculated risk scores, and a significant difference in survival was observed between the sub-cohorts (p < 0.05; Fig. 6). Kaplan–Meier Survival Plots Showing Patients at High or Low Risk in the (a) Training and (b) Validation Sets. Discussion In this study, a multimodal dataset was collected from patients with postoperative GBAC at two medical institutions. Prognostic models based on CT, histopathological, and clinical factors were developed to predict patient survival. A late-fusion approach using logistic regression integrated these single-modality models into a multimodal fusion model. Thus, multimodal data fusion notably improved survival prediction in these patients, supporting precision treatment. Multivariate analysis identified body mass index, age, platelet count, prothrombin time, pathological nodal stage, and liver invasion as independent risk factors in GBAC. However, limitations included potential inaccuracies and subjectivities in laboratory data. Moreover, postoperative histopathological assessments may vary, and challenges in applying these findings to preoperative prognosis prediction hinder their practical clinical utility. Nonetheless, these results emphasize the importance of aggressive adjuvant therapy in patients with high-risk postoperative factors to improve survival outcomes. With the advancement of cutting-edge technologies and improved computational efficiency, AI in medical imaging has gradually evolved into two branches: machine and deep learning. Previous studies have demonstrated that models developed using both approaches can predict the prognosis of GBAC24,25. Compared with those published models—which mostly rely on clinical or radiomics features and achieve AUCs of around 0.75–0.85—our multimodal model achieved a validation AUC of 0.896, suggesting competitive predictive potential. However, most existing studies used larger sample sizes (often > 200 cases) or external validation, whereas our cohort was limited to 177 cases without external validation. In this study, the conventional radiomics model outperformed the deep-learning model. This difference may be attributed to the use of deep learning for feature extraction from a two-dimensional slice with the largest tumor area, which may omit relevant three-dimensional characteristics. Furthermore, the relative performance of deep-learning methods when compared with traditional machine-learning approaches that rely on human-defined features is highly dependent on the cohort size16. In this study, the data of patients from two different institutions were included. The limited sample size, attributable to the rarity of GBAC, constrained the performance of data-intensive deep learning models. These findings highlight the need for larger cohorts in future studies to improve model robustness and refine prognostic insights. Moreover, a deep learning radiomics model was developed using a late-fusion strategy. This model outperformed two distinct image-based prognostic models. These results align with those of previous studies26–28. Integrating traditional radiomics analysis with deep learning applied to preoperative venous-phase, contrast-enhanced CT images can enhance prognostic prediction for patients. Previous studies have demonstrated that focusing on tumor regions rather than randomly selected slide areas improves the development of prognostic prediction models23,29. Therefore, in this study, only tumor lesions within the WSIs were annotated as target regions for analysis. High-resolution WSIs have provided extensive data for cancer research. However, because of the extremely large size of WSIs, direct processing with CNNs for classification or prediction remains computationally challenging for most systems. A common strategy is to first divide the WSI into smaller patches that serve as inputs to the neural network. The features extracted from these patches are subsequently aggregated to obtain slide-level representations using multiple instance learning (MIL)30. In this framework, individual patches are not labeled; only the entire slide carries a label, making MIL a form of weakly supervised machine learning. This method improves the accuracy of histopathological image analysis and enhances the model’s ability to detect subtle morphological variations. Pathomics has demonstrated promising prognostic value in several cancers, such as hepatocellular carcinoma and lung cancer. To the best of our knowledge, the present study is the first attempt to apply pathomics to GBAC prognosis. Notably, although the model performed poorly—showing zero sensitivity and calibration failure (Hosmer‑Lemeshow test, p = 0.008)—this exploratory effort remains valuable, as it highlights the challenges of transferring pathomics to rare cancers with limited sample sizes. The main reasons for the model’s failure include: (1) instability of probability estimation under small sample size; (2) batch effects between the training and validation sets; and (3) substantial deviation of the default threshold (0.5) from the optimal operating point. These findings further underscore the exploratory nature of this study and strongly highlight the need for larger sample sizes and cross‑center standardization. Tissue biomarkers play crucial roles in cancer diagnosis, prognosis, and treatment guidance. Deep-learning-based computational histopathology offers potential for survival prediction; however, limited interpretability and generalizability restrict widespread clinical adoption. To support histopathologists in identifying novel biological biomarkers using high-performance deep learning models, Liang et al.31 developed PathFinder: an interpretable intelligent platform for mining prognostic tumor biomarkers. Their study applied attribution methods to investigate a well-trained neural network and identify spatial ROIs within the model, enabling the discovery, characterization, and validation of potential biomarkers. By applying this platform to hepatocellular carcinoma WSIs, they have proposed two novel prognostic biomarkers, demonstrating a new paradigm for integrating deep learning into precision medicine. In the present study, probability heat maps were used to enhance the interpretability of the predictive model. This helped identify regions in the histopathological slides that most markedly influence patient prognosis. This approach supports understanding disease biology, guiding clinical decision-making, and improves the explainability of survival prediction models. The multimodal data-fusion strategy in this study used late fusion, additionally known as decision-level fusion. Models were independently developed for each single-modality dataset to generate prediction probabilities, which were subsequently integrated. This approach provides high flexibility, enables the use of the most appropriate analytical method for each modality, and remains robust to missing data in any modality, enhancing clinical feasibility32,33. The nomogram demonstrated superior performance in the validation set than the single-modality prediction models. DeLong’s test revealed a notable difference in the AUC–ROC between the nomogram and the clinical model, whereas no considerable differences were observed when compared with the pathomics or deep-learning radiomics models. This outcome may reflect the heterogeneity of dual-center data and a relatively limited sample size, highlighting the need for larger cohorts in future studies. Additionally, the DCA showed that the nomogram provided a net clinical benefit at prediction probability thresholds > 0.1, whereas the radiomics model showed a benefit > 0.26. This study had some limitations: First, the sample size was relatively small. Due to the low incidence of gallbladder cancer, only 177 cases were included—far below the sample size typically required for multimodal deep learning. This small sample inevitably leads to overfitting. The root cause is that model complexity exceeds the information content of the data, preventing the model from learning true patterns. For small-sample tasks, reducing model capacity and using strong regularization, data augmentation, or transfer learning is essential. We used strong regularization and K‑fold cross‑validation to improve generalizability, but future work must expand the sample size and employ nested cross‑validation or external validation. Second, the ROIs for both CT and WSIs were manually delineated, a process that is time-consuming and may introduce subjectivity. Future exploration of automated segmentation algorithms could improve efficiency and reproducibility. Third, the analysis focused on tumor regions and did not include peritumoral tissue or microenvironmental features. Fourth, molecular modality data, including genomics, were not incorporated. In conclusion, in this study, we developed a nomogram model to predict the postoperative survival of patients with GBAC by integrating preoperative CT and postoperative digital histopathological images, as well as clinical risk factors, through a multimodal data fusion approach. This study serves as a preliminary exploration of multimodal deep learning in a small sample, suggesting the potential of this approach. However, limited by the sample size, the reported performance metrics are likely optimistic due to high overfitting risk, and the model is not yet ready for actual clinical application. Large-scale, multi-center validation is required to assess its true efficacy. Future studies should prioritize external validation through larger populations and integration of additional dimensions of biological information. Supplementary Information Below is the link to the electronic supplementary material. Supplementary Material 1 Acknowledgements We sincerely thank the OnekeyAI platform for code consultation, and Editage (www.editage.cn) for English language editing. Author contributions Correspondence to: Qi LiFan-xiu Meng: Conceptualization, Methodology, Writing - Original Draft, Visualization.Ya-rong Guo: Conceptualization, Methodology.Si-yu Zhang: Validation, Data Curation.Jian-xin Zhang: Resources, Validation.Ling-jie Wang: Methodology, Formal analysis, Visualization.Qi Li: Supervision, Project administration, Conceptualization.All authors read and approved the final manuscript. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Data availability The datasets generated and analysed during the current study are partially included in this published article and its supplementary information files. The original imaging and pathological data are not publicly available due to ethical restrictions but are available from the corresponding author (Qi Li) upon reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets generated and analysed during the current study are partially included in this published article and its supplementary information files. The original imaging and pathological data are not publicly available due to ethical restrictions but are available from the corresponding author (Qi Li) upon reasonable request.