Development and external validation of an interpretable multimodal deep learning model for 5-year mortality in high-risk stage ii colorectal cancer Authors: Xin Li, Lei Liang, Zhong-Hua Liu, Chun Wang, Tawfik Ali Hamood Alburiahi, Zhen-Ya Yang, Ning Xu, Jun Yang Published: 2026 ABSTRACT AbstractPurposeHigh-risk stage II colorectal cancer (CRC) shows heterogeneous outcomes despite adjuvant chemotherapy. We developed and validated an interpretable multimodal deep learning model integrating clinical data, serum biomarkers, and venous-phase CT to predict 5-year CRC-specific mortality in high-risk stage II CRC.MethodsThis retrospective, multicenter cohort included 778 high-risk stage II CRC patients from three centers, all treated with adjuvant chemotherapy and with complete preoperative clinical, biomarker, and venous-phase CT data. Patients were split into a development cohort (Centers A + B, n = 720) and an external testing cohort (Center C, n = 58). A multimodal model combining numerical (clinical + biomarker) and imaging (CT) inputs was developed and internally validated using tenfold cross-validation in the development cohort and evaluated in the external cohort. Interpretability was assessed using SHAP and Grad-CAM.ResultsIn the development cohort, the multimodal model showed superior discrimination (AUC 0.89; 95% CI, 0.87–0.91) versus numerical-only (AUC 0.76) and imaging-only (AUC 0.69). In the external testing cohort (9/58 CRC-specific deaths), the multimodal model achieved an AUC of 0.88 (95% CI, 0.76–0.96). SHAP and Grad-CAM consistently highlighted age, CA125, and tumor regions on CT as key contributors.ConclusionThis interpretable multimodal approach, using routine clinical, biomarker, and CT data, improves 5-year mortality risk stratification in high-risk stage II CRC and may inform risk-adapted surveillance and clinical decision support; prospective validation is warranted before treatment modification.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00384-026-05132-8.Keywords: Colorectal cancer, High-risk stage II, CA125, Computed tomography, Deep learning, Prognostic prediction Introduction Colorectal cancer (CRC) is a major cause of global cancer mortality [1]. While early-stage CRC often has a good prognosis, stage II CRC exhibits heterogeneous outcomes, with some patients experiencing recurrence despite adjuvant chemotherapy [2–4]. Accurate risk stratification in this population is crucial for guiding treatment decisions, and we address this by focusing on 5-year CRC-specific mortality, a clinically relevant endpoint for assessing prognosis [5]. Traditional TNM staging has limitations in predicting outcomes for high-risk stage II CRC [6]. Recent studies have explored clinicopathological factors, molecular markers, and imaging features to improve risk prediction [7–9]. However, applying single factors clinically is limited [10]. Advances in artificial intelligence (AI) and deep learning offer new opportunities, particularly in integrating multimodal data [11, 12]. While MRI is a preferred method for local staging in rectal cancer, we leverage computed tomography (CT) which is more widely available and offers a broad overview of the disease including abdominal involvement and distant metastasis [13]. We focus specifically on upper rectal tumors above the peritoneal reflection where CT imaging has higher reliability. Deep learning-based analyses of CT images combined with clinical data have shown promise in predicting CRC outcomes [14, 15]. However, studies often suffer from single-center limitations, lack external validation, and lack interpretability. Importantly, the field is also moving toward prospective multicenter validation of colorectal AI tools, as exemplified by the CLASSICA software-as-medical-device trial evaluating real-time AI-enabled approaches in rectal cancer [16]. Recent MITAT contributions also highlight AI-driven standardization of endoluminal assessments, reinforcing the growing readiness of AI to augment colorectal decision-making across the continuum of care [17]. Given the expanding use of minimally invasive approaches in older colorectal cancer patients, the need for robust personalized prognostic models remains critical to align perioperative strategy and downstream adjuvant and surveillance pathways with true risk [18]. We aimed to develop and externally validate a multivariable model integrating clinical, laboratory, and venous-phase CT features to predict 5-year CRC-specific mortality in high-risk stage II colorectal cancer. The model integrates clinicopathological factors, serum biomarkers, and contrast-enhanced CT imaging features. Utilizing SHAP (SHapley Additive exPlanations) and Grad-CAM (Gradient-weighted Class Activation Mapping) methods will support clinical translation [19]. We aim to develop a tool for clinicians to re-stratify risk among chemotherapy-treated high-risk stage II patients and inform risk-adapted surveillance or clinical-trial consideration using a model developed in multicenter data with robust validation, to address the lack of such tools in existing clinical practice and literature. Methods Staging framework and cohort definition All cases were staged according to the AJCC Version 8 (V8) TNM system. High-risk stage II was defined in line with the prespecified statistical plan and contemporary guidelines as the presence of ≥ 1 of the following: < 12 lymph nodes examined, poor differentiation (except MSI-H), lymphovascular invasion, perineural invasion, bowel obstruction or perforation, and elevated preoperative CEA. Study design and patient population This retrospective, multicenter study (10-year period) involved three centers: Center A (n = 553), Center B (n = 481), and Center C (n = 58). Based on the AJCC V8 definition above, a total of 1,092 patients with high-risk stage II colorectal cancer were initially identified (Fig. 1A). This study was conducted and reported in accordance with the TRIPOD-AI guideline. Study workflow illustrating: a Data collection and division into training/testing sets; b Numerical and imaging feature selection; c Multimodal feature extraction using Dense ResNet and Vision Transformer modules; d tenfold cross-validation model training and evaluation (ROC-AUC); e Interpretability analyses using SHAP for numerical and Grad-CAM for imaging features The initial comparative analysis used the Center A cohort (n = 553), stratified by receipt of postoperative adjuvant chemotherapy: chemotherapy (n = 307) vs non-chemotherapy (n = 246). Kaplan–Meier analyses compared overall survival (OS) and disease-free survival (DFS) between these groups. This analysis was conducted to contextualize our cohort within existing evidence regarding chemotherapy benefit and is not the primary focus of this study. For model development, we focused on the 307 patients from Center A who received adjuvant chemotherapy; 239 patients with complete clinical and imaging data were retained for feature selection. Using these cases, LASSO (Least Absolute Shrinkage and Selection Operator) regression identified 15 numerical features associated with 5-year CRC-specific mortality (covering demographics, oncologic biomarkers and inflammatory markers). For validation, we then collected data on these same 15 features from all eligible patients in Center B (n = 481) and Center C (n = 58) who met all inclusion criteria, crucially including the receipt of adjuvant chemotherapy. Together with the Center A modeling set, a combined cohort of 778 patients was used to develop and validate the predictive model. Preoperative contrast-enhanced CT (venous-phase) was incorporated to provide imaging-based features. A complete-case analysis strategy was used; only patients with complete preoperative clinical, biomarker, and venous-phase CT data required for modeling were included, while cases with missing clinical or imaging data were excluded rather than imputed. Inclusion criteria were: (i) postoperative pathological confirmation of AJCC V8 stage II colorectal cancer with ≥ 1 high-risk feature (as defined above); (ii) availability of preoperative laboratory data and venous-phase contrast-enhanced CT; (iii) for all patients in the final modeling and validation cohorts, completion of adjuvant chemotherapy; (iv) colonic tumors and upper rectal tumors above the peritoneal reflection; and (v) adequate follow-up to ascertain 5-year CRC-specific mortality (≥ 5 years for survivors or death within 5 years). Patients lost to follow-up before 5 years without a documented CRC-specific death event were excluded. Exclusion criteria were incomplete clinical or imaging data, poor-quality CT images or insufficient follow-up to ascertain 5-year CRC-specific mortality (≥ 5 years for survivors or death within 5 years). This study was approved by the ethics committees of all participating centers, and patient data were anonymized. Outcome definition: 5-year CRC-specific mortality The primary outcome was 5-year CRC-specific mortality, defined as death directly attributable to CRC within five years post-surgery. The model was designed as a fixed-horizon binary classifier to predict whether CRC-specific death occurred within 60 months after surgery; it was not developed as a time-to-event survival model. Accordingly, no censored observations were included in the final modeling dataset. Cause of death was determined from hospital records, death certificates, and follow-up notes. Deaths were classified as cancer-specific if they resulted from: 1) liver metastasis leading to hepatic failure; 2) intestinal obstruction or perforation causing diffuse peritonitis and sepsis; 3) tumor cachexia and wasting leading to severe malnutrition and multi-organ failure; 4) widespread metastases (e.g., respiratory failure due to lung metastases); or 5) tumor-related complications such as pulmonary embolism from tumor emboli or massive gastrointestinal bleeding from tumor invasion of major vessels. This determination was independently reviewed by two clinician evaluators who were blinded to the model’s predictions. In cases of disagreement (which occurred for < 5% of deaths), consensus was reached through discussion with a third senior reviewer. For transparency, we report that the inter-reviewer agreement for cause of death adjudication was high (kappa = 0.92). Data collection and feature selection Clinical and laboratory data collected within 7 days before surgery included demographic characteristics (e.g., age, sex, BMI), tumor features (e.g., tumor size, tumor type), inflammatory markers (e.g., NLR, SII), and oncological biomarkers (e.g., CEA, CA199, CA125). Preoperative abdominal contrast-enhanced CT venous-phase images were obtained for all patients using standardized imaging protocols across the three centers. All original CT images underwent appropriate preprocessing to minimize batch effects arising from different hospitals and scanners. Figure 1A illustrates the data collection process. To identify the most relevant features for predicting 5-year CRC-specific mortality, feature selection was performed for both numerical and imaging data (Figs. 1B and 2). Numerical features were processed using LASSO (Least Absolute Shrinkage and Selection Operator) regression, which selected 15 key predictors, including age, CA125, and tumor size. Crucially, feature selection (using LASSO regression) was performed solely on the 239-patient subset from Center A. The resulting 15 numerical features were then locked and extracted from patients in Centers B and C, ensuring no data leakage from the development or external test sets. For imaging data, a semi-automatic process was employed: three senior radiologists manually annotated slices containing visible tumor regions, which were used to train a ResNet16 model. The trained model automated the identification of tumor-containing slices for the rest of the dataset, ensuring consistency and efficiency. Feature extraction workflow: a Numerical data selection via LASSO and processing with Dense ResNet module; b Imaging data semi-automatic tumor selection using ResNet16, followed by Vision Transformer-based feature extraction To minimize batch effects arising from different hospitals and scanners, all CT images underwent rigorous preprocessing. First, three senior radiologists with over 10 years of experience in gastrointestinal imaging independently reviewed all venous-phase contrast-enhanced CT images. For each patient, they selected the axial slice containing the maximum tumor cross-sectional area, ensuring that the selected images were free of motion artifacts and that the tumor boundaries were clearly distinguishable from surrounding normal tissue. In this multicenter retrospective dataset, we used the axial slice containing the maximum tumor cross-sectional area as the imaging input because it provided a standardized and clinically informative representation of the tumor while reducing variability related to heterogeneous acquisition parameters, slice thickness, and volumetric coverage across scanners and centers. This approach also reduced annotation burden and facilitated more consistent tumor localisation across cases. Due to variations in the original image dimensions across different scanners and centers, we standardized all selected images to ensure consistency for model input. Each image was resized to 224× 224 pixels using bicubic interpolation, which preserves image smoothness while minimizing information loss. Pixel intensities were then normalized to a range of [0, 1] by dividing by the maximum pixel value. All scans were contrast-enhanced with a portal venous phase delay of 70 s, and slice thickness ranged from 1.25 to 5 mm across centers. Consequently, each patient contributed a single 2D CT slice for model input, enabling patient-level prediction without the need for slice aggregation. Multimodal prediction model The multimodal deep learning model integrated numerical and imaging features to predict 5-year CRC-specific mortality, as illustrated in Figs. 1C and 2. Numerical features were processed using a Dense ResNet Module (DRM), which consisted of four Dense-Dropout blocks connected via residual layers to extract key features while minimizing overfitting and improving gradient flow. Imaging features were extracted using a Vision Transformer (ViT) module. CT slices were divided into fixed-size patches, embedded into a high-dimensional feature space, and passed through multi-head self-attention mechanisms to capture both global and local dependencies. Numerical and imaging features were concatenated and passed through two separate task-specific fully connected layers: one for binary classification. The model focused on predicting 5-year mortality as a binary classification problem. CrossEntropyLoss was used to optimize the binary classification output. The training process utilized the shared numerical and imaging features. Hyperparameters were empirically selected within the development cohort based on preliminary experiments and validation performance. The final model configuration is summarized in Supplementary Table S1. Model training and validation The combined data from Centers A and B (n = 720) constituted the development set. This set was randomly divided into training (80%) and internal validation (20%) sets for hyperparameter tuning and tenfold cross-validation. The external test set consisted of the 58 patients from Center C, which was held out and used only for the final performance evaluation reported in the results. The model was trained using CrossEntropyLoss as the loss function, the Adam optimizer with a learning rate of 0.001, and a batch size of 32. These hyperparameters were chosen based on preliminary experiments in the development cohort to achieve stable validation performance. Training was conducted for 50 epochs per fold. To ensure robust performance, a tenfold cross-validation approach was applied (Fig. 1D). All experiments were conducted on a workstation equipped with an Intel i9-13900 K CPU, NVIDIA RTX 4090 GPU (24 GB), and 128 GB RAM, using PyTorch (2.0.0), Torchvision (0.15.0), and Scikit-learn (1.3.2). Feature selection and all preprocessing steps (e.g., normalization parameters for numerical features, image resampling parameters) were determined exclusively using the training data within each cross-validation fold. The external test set (Center C, n = 58) was held out and used only once for the final performance evaluation after all model development and tuning was complete. External validation and uncertainty Discrimination in the external cohort (Center C) was quantified using ROC-AUC with 95% confidence intervals estimated by bootstrap resampling (2,000 iterations). Calibration in Center C was assessed using the Brier score and logistic recalibration without updating the original model parameters, reporting calibration-in-the-large (CITL) and calibration slope. Given the limited sample size, we additionally provide a binned calibration summary (quartiles of predicted risk) in the Supplementary Appendix. In addition, graphical calibration curves were generated to visually assess agreement between predicted and observed risks, and decision curve analysis was performed to evaluate the potential clinical net benefit of the multimodal model across a range of threshold probabilities. Model evaluation Model performance was evaluated using standard classification metrics, including ROC-AUC, accuracy, recall, precision, and F1 score. To assess the contribution of each modality, ablation studies compared the performance of the numerical-only (DRM) and imaging-only (ViT) models to the multimodal fusion model. To formally compare the AUCs of the different models, we used the DeLong test. As shown in Fig. 1D, the multimodal model achieves the highest performance with an AUC of 0.89, significantly outperforming the numerical-only model (AUC = 0.76) and the imaging-only model (AUC = 0.69). Model interpretability To ensure transparency and clinical relevance, SHAP and Grad-CAM were used to interpret the model’s predictions (Fig. 1E). SHAP analysis quantified the contributions of numerical features to the model’s predictions, highlighting key predictors such as CA125, Tumor size, and Age. Grad-CAM visualized the tumor regions in CT slices that contributed most to predictions. The generated heatmaps were qualitatively assessed by two senior radiologists blinded to the clinical information, who confirmed a high degree of spatial concordance between the high-activation regions in the heatmaps and the tumor contours in their manual annotations. Results Exploratory analysis of adjuvant chemotherapy benefit To confirm the association between adjuvant chemotherapy and survival outcomes in this study cohort, we conducted exploratory survival analyses. The patient selection process for this study involved a series of steps to define the study cohort (Supplementary Table S2). This retrospective, multicenter study initially screened 553 patients from Center A diagnosed with high-risk stage II colorectal cancer (Step 1). To validate the benefit of postoperative adjuvant chemotherapy for these patients, we stratified them into two groups based on their receipt of adjuvant chemotherapy (Step 2): 307 patients in the chemotherapy group and 246 patients in the non-chemotherapy group. Kaplan–Meier survival analysis shows that patients receiving postoperative adjuvant chemotherapy demonstrate a statistically significant improvement in both overall survival (OS) and disease-free survival (DFS) compared to those who did not (log-rank test, p < 0.05; Fig. 3), consistent with prior clinical trial evidence. This result supported the use of adjuvant chemotherapy in this high-risk population. Kaplan–Meier curves demonstrating significant improvement in a overall survival (OS) and b disease-free survival (DFS) among patients receiving adjuvant chemotherapy compared with those without chemotherapy (log-rank test, p < 0.05) Enhanced risk prediction in high-risk stage II colorectal cancer: Feature selection and model development While the initial analysis confirmed the survival benefits of adjuvant chemotherapy, it also highlighted a critical need: despite treatment, a significant proportion of high-risk stage II colorectal cancer patients continue to experience adverse outcomes, most notably 5-year CRC-specific mortality. This underscores the inherent limitations of current risk stratification approaches, and the necessity for more precise methods to identify patients who remain at high risk, despite treatment, who would most benefit from adjunctive therapies. The details of the complete clinical, laboratory, and imaging data available for these patients from Center A are found in Supplementary Table S3. To that end, we developed a multimodal deep learning predictive model capable of integrating clinical, laboratory, and imaging features, with a specific goal of distinguishing between those patients likely to experience poor prognoses despite chemotherapy. Our initial step for the development of this model focused on the 239 patients from Center A, for whom comprehensive clinical data were available. A feature importance analysis, using SHAP (Shapley Additive Explanations) values, was performed to rank all numerical features associated with 5-year CRC-specific mortality. This process, utilizing LASSO (Least Absolute Shrinkage and Selection Operator) regression, identified 15 key features that were most strongly associated with the study’s outcome. As depicted in Fig. 4, which presents a summary of feature importance, age (V1), CA125 (V19), urea (V37), and RBC (V12) emerged as the top predictors. SHAP values indicated that high values of CA125 and age were associated with increased mortality risk, while lymphocyte count (V10) was associated with a decreased risk. The resulting 15 numerical features, particularly age and CA125, showed strong predictive power and formed the backbone of the numerical-only model, and were also then used in subsequent modeling. SHAP summary plot displaying the contributions of the top 15 numerical features to model predictions. Features are ranked by importance, with colors indicating feature value magnitudes (red = high, blue = low). The features are defined as follows: V1 = Age (patient age in years); V5 = Platelet-to-Lymphocyte Ratio (PLR); V10 = Lymphocyte Count (number of lymphocytes in blood); V11 = Monocyte Count (number of monocytes in blood); V12 = Red Blood Cell Count (number of red blood cells in blood); V17 = AFP (Alpha-fetoprotein); V19 = Cancer Antigen 125 (CA125) (tumor marker); V22 = Tumor Size (maximum tumor dimension); V23 = Chemotherapy Cycle (number of cycles); V29 = AST/ALT; V33 = TBA (Total bile acid level in serum); V35 = GGT(Gamma-Glutamyl transferase); V36 = CHE(Cholinesterase); V37 = Urea (serum urea level); V39 = UA (serum uric acid level) Multimodal model development: integrating clinical features and CT imaging To develop and validate our multimodal predictive model for 5-year CRC-specific mortality, we extended our patient cohort beyond Center A. As described in the Supplementary Table S2, which outlines the patient selection process, we collected data on the 15 key numerical features, previously identified, from two additional centers: 481 patients from Center B and 58 patients from Center C (Step 5). Preoperative contrast-enhanced CT venous-phase images were also collected for all 778 patients. These data were then combined with the Center A feature data, with patients from Centers A and B used as a training set and patients from Center C used as a test set (Step 6). In the development set (Centers A and B, n = 720), there were 77 CRC-specific mortality events (10.7%) and 643 non-events (89.3%) within five years. In the external test set (Center C, n = 58), there were 9 CRC-specific mortality events (15.5%) and 49 non-events (84.5%) within five years. Table 1 summarizes the descriptive statistics of the 15 numerical features for the training, testing, and total cohorts. These key characteristics of our patient cohort are vital for assessing the validity and generalizability of our model. The median follow-up duration was 73.0 months (IQR, 53.0–96.2 months; range, 3.0–150.0 months). The lower bound of the range reflects patients who experienced CRC-specific death early after surgery rather than survivors with incomplete follow-up. All patients alive without the study endpoint had at least 5 years of follow-up, whereas patients who died from CRC within 5 years were included even if their total follow-up duration was shorter. Table 1Fifteen key numerical features of the combined cohort, with descriptive statistics for the training set (A + B), testing set (C), and total cohort (A + B + C)FeatureTraining Set (A + B)Testing Set (C)Total (A + B + C)Age65.00 (55.00–72.00)62.00 (54.00–68.00)65.00 (55.00–71.00)Platelet-to-Lymphocyte Ratio (PLR)136.44 (90.65–243.43)187.95 (112.61–310.57)140.43 (92.83–266.37)Lymphocyte Count1.55 (1.10–2.40)1.52 (1.19–2.08)1.55 (1.10–2.47)Monocyte Count0.40 (0.25–0.60)0.36 (0.22–0.58)0.40 (0.25–0.60)Red Blood Cell Count4.67 (4.00–5.07)4.76 (4.20–5.30)4.68 (4.04–5.16)KI-672.15 (1.49–3.70)2.95 (1.97–4.39)2.20 (1.51–3.91)Cancer Antigen 125 (CA125)12.08 (5.80–18.10)14.44 (7.04–25.27)12.30 (5.90–18.70)Tumor Size2.50 (2.00–5.00)4.00 (2.20–5.00)3.00 (2.00–5.00)Chemotherapy Cycle3.00 (1.00–6.00)3.00 (1.00–5.00)3.00 (1.00–6.00)Albumin1.36 (0.78–1.86)1.47 (0.99–1.99)1.37 (0.79–1.90)Total Protein4.50 (4.00–7.10)4.90 (4.00–8.52)4.80 (4.10–7.30)Globulin21.00 (15.00–34.20)21.50 (16.76–33.17)21.00 (15.12–34.60)A/G Ratio6.10 (4.60–8.20)6.20 (4.87–8.40)6.10 (4.70–8.30)Urea4.46 (3.36–5.81)5.52 (4.10–6.95)4.56 (3.46–6.07)Creatinine304.75 (231.20–391.12)279.85 (163.67–377.09)301.35 (231.08–392.22) Fifteen key numerical features of the combined cohort, with descriptive statistics for the training set (A + B), testing set (C), and total cohort (A + B + C) To enhance the predictive capabilities of our model beyond the 15 key numerical features, we integrated CT imaging data, which enabled us to capture spatial information not available from clinical and laboratory data alone. To assess how our model utilized CT imaging features, Grad-CAM (Gradient-weighted Class Activation Mapping) was applied to visualize regions of the CT images that most contributed to the model’s predictions of 5-year CRC-specific mortality. As shown in Fig. 5, the model focuses primarily on tumor regions and their immediate surroundings. The Grad-CAM heatmaps overlaid on the original CT slices highlighted areas with the highest predictive weight, which were highly consistent with areas identified by radiologists in their manual annotations. These findings confirm that the model focused on biologically relevant areas and provided complementary spatial information relevant to long-term outcomes. Grad-CAM heatmaps showing CT regions predictive of mortality across colon segments: a Ascending; b Transverse; c Descending; d Sigmoid. Highlighted areas correlate strongly with radiologist-identified tumor regions Multimodal model achieves superior predictive performance The predictive performance of the three models—numerical-only, imaging-only, and multimodal—was evaluated using standard classification metrics, including ROC-AUC, accuracy, recall, precision, and F1 score across tenfold cross-validation (Table 2). The numerical-only model, which used the Dense ResNet Module (DRM) for feature extraction, achieved a mean AUC of 0.76 (Fig. 6A and Table 2). Although this model effectively utilized key features such as age and CA125, its performance was limited due to the lack of spatial information from imaging data. The imaging-only model, based on the Vision Transformer (ViT), achieved a mean AUC of 0.69 (Fig. 6B and Table 2). While this model successfully extracted relevant structural features from CT images, its reliance on imaging alone resulted in lower predictive power compared to the numerical-only model. The multimodal model, which fused numerical and imaging features, demonstrated superior performance with a mean AUC of 0.89 (Fig. 6C and Table 2). It also attained the highest mean weighted precision (0.91), recall (0.92), and F1-score (0.91). The multimodal model significantly outperformed both the numerical-only model (AUC = 0.76) and the imaging-only model (AUC = 0.69) (DeLong test p < 0.001 for both comparisons). By integrating complementary information from both clinical numerical and spatial information from imaging data, the multimodal approach significantly outperformed the single-modality models. This improvement underscores the value of combining numerical and spatial features for more accurate predictions. The multimodal model achieved the highest predictive accuracy, highlighting the value of combining numerical and imaging data. Table 2Performance of the multimodal prediction modelModelFoldWeighted average precisionWeighted average recallWeighted average F1-scoreAUROCImage-based prediction model (ViT)10.750.690.710.7020.700.660.690.6730.630.600.610.5940.680.650.670.6550.720.680.730.6960.830.790.830.8170.630.640.630.6280.650.650.640.6390.760.720.740.73100.750.730.740.72Average0.710.680.700.69Value-based prediction model (DRM)10.760.720.750.7320.730.710.720.6930.810.790.800.7740.700.710.710.6850.780.770.770.7560.820.800.810.7870.830.820.810.8080.750.750.750.7390.840.820.830.80100.830.810.820.80Average0.790.770.780.76Multi-modal fusion prediction Model10.920.870.890.8720.940.940.940.9130.960.950.940.9340.940.930.910.9050.930.940.910.9060.860.930.890.8670.860.920.900.8680.870.910.890.8690.920.910.910.89100.860.930.890.86Average0.910.920.910.89 Performance of the multimodal prediction model ROC curves comparing predictive performance: a Numerical-only model (AUC = 0.76); b Imaging-only model (AUC = 0.69); c Multimodal model (AUC = 0.89), showing superiority of multimodal integration External validation and calibration in center C In the external testing cohort from Center C (n = 58), there were 9 CRC-specific deaths (15.5%) within 5 years. The multimodal model achieved an AUC of 0.88 (95% CI, 0.76–0.96; bootstrap, 2,000 resamples) for predicting 5-year CRC-specific mortality. Using the prespecified operating threshold fixed from the development cohort (threshold = 0.35), the model yielded a sensitivity of 0.67 (6/9) and specificity of 0.86 (42/49). Positive and negative predictive values were 0.46 (6/13) and 0.93 (42/45), respectively. Overall accuracy was 0.83 (48/58) with an F1 score of 0.55. The corresponding confusion matrix is TP = 6, FP = 7, TN = 42, and FN = 3 (Table 3). Table 3External validation performance of the multimodal model in Center C (n = 58)MetricValueEvent rate (CRC-specific mortality)9/58 (15.5%)AUC (95% CI)0.88 (0.76–0.96)Threshold applied (fixed from development cohort)0.35Sensitivity0.67 (6/9)Specificity0.86 (42/49)PPV0.46 (6/13)NPV0.93 (42/45)Accuracy0.83 (48/58)F1 score0.55Confusion matrixTP = 6; FP = 7; TN = 42; FN = 3Brier score0.114Calibration slope0.71Calibration-in-the-large (CITL) − 0.25 (approx.) External validation performance of the multimodal model in Center C (n = 58) Calibration assessment in Center C showed a Brier score of 0.114. The calibration slope was 0.71 and CITL was approximately − 0.25, suggesting that predicted risks were somewhat overly extreme and slightly overestimated on average in the external cohort. A binned calibration summary by quartiles is provided in Supplementary Table S4. The calibration curve showed reasonable agreement between predicted and observed risks in the external cohort, although some overestimation was observed at higher predicted risk levels, consistent with the calibration slope and CITL results. Decision curve analysis shows that the multimodal model provides greater net benefit than the treat-all and treat-none strategies across low-to-intermediate threshold probabilities (Figs. 7 and Supplementary Figure S1). Calibration and decision curve analysis of the multimodal model in the external validation cohort. a Calibration curve showing agreement between predicted and observed 5-year CRC-specific mortality risk. The dotted line represents ideal calibration. b Decision curve analysis showing the net benefit of the multimodal model compared with the treat-all and treat-none strategies across threshold probabilities Discussion This study successfully developed and validated a multimodal deep learning model to predict 5-year CRC-specific mortality in high-risk stage II colorectal cancer, encompassing both colon and upper rectal tumors. By integrating clinical, biomarker, and contrast-enhanced CT imaging data, the model improved risk stratification over clinical-only or imaging-only models [20–23]. The interpretability provided by SHAP and Grad-CAM methods enhanced the model’s transparency, highlighting clinically relevant predictors such as CA125, age, and tumor location. Key strengths of this study include its multicenter design, robust tenfold cross-validation, and the use of routinely collected clinical and imaging data. The model’s applicability is further supported by its interpretability, which aligns well with clinical decision-making needs [24]. By focusing on upper rectal tumors—where CT imaging is reliable—this model remains grounded in practical, widely available diagnostic tools. Despite its strengths, this study has several limitations. First, its retrospective design introduces potential selection bias. In addition, the complete-case design may have introduced selection bias and reduced the representativeness of the final analytic cohort. Although standardized preprocessing was applied to reduce inter-center and inter-scanner variability, residual heterogeneity may still remain and may have affected model generalizability. Second, although CT imaging is suitable for upper rectal tumors, it remains inferior to MRI for detailed local staging, particularly in lower rectal cancers [25, 26]. Another limitation is that the imaging model was based on a single axial slice rather than the full 3D tumor volume. Although this improved feasibility and consistency, it may not fully capture spatial heterogeneity. Future studies should explore 3D approaches in larger multicenter datasets. Third, while the model focused on binary mortality outcomes, incorporating more granular time-to-event data in future work could enhance prediction accuracy. Fourth, As the model was trained solely on patients receiving adjuvant chemotherapy, it is not applicable for predicting the prognosis of untreated patients or guiding decisions regarding the initiation of chemotherapy. Its clinical value lies in the risk re-stratification of high-risk stage II patients already scheduled for chemotherapy. Fifth, the interpretability methods used have inherent limitations: Grad-CAM provides qualitative heatmaps that are useful for visualization but do not quantify feature importance, and the prominence of CA125 in SHAP analysis should be interpreted cautiously, as CA125 is not a standard cornerstone biomarker in stage II CRC and may reflect confounding by systemic inflammation or other unmeasured factors. Prospective validation is required before any treatment adaptation based on this model can be considered. Last, the external testing cohort (Center C) was relatively small (n = 58) with only 9 CRC-specific death events, which limits statistical power and increases uncertainty around both discrimination and calibration estimates. Although external discrimination remained favorable, the confidence intervals and calibration metrics should be interpreted cautiously, and confirmation in larger prospective multicenter cohorts is still required before clinical deployment or treatment adaptation. The calibration slope < 1 and negative CITL suggest that predicted risks may be overly extreme and slightly overestimated on average in the external cohort. Larger multicenter and prospective validations, with potential model recalibration, are warranted before clinical deployment or treatment adaptation. Prior studies have explored radiomics and AI-based approaches in CRC prognosis [8, 10, 27]. For example, Badic et al. [28] integrated CT-derived morphological features with clinical data (AUC = 0.75), and Fan et al. [7] incorporated radiomics with MMR status (AUC = 0.954). Our model adds value using a Vision Transformer to autonomously extract spatial features, avoiding bias from handcrafted feature selection. Grad-CAM visualization further verified that the model concentrated on biologically plausible tumor regions, supporting the reliability of its predictions. Multimodal data integration proves especially beneficial in this context. Clinical and biomarker data alone often lack the granularity to capture tumor heterogeneity, while imaging alone may miss systemic features such as inflammation or nutritional status. As endoluminal and robotic platforms evolve, the integration of AI-driven analytics is expected to accelerate, making interpretable, multimodal models particularly attractive for adoption within next-generation minimally invasive colorectal workflows [29]. In contrast to prior studies that relied on resource-intensive assays—such as RNA sequencing or ctDNA—our model uses routinely available variables such as age, CA125, and preoperative CT, reducing costs and increasing clinical accessibility. Although promising molecular tools such as ctDNA and dMMR status may improve prognostic precision, they are not yet universally available [30, 31]. Our approach thus offers an immediately applicable framework in diverse healthcare settings. Importantly, current guidelines regarding adjuvant chemotherapy for high-risk stage II CRC remain inconsistent [24, 32]. A reliable, interpretable tool such as ours could therefore inform discussions with patients regarding follow-up intensity or eligibility for trials of novel adjunctive therapies—particularly for those who remain at high residual risk despite standard chemotherapy. Future work should prioritize prospective validation in external cohorts and real-world workflows. Integrating molecular signatures, socioeconomic determinants, and time-to-event endpoints may improve precision. Finally, coupling the model with clinical decision support systems and monitoring calibration over time could facilitate clinician trust and adoption. Conclusion We developed and validated an interpretable multimodal deep learning model that predicts 5-year CRC-specific mortality in AJCC V8 high-risk stage II colorectal cancer. By combining clinical, biomarker, and CT features, it outperformed single-modality models and may support risk-adapted surveillance and clinical trial selection among high-risk stage II patients already scheduled for chemotherapy; prospective multicenter validation and potential recalibration are warranted before clinical deployment. Author contributor Conceptualization: Methodology: Xin Li and Lei Liang; Formal analysis and investigation: Xin Li, Lei Liang and Zhong-Hua Liu; Writing—original draft preparation: Xin Li, Lei Liang and Chun Wang; data curation: Zhong-Hua Liu, Tawfik Ali Hamood Alburiahi, Zhen-Ya Yang; Writing—review and editing: Jun Yang and Ning Xu; Funding acquisition: Jun Yang; Supervision: Jun Yang and Ning Xu; project administration: Jun Yang and Ning Xu. All authors have read and agreed to the published version of the manuscript. Supplementary Information Below is the link to the electronic supplementary material. (DOCX 304 KB) (DOCX 31.3 KB) Author contributions Conceptualization: Methodology: Xin Li and Lei Liang; Formal analysis and investigation: Xin Li, Lei Liang and Zhong-Hua Liu; Writing - original draft preparation: Xin Li, Lei Liang and Chun Wang; data curation : Zhong-Hua Liu, Tawfik Ali Hamood Alburiahi, Zhen-Ya Yang;Writing - review and editing: Jun Yang and Ning Xu; Funding acquisition: Jun Yang; Supervision: Jun Yang and Ning Xu; project administration: Jun Yang and Ning Xu. All authors have read and agreed to the published version of the manuscript. Funding This work was supported by the National Natural Science Foundation of China (No.82560569); The Yunnan Revitalization Talent Support Program (No. RLQB20200004 and RLMY20220013), and Education teaching research project of the First Affiliated Hospital of Kunming Medical University (No.2024-JY-19). Science and Technology Projects of Yunnan Universities Serving Key Industries (No.FWCY-BSPY2025076). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data availability Due to patient privacy and ethical restrictions imposed by the participating institutions, the raw data supporting the findings of this study are not publicly deposited. De-identified data and the custom code used for model development are available from the corresponding author for reasonable research requests, pending institutional review and data transfer agreements. Declarations Ethics approval This research study was approved by the Ethics Committee of The First Affiliated Hospital of Kunming Medical University (2024-L-124), the Ethics Committee of The Third People’s Hospital of Honghe Prefecture(2024-KYXM-29) and the Ethics Committee of The Third Affiliated Hospital of Kunming Medical University (KYLX2024-146). Consent to participate Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Xin Li and Lei Liang contributed equally to this study. Contributor Information Ning Xu, Email: xuning@kmmu.edu.cn. Jun Yang, Email: yangjun6@kmmu.edu.cn. References Associated Data Supplementary Materials (DOCX 304 KB) (DOCX 31.3 KB) Data Availability Statement Due to patient privacy and ethical restrictions imposed by the participating institutions, the raw data supporting the findings of this study are not publicly deposited. De-identified data and the custom code used for model development are available from the corresponding author for reasonable research requests, pending institutional review and data transfer agreements.