Interpretable multimodal radiopathomics model predicting pathological complete response to neoadjuvant chemoimmunotherapy in esophageal squamous cell carcinoma Authors: Baojia Qi, Zhaoyu Jiang, Haixia Shen, Jiacheng Li, Zhixiang Wang, Min Fang, Changchun Wang, Youhua Jiang, Jingping Yuan, Inigo Bermejo, Andre Dekker, Dirk De Ruysscher, Leonard Wee, Wencheng Zhang, Yongling Ji, Zhen Zhang Published: 2025 ABSTRACT BackgroundAccurate preoperative prediction of pathological complete response (pCR) following neoadjuvant chemoimmunotherapy (nCIT) could help individualize treatment for patients with esophageal squamous cell carcinoma (ESCC). This study aimed to develop and externally validate an interpretable multimodal machine learning framework that integrates CT radiomics and H&E-stained whole-slide images pathomics to predict pCR.MethodsIn this multicenter, retrospective study, 335 patients with ESCC who received nCIT followed by esophagectomy were enrolled from three institutions. Patients from one center were divided into a training set (181 patients) and an internal test set (115 patients), while data from the other two centers comprised an external test set (39 patients). We developed unimodal radiomics and pathomics models, and two multimodal fusion models—an intermediate fusion model (MIFM) and a late fusion model (MLFM). Model performance was evaluated using the area under the curve (AUC), accuracy, sensitivity, specificity, and F1 score, with exploratory survival stratification by observed and model-predicted pCR status. Interpretability was treated as a design constraint and operationalized at both the feature and model levels.ResultsThe MIFM outperformed unimodal models and the MLFM across all cohorts, achieving AUC/accuracy/sensitivity/specificity/F1 score of 0.97/0.93/0.84/0.96/0.86 (training set), 0.78/0.87/0.62/0.93/0.63 (internal test set), and 0.76/0.77/0.54/0.88/0.61 (external test set). Both observed and predicted pCR status showed exploratory prognostic stratification for overall survival. Feature definitions were mathematically or morphologically explicit, and case-level/cohort-level explanations together with decision-pathway views provided insights into model reasoning. We additionally provide a user-friendly Graphical User Interface to facilitate clinical practice.ConclusionsWe developed and externally validated an interpretable radiopathomics fusion framework that predicts pCR after nCIT in ESCC using standard-of-care data. This model holds promise as an effective tool for guiding individualized decisions between surveillance and timely surgery. WHAT IS ALREADY KNOWN ON THIS TOPIC Neoadjuvant chemoimmunotherapy (nCIT) is a promising treatment for esophageal squamous cell carcinoma (ESCC), but accurately predicting pathologic complete response (pCR) remains challenging. Traditional biomarkers have limited predictive capacity and are hindered by high detection costs and operational complexity. Although the role of multimodal radiopathomics in predicting treatment outcomes has been studied in various cancers, its application in nCIT remains limited. Furthermore, the interpretability of predictive models requires further exploration. Neoadjuvant chemoimmunotherapy (nCIT) is a promising treatment for esophageal squamous cell carcinoma (ESCC), but accurately predicting pathologic complete response (pCR) remains challenging. Traditional biomarkers have limited predictive capacity and are hindered by high detection costs and operational complexity. Although the role of multimodal radiopathomics in predicting treatment outcomes has been studied in various cancers, its application in nCIT remains limited. Furthermore, the interpretability of predictive models requires further exploration. WHAT THIS STUDY ADDS This study developed a multimodal radiopathomics model that predicts pCR in patients with ESCC following nCIT by integrating CT-based radiomics and whole-slide images-based pathomics features. The proposed model demonstrated superior performance over unimodal models, achieving high area under the curve, accuracy, sensitivity, specificity, and F1-score across multiple validation cohorts. Interpretability was treated as a design constraint and operationalized at both the feature and model levels. A user-friendly Graphical User Interface is additionally provided to facilitate clinical practice. This study developed a multimodal radiopathomics model that predicts pCR in patients with ESCC following nCIT by integrating CT-based radiomics and whole-slide images-based pathomics features. The proposed model demonstrated superior performance over unimodal models, achieving high area under the curve, accuracy, sensitivity, specificity, and F1-score across multiple validation cohorts. Interpretability was treated as a design constraint and operationalized at both the feature and model levels. A user-friendly Graphical User Interface is additionally provided to facilitate clinical practice. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY This study highlights the potential of multimodal radiopathomics model to improve clinical decision-making for ESCC. The model’s ability to predict pCR could guide individualized decisions between surveillance and timely surgery. Further refinement and validation through large-scale prospective trials remain essential to establish its utility in clinical practice. This study highlights the potential of multimodal radiopathomics model to improve clinical decision-making for ESCC. The model’s ability to predict pCR could guide individualized decisions between surveillance and timely surgery. Further refinement and validation through large-scale prospective trials remain essential to establish its utility in clinical practice. Introduction Esophageal squamous cell carcinomas (ESCC) remain one of the most prevalent and aggressive cancers worldwide.1 Neoadjuvant chemoradiotherapy (nCRT) followed by surgery is the current standard care for locally advanced ESCC.2 3 Recent clinical trials, however, have highlighted neoadjuvant chemoimmunotherapy (nCIT) followed by surgery as a promising alternative, reporting R0 resection rates ranging from 80.9% to 98.0% and pathologic complete response (pCR) rates between 16.7% and 50.0%.4 5 A prospective study further suggested that, compared with nCRT, nCIT may yield superior 2-year overall survival (OS) and disease-free survival (DFS) despite similar pCR rates (22.9% vs 25.9%).6 Achieving pCR correlates with improved long-term survival outcomes and may permit the implementation of watch-and-wait strategies, thereby preserving organ functionality and enhancing quality of life.4 7 8 Consequently, accurate preoperative prediction of pCR following nCIT is critical for identifying suitable candidates and personalizing therapeutic approaches. Despite this clinical need, robust biomarkers capable of accurately predicting pCR to nCIT require further exploration. Established tissue biomarkers, including microsatellite instability,9 10 programmed cell death ligand-1 (PD-L1) expression,11 12 and tumor mutational burden (TMB),13–15 have limited predictive capacity and are hindered by high detection costs and operational complexity. Therefore, there is an urgent need to develop accessible, reliable, and cost-effective predictive tools. Medical imaging provides rich macro-scale and micro-scale information that is well suited to artificial intelligence (AI)-based prediction. Macroscopic radiologic images (eg, contrast-enhanced CT) and microscopic histopathological images (H&E-stained whole-slide images (WSIs)) are complementary, and multimodal fusion may improve predictive accuracy.16 Radiomics and pathomics enable quantitative characterization of tumor phenotype and microenvironment respectively and have shown promise in outcome prediction across cancers, including ESCC.17–19 Building on prior work demonstrating the feasibility of radiomics-based pCR prediction following nCIT,20 and evidence that nuclei-level morphology and texture carry prognostic information,21 22 integrating radiomics and pathomics features represents a rational strategy to enhance preoperative prediction of pCR in ESCC. Translating such multimodal predictors into practice requires more than accuracy. Accordingly, we treat interpretability as a design constraint and frame it along two axes—model-level and feature-level. At the model level, we prioritize algorithms with auditable decision functions and stable post-hoc explanations (eg, Shapley-value attribution), enabling visualization of case-level and cohort-level contributions while mitigating the black-box concerns typically associated with deep neural networks.23 At the feature level, we emphasize mathematically defined radiomics features and explicitly defined pathomics descriptors of nuclear and tissue architecture (eg, nuclear area, eccentricity, perimeter, chromatin texture), selected for their clear clinical semantics and communicability to clinicians. In this study, we developed an interpretable multimodal machine learning framework to preoperatively predict pCR to nCIT in ESCC using data from three independent patient cohorts. We systematically benchmarked multiple machine learning algorithms and fusion strategies to integrate CT-based radiomics and WSI-based pathomics features, while formalizing transparency at both the model and feature levels. To facilitate clinical communication and workflow fit, we specified case-level and cohort-level explanatory outputs (eg, contribution-based attributions) and implemented a user-facing software prototype to illustrate potential applicability and practicality. Methods Given the retrospective design, the informed consent requirements were waived. The study adhered to the principles of the Declaration of Helsinki and followed established methodological guidance for radiomics research.24 To promote methodological rigor and transparency, we evaluated protocol adherence using a 12-item methodology-evaluation checklist that we previously proposed.25 The checklist scoring sheet is provided in the online supplemental table 1. The overarching study flow is presented in figure 1. Study pipeline. Preoperative contrast-enhanced CT and H&E-stained WSIs from 335 patients with ESCC across three centers were analyzed. The tumors were manually contoured on CT images and tumor-rich fields were selected on WSIs. Radiomics (PyRadiomics) and pathomics (CellProfiler) features were extracted and screened. Four predictors were built—unimodal radiomics, unimodal pathomics, MIFM, and MLFM—and evaluated in training, internal, and external cohorts using ROC curves, confusion matrix, reclassification Sankey diagrams, and survival analysis. Interpretability analyses included SHAP analysis, case-level decision-pathway views and cell-type quantification. A browser-based Graphical User Interface accepts the CT/ROI and CellProfiler inputs and outputs the patient-level pCR probability. BNB, Bernoulli Naïve Bayes; ESCC, esophageal squamous cell carcinoma; GNB, Gaussian Naïve Bayes; KNN, k-nearest neighbors; LASSO, Least Absolute Shrinkage and Selection Operator; LR, logistic regression; MIFM, multimodal intermediate fusion model; MLFM, multimodal late fusion model; pCR, pathologic complete response; RF, random forest; ROC, receiver operating characteristic; ROI, region of interest; SHAP, SHapley Additive exPlanations; SMOTE, Synthetic Minority Over-sampling Technique; SVM-RFE, Support Vector Machines-Recursive Feature Elimination; XGBoost, eXtreme Gradient Boosting; WSIs, whole-slide images. Patient enrollment Consecutive patients with histologically confirmed ESCC who received nCIT followed by curative-intent esophagectomy were retrospectively identified across three academic medical centers—Zhejiang Cancer Hospital, Renmin Hospital of Wuhan University, and Tianjin Medical University Cancer Institute and Hospital—from July 2019 to July 2023 (n=335). At Zhejiang Cancer Hospital (n=296), patients were randomly allocated 6:4 to a training set and an independent internal validation cohort (Test-set-1). The external validation cohort (Test-set-2, n=39) comprised patients treated at Renmin Hospital of Wuhan University from July 2020 to September 2023 (n=22) and at Tianjin Medical University Cancer Institute and Hospital from June 2020 to February 2022 (n=17). For each patient, a contrast-enhanced chest CT was acquired within 14 days prior to nCIT initiation, and H&E-stained WSIs were digitized from pretreatment endoscopic biopsy specimens obtained within 7 days of the CT. Detailed inclusion and exclusion criteria and a patient selection flowchart are provided in online supplemental file A1 and figure 1. Treatment protocol and pathological evaluation Patients received at least one cycle of neoadjuvant immunotherapy concurrently with chemotherapy. Immunotherapy consisted of standard doses (200 mg every 3 weeks per cycle) of programmed cell death protein 1 or PD-L1 monoclonal antibodies (tislelizumab, sintilimab, durvalumab, envafolimab, pembrolizumab, camrelizumab, or nivolumab). Platinum-based chemotherapy employed two-drug regimens: (1) TC regimen (every 3 weeks): one to four cycles of nab-paclitaxel 260 mg/m2 (day 1) or paclitaxel 135–175 mg/m2 (day 1) + carboplatin area under the curve (AUC) 5 mg/mL/min (day 1) every 21 days; (2) TP regimen (every 3 weeks): one to four cycles of nab-paclitaxel 260 mg/m2 (day 1) or paclitaxel 175 mg/m2 (day 1) + cisplatin 75 mg/m2 (day 1). Radical esophagectomy was undertaken 4–8 weeks after completion of nCIT. Surgical approach (minimally invasive or open) and lymphadenectomy extent (two-field or three-field) were determined by tumor location and surgeon assessment. Resected specimens were examined by an experienced pathologist and reviewed by a senior esophageal cancer pathologist. Tumor regression grade (TRG) was classified according to the College of American Pathologists Esophageal Carcinoma Protocol26: TRG 0 (no histologically identifiable cancer cells); TRG 1 (single cell or rare small groups of cancer cells); TRG 2 (residual cancer with evident tumor regression but more than single cell or rare small groups of cancer cells); TRG 3 (extensive residual cancer with no evident tumor regression). pCR was defined as TRG 0 at the primary site, with TRG 1–3 being classified as non-pCR. This pCR/non-pCR binary outcome served as the prespecified endpoint for model development and evaluation. Imaging acquisition and segmentation CT acquisition parameters from the three centers are summarized in online supplemental table 2. Two physicians (HS, XW), each with over 3 years of experience, performed manual segmentation of the primary esophageal tumors on CT images to generate regions of interest (ROIs). Assessors were blinded to pathological outcomes and model outputs. All contours were subsequently reviewed and, when necessary, refined by a senior physician (YJ) with over 25 years of experience. Any discrepancies were resolved by consensus adjudication, and the finalized ROIs served as the ground truth for radiomics feature extraction. Segmentations were performed using 3D Slicer software (V.5.1.0).27 Formalin-fixed, paraffin-embedded H&E-stained slides were scanned at 20×magnification and digitized into WSIs. For each WSI, a thoracic pathologist with 3 years of experience (BQ), blinded to clinical outcomes, selected five representative tumor-rich fields of view (FOVs). Each FOV was cropped into a 512×512-pixel patch and saved in PNG format. All patches were visually inspected to guarantee their quality. Visual quality control (QC) was performed to exclude patches with over/under-staining, folds, chatter, inadequate tissue, air bubbles, pen marks or stripping artifacts. Feature extraction and selection Radiomics features were computed from the finalized CT ROIs using PyRadiomics28 (V.3.0.1). A total of 1,094 features were extracted, encompassing shape and size descriptors, first-order intensity statistics, and multiple texture families—gray-level co-occurrence matrix, gray-level size zone matrix, gray-level run length matrix, gray-level dependence matrix and neighboring gray-tone difference matrix—together with wavelet-derived features. Pathomics features were quantified from H&E-stained WSIs using CellProfiler29 (V.4.2.8) via an automated pipeline that measures intensity distributions, neighborhood relationships, morphological/shape attributes, texture statistics, and areas-fraction metrics (details in online supplemental file A2 and figure 2). For each case, features were calculated on the five 512×512-pixel patches and averaged to obtain slide-level descriptors, yielding a total of 4,892 quantitative pathomics features covering nuclear, cytoplasmic, and tissue-level characteristics. Feature selection was conducted independently for radiomics and pathomics, with all procedures confined to the training set. The selection workflow comprised the following steps: first, we used the Synthetic Minority Over-sampling Technique (SMOTE) to address class imbalance. Standardization of extracted radiomics and pathomics features was carried out using Z-Scores (original value−mean value/SD). Then, univariate logistic regression was implemented to identify features with a p value <0.01 (for pathomics features)/0.05 (for radiomics features) for subsequent analysis. Spearman correlation coefficients (ρ) were computed for each pair of features. Redundancy was reduced by computing pairwise Spearman correlations and, for any pair with |ρ| > 0.85, retaining the feature showing the stronger association with the outcome. Finally, two selectors—Least Absolute Shrinkage and Selection Operator (LASSO) with 10-fold cross-validation and Support Vector Machines-Recursive Feature Elimination (SVM-RFE)—were applied separately, and their intersection constituted the modality-specific feature set used for final model construction. A schematic of this pipeline is provided in online supplemental figure 3. Model construction and validation For unimodal modeling, we evaluated seven machine learning algorithms—logistic regression, Gaussian/Bernoulli Naïve Bayes, SVM, random forests, K-nearest neighbors, and eXtreme Gradient Boosting (XGBoost)—for the radiomics and pathomics feature sets. Hyperparameter optimization employed grid search with fivefold cross-validation. A fixed random seed was applied throughout parameter tuning to ensure reproducibility. Key hyperparameters are shown in the online supplemental file A3. For multimodal learning, we considered two fusion strategies. For the multimodal intermediate fusion model (MIFM), radiomics and pathomics features were concatenated into a joint representation, and the algorithm identified as optimal in the unimodal screen was used to fit the fused model on the development set. For the multimodal late fusion model (MLFM), the best-performing radiomics model and pathomics model from the unimodal stage were first trained on the development set. Their probabilistic risk scores were then used as inputs to a logistic regression model. In total, we established four types of models, including the unimodal radiomics models, unimodal pathomics models, MIFM, and MLFM. External validation was performed on the original Test-set-1 and the Test-set-2 with no further tuning. Test-set data was processed strictly through the same preprocessing pipeline fitted on the development set, and no data augmentation (eg, SMOTE), no additional normalization or standardization operations were applied to test datasets. Predictive performance was quantified by the area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, specificity, and F1 score. To obtain CIs, we used 1,000-iteration bootstrap resampling of each test cohort and reported 95% CIs for all metrics. ROC and precision-recall (PR) curves were plotted for visual comparison. Furthermore, for the best-performing model, decision curve analysis (DCA) was performed to assess its potential clinical utility. Model interpretation To interpret the radiomics and pathomics feature contributions to the models, we applied SHapley Additive exPlanations (SHAP) analysis, which quantifies individual feature influence on probability of pCR at both the case level and the cohort level. The Shapley value is defined as follows: ∅j=∑ S⊆N∖{j}|S|!(N−|S|−1)!N!(v(S∪{j})−v(S)) where: v(S∪{j})-v(S) is the specific contribution of j to the coalition S; ∑[S⊆N{j}] is the summation over all possible coalitions; and (S!N-S-1!)/N! is the weight factor of this particular coalition. Case-level plots display how a patient’s specific feature values increase or decrease the predicted risk relative to the model baseline, whereas cohort-level summaries rank features by overall impact and indicate the dominant direction of effect. To provide a stepwise view of the model’s decision logic, we derived patient-specific decision pathways from the trained model. For each patient, the pathway enumerates the sequence of decision rules applied by the model, annotating the feature and threshold at each step and the incremental change in the prediction (on the log-odds/probability scale) contributed by that step, culminating in the final predicted probability. This visualization clarifies which features—and which value ranges—most strongly pushed the prediction toward pCR or non-pCR for that specific case, thereby linking global feature importance with case-level rationale in a single view. The workflow for constructing the contribution plots and decision-path visualizations is presented as a flowchart in online supplemental figure 4. To characterize the tumor microenvironment (TME) on H&E-stained WSIs, we implemented Hover-Net,30 an open-source deep learning network model, for nuclear segmentation and coarse cell-type assignment within the analyzed patches. This framework identified the following cell types: tumor cells, lymphocytes, connective cells, necrotic cells, and other cells. Finally, cell-type fractions were compared based on observed pCR status (observed pCR vs observed non-pCR), and separately, based on model-predicted status (predicted pCR vs predicted non-pCR). Statistical analysis Patient characteristics were evaluated using SPSS V.27. Pearson's chi-square/Likelihood-ratio tests were applied to compare categorical variables. ANOVA/Kruskal-Wallis H tests analyzed the continuous variables. To assess the predictive value of clinical parameters, univariable logistic regression analyses were conducted. Statistical significance was set at a p value of <0.05 for two-tailed tests. Survival was compared between observed pCR and observed non-pCR, and between predicted pCR and predicted non-pCR (Kaplan-Meier with log-rank tests; threshold fixed a priori). Differences in survival outcomes were analyzed using the log-rank test. HRs and 95% CIs were estimated using the Cox proportional hazards model. Survival analysis was conducted using R software V.4.4.2 with the “survival” package V.3.8.3, and results were visualized using the “survminer” package V.0.5.0. All machine learning models were constructed using Python V.3.13.1 with the “scikit-learn” V.1.6.1 and “xgboost” V.3.0.0 packages. Details of main packages can be found in the online supplemental table 3. Results Patient characteristics Patient characteristics stratified by observed response status are summarized in table 1. Overall, 77/335 patients (22.99%) achieved pCR and 258/335 (77.01%) did not. No statistically significant differences were observed in most clinicopathological characteristics, except for smoking status, NCIT cycle and s-LN number (p<0.05). Table 1Patients’ clinical characteristics across all data setsCharacteristicsOverallTraining-setTest-set-1Test-set-2P value(N=335)(n=181)(n=115)(n=39)Sex0.189 Female23 (6.87)13 (7.18)5 (4.35)5 (12.82) Male312 (93.13)168 (92.82)110 (95.65)34 (87.18)Age64 (44–82)64 (44–82)65 (46–77)62 (48–76)0.825Smoking status0.286 Never115 (34.33)57 (31.49)46 (40.00)12 (30.77) Current or former220 (65.67)124 (68.51)69 (60.00)27 (69.23)Drinking status0.047* Never96 (28.66)44 (24.31)35 (30.43)17 (43.59) Current or former239 (71.34)137 (75.69)80 (69.57)22 (56.41)ECOG performance status0.269 0146 (43.58)81 (44.75)44 (38.26)21 (53.85) 1184 (54.93)96 (53.04)70 (60.87)18 (46.15) 25 (1.49)4 (2.21)1 (0.87)0 (0)Tumor location0.251 Upper46 (13.73)26 (14.36)17 (14.78)3 (7.69) Middle169 (50.45)99 (54.70)51 (44.35)19 (48.72) Lower120 (35.82)56 (30.94)47 (40.87)17 (43.59)cT0.127 13 (0.90)2 (1.10)1 (0.87)0 (0) 250 (14.92)29 (16.02)19 (16.52)2 (5.13) 3267 (79.70)145 (80.11)90 (78.26)32 (82.05) 4a15 (4.48)5 (2.77)5 (4.35)5 (12.82)cN0.707 047 (14.03)24 (13.26)16 (13.92)7 (17.95) 1170 (50.75)95 (52.49)60 (52.17)15 (38.46) 2105 (31.34)56 (30.94)35 (30.43)14 (35.90) 313 (3.88)6 (3.31)4 (3.48)3 (7.69)cTNM stage (AJCC Eighth)0.475 I3 (0.90)2 (1.10)1 (0.87)0 (0) II74 (22.09)40 (22.10)26 (22.61)8 (20.52) III228 (68.06)127 (70.16)78 (67.82)23 (58.97) IVA30 (8.95)12 (6.64)10 (8.70)8 (20.51)Immunotherapy regimen0.199 PD-1298 (88.96)159 (87.85)101 (87.83)38 (97.44) PD-L137 (11.04)22 (12.15)14 (12.17)1 (2.56)NCIT cycle<0.001* ≤2271 (80.90)153 (84.53)96 (83.48)22 (56.41) >264 (19.10)28 (15.47)19 (16.52)17 (43.59)R0 resection0.556 No21 (6.27)13 (7.18)7 (6.09)1 (2.56) Yes314 (93.73)168 (92.82)108 (93.91)38 (97.44)Surgical approach0.103 Minimally310 (92.54)171 (94.48)106 (92.17)33 (84.62) Open25 (7.46)10 (5.52)9 (7.83)6 (15.38)Lymphadenectomy extent0.243 Two-field36 (10.75)16 (8.84)13 (11.30)32 (82.05) Three-field299 (89.25)165 (91.16)102 (88.70)7 (17.95)Tumor pCR0.144 No258 (77.01)138 (76.24)94 (81.74)26 (66.67) Yes77 (22.99)43 (23.76)21 (18.26)13 (33.33)ypT stage0.052 077 (22.99)43 (23.76)21 (18.26)13 (33.33) 170 (20.90)33 (18.23)24 (20.87)13 (33.33) 262 (18.51)32 (17.68)23 (20.00)7 (17.96) 3126 (37.60)73 (40.33)47 (40.87)6 (15.38)ypN stage0.465 0189 (56.42)104 (57.46)67 (58.26)18 (46.15) 191 (27.16)44 (24.31)32 (27.83)15 (38.46) 241 (12.24)26 (14.36)10 (8.70)5 (12.83) 314 (4.18)7 (3.87)6 (5.21)1 (2.56)ypTNM stage (AJCC Eighth)0.550 I151 (45.07)83 (45.86)51 (44.35)16 (41.03) II50 (14.93)27 (14.92)20 (17.39)3 (7.69) III134 (40.00)71 (39.22)44 (38.26)20 (51.28)s-LN number (median)23 (5–78)24.0 (6-63)22 (5–78)23 (8–61)0.042*Survival time (median)692 (96–1772)727 (100–1661)716 (96–1172)381 (136–1240)0.137Data are n (%), unless otherwise stated.P value was calculated comparing the Training-set, Test-set-1 and Test-set-2.s-LN number, defined as the number of lymph nodes removed from surgery.*P value below 0.05 was considered statistically significant.AJCC, American Joint Committee on Cancer; cN, clinical node stage; cT, clinical tumor stage; cTNM, Clinical Tumor-Node-Metastasis; ECOG, Eastern Cooperative Oncology Group; NCIT, neoadjuvant chemoimmunotherapy; pCR, pathologic complete response; PD-1, programmed cell death protein 1; PD-L1, programmed cell death ligand 1; s-LN number, surgical lymph node number; ypN, neoadjuvant pathologic node stage; ypT, neoadjuvant pathologic tumor stage; ypTNM, neoadjuvant pathologic Tumor-Node-Metastasis. Patients’ clinical characteristics across all data sets Data are n (%), unless otherwise stated. P value was calculated comparing the Training-set, Test-set-1 and Test-set-2. s-LN number, defined as the number of lymph nodes removed from surgery. *P value below 0.05 was considered statistically significant. AJCC, American Joint Committee on Cancer; cN, clinical node stage; cT, clinical tumor stage; cTNM, Clinical Tumor-Node-Metastasis; ECOG, Eastern Cooperative Oncology Group; NCIT, neoadjuvant chemoimmunotherapy; pCR, pathologic complete response; PD-1, programmed cell death protein 1; PD-L1, programmed cell death ligand 1; s-LN number, surgical lymph node number; ypN, neoadjuvant pathologic node stage; ypT, neoadjuvant pathologic tumor stage; ypTNM, neoadjuvant pathologic Tumor-Node-Metastasis. Performance of the unimodal model After feature selection, 14 radiomics and 11 pathomics features constituted the final unimodal signatures. The definitions of each feature are shown in online supplemental tables 4 and 5. Among the seven unimodal machine learning models, the XGBoost yielded the most consistent discrimination in the Training-set, Test-set-1 and Test-set-2 (figure 2A,B and online supplemental table 6). Therefore, unimodal radiomics and pathomics models were implemented using XGBoost with their respective optimal feature sets. Performance of different models for predicting the pCR. Radar chart comparing the AUC values of seven machine learning algorithms in unimodal pathomics model (A) and unimodal radiomics model (B) in the Training-set. The Sankey diagram depicted reclassification from unimodal predictions to MIFM predictions in reference to the ground truth (C). The ROC curves for the unimodal pathomics model, unimodal radiomics model, MIFM, and MLFM are presented for both the Training-set (D), the Test-set-1 (E) and the Test-set-2 (F). The PR curves for the unimodal pathomics model, unimodal radiomics model, MIFM and MLFM for both the Training-set (G), the Test-set-1 (H) and the Test-set-2 (I). Confusion matrix of the MIFM in the Training-set (J), the Test-set-1 (K) and the Test-set-2 (L). Flow diagrams summarizing MIFM-assigned class versus observed outcome in Test-set-1 (M) and Test-set-2 (N), indicating counts of true positives/negatives and false positives/negatives. AUC, area under the curve; AUPRC, area under the precision-recall curve; BNB, Bernoulli Naïve Bayes; GNB, Gaussian Naïve Bayes; KNN, k-nearest neighbors; LR, logistic regression; MIFM, multimodal intermediate fusion model; MLFM, multimodal late fusion model; pCR, pathological complete response; PR, precision-recall; RF, random forest; ROC, receiver operating characteristic; SVM, Support Vector Machines; XGB, eXtreme Gradient Boosting. The pathomics model achieved AUCs of 0.88 (95% CI 0.82 to 0.94) in the Training-set, 0.68 (95% CI 0.55 to 0.81) in the Test-set-1, and 0.67 (95% CI 0.48 to 0.86) in the Test-set-2. The radiomics model achieved AUCs of 0.90 (95% CI 0.84 to 0.95) in the Training-set, 0.74 (95% CI 0.62 to 0.85) in the Test-set-1, and 0.68 (95% CI 0.51 to 0.85) in the Test-set-2 (figure 2D–F). In the Training-set, the radiomics model showed higher accuracy, sensitivity and specificity (0.84, 0.77 and 0.86, respectively) than the pathomics model (0.81, 0.72 and 0.83, table 2 and online supplemental figure 5). Table 2Performance of the models for predicting pathologic complete responseAUC (95% CI)Accuracy (95% CI)Sensitivity (95% CI)Specificity (95% CI)F1 score (95% CI)Training-set Unimodal pathomics model0.88 (0.82 to 0.94)0.81 (0.75 to 0.86)0.72 (0.58 to 0.85)0.83 (0.77 to 0.89)0.64 (0.52 to 0.74) Unimodal radiomics model0.90 (0.84 to 0.95)0.84 (0.79 to 0.90)0.77 (0.64 to 0.89)0.86 (0.80 to 0.92)0.69 (0.58 to 0.80) MIFM0.97 (0.94 to 0.99)0.93 (0.90 to 0.97)0.84 (0.71 to 0.95)0.96 (0.93 to 0.99)0.86 (0.77 to 0.93) MLFM0.93 (0.88 to 0.97)0.89 (0.85 to 0.93)0.79 (0.67 to 0.91)0.92 (0.87 to 0.96)0.77 (0.68 to 0.86)Test-set-1 Unimodal pathomics model0.68 (0.55 to 0.81)0.68 (0.59 to 0.77)0.52 (0.32 to 0.75)0.71 (0.61 to 0.80)0.37 (0.21 to 0.52) Unimodal radiomics model0.74 (0.62 to 0.85)0.69 (0.61 to 0.77)0.62 (0.39 to 0.83)0.70 (0.61 to 0.80)0.42 (0.26 to 0.57) MIFM0.78 (0.64 to 0.90)0.87 (0.81 to 0.92)0.62 (0.41 to 0.83)0.93 (0.87 to 0.98)0.63 (0.44 to 0.79) MLFM0.77 (0.66 to 0.86)0.70 (0.61 to 0.77)0.57 (0.35 to 0.78)0.72 (0.63 to 0.81)0.41 (0.24 to 0.56)Test-set-2 Unimodal pathomics model0.67 (0.48 to 0.86)0.69 (0.56 to 0.82)0.54 (0.27 to 0.82)0.77 (0.60 to 0.92)0.54 (0.27 to 0.74) Unimodal radiomics model0.68 (0.51 to 0.85)0.59 (0.44 to 0.74)0.54 (0.29 to 0.80)0.62 (0.42 to 0.81)0.47 (0.22 to 0.67) MIFM0.76 (0.55 to 0.94)0.77 (0.64 to 0.90)0.54 (0.27 to 0.83)0.88 (0.74 to 1.00)0.61 (0.33 to 0.82) MLFM0.73 (0.56 to 0.89)0.64 (0.49 to 0.79)0.54 (0.25 to 0.82)0.69 (0.50 to 0.86)0.50 (0.24 to 0.71)AUC, area under curve; MIFM, multimodal intermediate fusion model; MLFM, multimodal late fusion model. Performance of the models for predicting pathologic complete response AUC, area under curve; MIFM, multimodal intermediate fusion model; MLFM, multimodal late fusion model. Given class imbalance, we additionally evaluated PR performance. The pathomics model yielded area under the precision–recall curves (AUPRCs) of 0.73 in the Training-set, 0.37 in the Test-set-1 and 0.50 in the Test-set-2, while the radiomics model achieved AUPRCs of 0.81, 0.45 and 0.55 in the same cohorts (figure 2G–I). The confusion matrix for both unimodal models across test sets is provided in online supplemental figure 6. Performance of multimodal model We developed intermediate-fusion and late-fusion multimodal models using the fusion strategies prespecified in Methods. Across cohorts, both fusion approaches performed better than the unimodal radiomics and pathomics models across evaluation metrics (table 2 and figure 2D–I). Between the two multimodal approaches, MIFM demonstrated higher sensitivity, specificity, accuracy, and F1 score compared with MLFM (table 2 and online supplemental figure 5). For MIFM, the confusion matrix (figure 2J–N) indicated strong exclusion of non-pCR cases: true negatives numbered 87 in the Test-set-1 and 23 in the Test-set-2, and specificity was the highest among all models (table 2). The Sankey diagram depicted reclassification from unimodal predictions to MIFM predictions in reference to the ground truth, highlighting the net movement toward correct labels (figure 2C). The DCA was illustrated in online supplemental figure 7. Exploratory prognostic stratification by observed and model-predicted pCR status We examined whether observed pCR status and model-predicted pCR status (from the MIFM at its fixed operating threshold) stratified OS. In the Training-set, patients with observed pCR showed longer OS with visible separation of Kaplan-Meier curves (figure 3A), whereas this separation did not reach statistical significance in the Test-set-1 or Test-set-2 (figure 3C,E). Stratifying patients by the model’s predicted status yielded a qualitatively similar pattern but did not achieve statistical significance in any cohort (figure 3B,D and F). Univariate Cox regression analysis confirmed that observed pCR, and model-predicted pCR were significantly associated with OS (p value<0.005; online supplemental figure 8). Prognostic stratification performance. KM curves for OS stratified by observed pCR versus non-pCR in the Training-set (A), Test-set-1 (C) and Test-set-2 (E) and stratified by MIFM predicted pCR versus non-pCR in the Training-set (B), Test-set-1 (D) and Test-set-2 (F). KM, Kaplan-Meier; MIFM, multimodal intermediate fusion model; OS, overall survival; pCR, pathologic complete response. Interpretability analyses and software prototype SHAP summaries illustrated the contribution of individual radiomics and pathomics features to the predictions of the MIFM (figure 4A). Across all features retained in the final model, cross-modality correlations were mild to low (figure 4B and online supplemental figure 9), with values between −0.57 and 0.69, suggesting that the two modalities may capture complementary aspects of tumor biology and treatment response. Interpretability analyses. (A) The SHAP summary of the MIFM: right, beeswarm plot showing per-feature effects on model output; left, bar chart ranking global importance by mean |SHAP|; inset shows proportional contributions. (B) Cross-modality Spearman correlation network for features retained in the final model (edge width reflects correlations). (C) Boxplots of WSI-derived cell-type fractions (tumor, lymphocyte, stromal, necrotic, other): left, comparison by observed status (observed pCR vs observed non-pCR); right, comparison by model-predicted status (predicted pCR vs predicted non-pCR). (D) Case vignettes illustrating discordant unimodal predictions reconciled by MIFM (Patient A, observed pCR) and concordant unimodal predictions (Patient B, observed non-pCR); for each, feature maps, top-20 contribution ranking, and the decision-pathway depict how modality-specific features accumulate to the final prediction. MIFM, multimodal intermediate fusion model; pCR, pathologic complete response; SHAP, SHapley Additive exPlanations; WSI, whole-slide image. We compared the cell-type fractions by observed pCR status and, separately, by model-predicted pCR status (figure 4C). Relative to their respective non-pCR groups, both the observed pCR group and the predicted pCR group tended to show higher tumor and lymphocyte fractions and lower necrotic fractions. Figure 4D presents two representative samples to make the decision process transparent. In one case, the unimodal radiomics and pathomics models offered discordant opinions, and the fused model reconciled these by weighting modality-specific evidence; in the other, both modalities were concordant. For each sample, feature maps displayed the spatial distribution of salient imaging and tissue descriptors (online supplemental figure 10), per-feature contribution plots show how specific values pushed the prediction toward pCR or non-pCR, and a decision-pathway view summarizes the stepwise reasoning leading to the final probability. Together, these views link feature-level signals with the model’s case-level rationale. Finally, we provide a browser-based, Graphical User Interface tool that requires no coding. Users upload the pretreatment CT, the corresponding tumor ROI mask, and the CellProfiler-derived comma-separated values (CSV) file; the tool returns the patient-level predicted probability of pCR along with basic input checks. A concise user guide and screenshots are available in the online supplemental figure 11. (This prototype is intended for research use). Discussion In this multicenter retrospective study across three academic hospitals, we developed and externally validated an interpretable multimodal framework that integrates routinely available contrast-enhanced CT radiomics with H&E-stained WSI pathomics to preoperatively predict pCR after nCIT in ESCC. Compared with unimodal radiomics or pathomics, the intermediate-fusion model showed more robust discrimination across the development set and two validation cohorts, underscoring the value of combining complementary imaging and tissue information. The work’s key strengths are its clinical practicality—it relies solely on standard-of-care data without additional testing or cost—and its interpretability-first design, where feature definitions are mathematically or morphologically explicit and case-level/cohort-level explanations (eg, SHAP summaries, feature maps) together with decision-pathway views render model reasoning transparent. Finally, we provide a browser-based Graphical User Interface tool that requires no coding and returns a patient-level pCR probability, facilitating exploratory use in multidisciplinary settings. Together, these elements highlight a feasible and transparent pathway toward translating multimodal AI into decision support for nCIT in ESCC. The chemoradiotherapy for esophageal cancer followed by surgery study (CROSS) trial established the superiority of nCRT over surgery alone for locally advanced ESCC.31 Nonetheless, distant metastasis remains the dominant mode of failure after nCRT—far exceeding local recurrence (22.0% vs 5.9%)—underscoring the need for enhanced systemic therapies to improve outcomes. Recently, it was shown that intensive chemotherapy improved the OS and local control over nCRT.32 33 These improved regimens could lead to the omission of esophagectomy in patients achieving a pCR after induction therapy.34 However non-invasive ways to determine pCR are not available. In the era of immunotherapy, multiple studies have reported that the combination of immunotherapy and chemotherapy has achieved favorable outcomes as the first-line treatment for advanced esophageal cancer,35 36 suggesting translational potential in the neoadjuvant setting. Head-to-head comparisons of nCIT versus nCRT in locally advanced ESCC are still accruing. In a prospective multicenter study across eight high-volume centers, Guo et al reported superior 2-year OS (81.3% vs 71.3%) and DFS (73.9% vs 63.4%) with nCIT compared with nCRT, while pCR rates were similar (22.9% vs 25.9%) and major pathologic response favored nCRT (61.5% vs 71.8%).6 Taken together, although the optimal neoadjuvant strategy is not yet settled, current evidence indicates substantial promise for chemoimmunotherapy in this population. This evolving landscape motivates the present work. In routine care, pCR can only be histologically confirmed postoperatively. For patients likely to achieve pCR after neoadjuvant therapy, a watch-and-wait strategy may avert unnecessary esophagectomy, preserve organ function, and improve quality of life.37–39 Conversely, for patients unlikely to achieve pCR, proceeding to timely esophagectomy to eradicate residual disease remains the standard curative pathway. Consequently, there is a clear clinical need for an accurate, preoperative, non-invasive predictor of pCR to guide individualized decision-making between surveillance and prompt surgery. Our multimodal, interpretability-constrained framework directly addresses this gap, aiming to inform neoadjuvant pathways in ESCC. Integration of multimodal data has emerged as a promising approach for predicting treatment response across various cancers. For example, Mao et al combined pretreatment MRI, WSIs, and clinical risk factors to predict pCR following neoadjuvant chemotherapy in breast cancer,40 although with deep learning-derived features whose semantics were less explicit. For ESCC, Qi et al reported that incorporating CT images and WSIs could predict pCR after nCIT, supporting the utility of multimodal fusion41; however, pathomics features were extracted by deep learning models, and the paired CT-WSI cohort was relatively limited (n=89). Against this backdrop, our study provides, to our knowledge, one of the largest multimodal evaluations of pCR prediction after nCIT in ESCC, with paired CT and biopsy WSIs across three centers and external validation. By employing mathematically defined radiomics features and explicitly defined morphologic/texture descriptors from WSIs, the framework enhances feature-level interpretability while maintaining competitive discrimination. compared with unimodal models, the MIFM achieved higher specificity across all cohorts (table 2), indicating strong exclusion of non-pCR cases. Clinically, such operating characteristics primarily reduce the risk of misclassifying non-pCR as pCR, thereby avoiding inappropriate surveillance and supporting timely surgery for those unlikely to achieve pCR. Conversely, patients predicted as pCR could be considered for cautious watch-and-wait, where confirmatory assessment and close monitoring are in place. Prospective studies are warranted to determine thresholds and workflows that safely translate these findings into practice. To reduce risk inherent to high-dimensional features and modest cohort sizes,42 43 we implemented a rigorous, training-only feature screening pipeline. From 1,094 radiomics features and 4,892 pathomics features, we applied two complementary selectors—LASSO and SVM-RFE—and used the intersection of their selected features for the final model construction. For multimodal models, we implemented intermediate-fusion and late-fusion techniques to integrate the 14 selected radiomics features and 11 selected pathomics features. While some prior reports favor late-fusion models for its robustness,44 our results indicate that intermediate-fusion—which directly models complementary information across modalities—can yield superior specificity in this setting (table 2 and figure 2D–I). We hypothesize that retaining original, modality-specific feature information and explicitly leveraging cross-modality complementarity facilitates building predictors that are both discriminative and robust. We further used SHAP to interpret our machine learning model, quantifying the global influence of each feature. For instance, radiomics feature R8 (wavelet.HHL_glszm_SizeZoneNonUniformityNormalized) captures textural heterogeneity—higher values indicate a more uneven distribution of same-intensity zones after high-frequency filtering, which could be related to mixed viable/necrotic components or perfusion variability. Radiomics feature R4 (wavelet.LLH_firstorder_90Percentile) summarizes the brightest voxel intensities in the low–low–high (LLH) sub-band. Higher values may indicate areas of rich vascular supply and active tumor proliferation, while lower values could correspond to necrotic or low-density regions. Among pathomics, P6 (Texture_InverseDifferenceMoment_Hematoxylin) quantifies the uniformity of the staining intensity of hematoxylin within the nuclear region, with higher values indicating high nuclear heterogeneity, while lower values indicate low nuclear heterogeneity. Pathomics feature P11 (Mean_FilteredNuclei_AreaShape_HuMoment) quantifies nuclear asymmetry and morphological heterogeneity, with higher values indicating greater nuclear irregularity and lower values reflecting more symmetrical nuclear morphology. Nuclear morphology reflects cellular proliferation status and abnormal development, where generally enlarged or irregular nuclei are associated with malignant potential.45 These key features provide, to some extent, biologically plausible links between image-derived measurements and tumor phenotype, helping bridge model outputs with clinical reasoning. We evaluated case-level interpretability using the decision-pathway views in figure 4D, making the stepwise reasoning of the tree-based learner explicit. Patient A (observed pCR) represents a discordant unimodal scenario: unimodal radiomics and pathomics models predicted pCR (risk score: 0.00224) and non-pCR (−0.00016), respectively, while MIFM correctly predicted pCR. Feature map panels display the three most influential radiomics and pathomics features, and a top-20 contribution ranking shows radiomics features predominating for this case. The decision pathway traces how successive radiomics thresholds progressively increased the cumulative score above the fixed operating threshold, while several pathomics features exerted negative contributions toward non-pCR. Patient B (observed non-pCR) illustrates a concordant scenario: both unimodal models correctly predicted non-pCR (radiomics: −0.00408; pathomics: −0.00232), and MIFM also yielded accurate predictions. Here, the contribution histogram shows a more balanced mix of radiomics and pathomics influences, and the decision pathway depicts cumulative decrements that keep the prediction below the threshold. These two examples not only enhance the transparency of the decision-making process but also suggest a complementary and synergistic relationship between macroradiological information and micropathological features. The TME constitutes a complex network comprising diverse components including cancer cells, stromal cells, blood vessels, nerve fibers, and extracellular matrix.46 47 This system plays a crucial role in tumor progression, prognosis, and response to immunotherapy.48 In our cohort, we quantified cell-type fractions on H&E-stained WSIs (tumor/epithelial, lymphocyte, stromal/spindle, necrotic, other) and compared distributions by observed pCR status and, separately, by model-predicted status (figure 4C). In both comparisons, the pCR groups tended to exhibit higher tumor and lymphocyte fractions and lower necrotic fractions relative to their respective non-pCR groups—directionally consistent with prior reports linking viable tumor architecture and lymphocytic infiltration to treatment sensitivity.49 50 Although these trends did not reach statistical significance, plausibly reflecting limited sample size, biopsy sampling variability, and potential variability in Hover-Net segmentation, they are biologically plausible and hypothesis-generating, warranting confirmation in larger, prospective datasets. Although the results are encouraging, our study has several limitations. First, the retrospective design and limited external validation cohorts may introduce potential bias, necessitating prospective validation in larger populations. Second, established predictive biomarkers for immunotherapy efficacy—including TMB, PD-L1 expression levels, and combined positive score—were excluded from our analysis because these tests were not uniformly available and would add cost; integrating such markers could further improve performance. Third, despite predefined procedures, manual segmentation and visual QC for CT and WSIs inevitably introduce subjectivity. Developing and validating automated QC and segmentation pipelines should be prioritized. Fourth, no stain normalization or color augmentation was used in this study, and these strategies should be explored in future work. Fifth, statistically significant Kaplan-Meier separation was observed only for observed pCR in the training cohort, and the lack of statistical significance in the other cohorts may be attributable to the limited number of deaths. In addition, pCR is not the only determining prognostic factor for long-term survival. Therefore, these findings should be considered exploratory and hypothesis-generating. Finally, although our discussion of transparency in this study focuses on both the machine learning model level and the handcrafted feature level, interpretable deep learning could potentially enhance both model performance and transparency in future work. Furthermore, the biological hypotheses generated from this study remain preliminary, and genomic-level evidence will be essential to validate and substantiate these interpretations. In summary, we developed and externally validated an interpretable multimodal machine learning framework that integrates contrast-enhanced CT radiomics with H&E-stained WSI pathomics to preoperatively predict pCR after nCIT in ESCC. Our findings demonstrate the clinical potential of this multimodal approach for guiding individualized decisions between surveillance and timely surgery. Further refinement and validation through large-scale prospective trials remain essential to establish its utility in clinical practice.