Transformer-based Siamese multimodal imaging model for predicting pathological complete response after neoadjuvant chemoradiotherapy in locally advanced rectal cancer Authors: Jing Xu, Xin Chen, Yeting Hu, Suyun Li, Qichun Wei, Sebastian M Dieter, Bohai Feng Published: 2026 ABSTRACT AbstractBackgroundAccurate prediction of pathological complete response (pCR) following neoadjuvant chemoradiotherapy (nCRT) remains a critical yet unresolved challenge in locally advanced rectal cancer (LARC). Reliable noninvasive biomarkers could facilitate organ-preserving strategies and help avoid unnecessary radical surgery.MethodsWe retrospectively analyzed 440 patients with LARC from three centers who underwent nCRT and multimodal magnetic resonance imaging (MRI). A total of 1,197 handcrafted radiomic features were extracted using PyRadiomics, while deep learning features were derived from a Med3D-pretrained ResNet101 and subsequently compressed via principal component analysis. To integrate pre- and post-treatment MRI features, we developed a transformer-based Siamese multimodal framework. Model performance was systematically evaluated using receiver operating characteristic (ROC) curves, decision curve analysis, calibration, and survival outcomes.ResultsIn the independent validation cohort, the Siamese multimodal transformer achieved the highest predictive performance (AUC = 0.789), outperforming early-fusion machine learning models (AUC = 0.629) and single-timepoint transformer models (AUC = 0.739 and 0.728). Decision curve analysis demonstrated superior net clinical benefit, and calibration curves indicated close agreement between predicted and observed outcomes. Notably, in the training cohort, transformer-based risk stratification was significantly associated with progression-free survival (p < 0.05).ConclusionsWe propose and validate a transformer-based multimodal imaging framework that effectively integrates Siamese multimodal features to enable individualized prediction of pCR in LARC. These findings underscore the potential of transformer architectures for noninvasive risk stratification and provide a novel approach to inform precision treatment strategies.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12885-026-16049-4.Keywords: Locally advanced rectal cancer, Neoadjuvant chemoradiotherapy, Pathological complete response, Multimodal MRI, Transformer architecture Introduction Locally advanced rectal cancer (LARC) remains a major clinical challenge, accounting for a substantial proportion of colorectal cancer cases worldwide [1, 2]. Neoadjuvant chemoradiotherapy (nCRT) -alone or as part of total neoadjuvant therapy concept which includes an additional section of neoadjuvant chemotherapy- followed by total mesorectal excision has become the standard treatment for patients with LARC, resulting in improved local control and sphincter preservation [3]. However, pathological complete response (pCR) after nCRT occurs in only 15–30% of patients, and the ability to accurately predict pCR remains limited [4]. Patients who achieve a pCR to nCRT demonstrate superior prognosis compared to patients without pCR (non-pCR) [5]. Besides a favourable prognosis, reliable biomarkers of treatment response are essential to identify patients who may benefit from nonoperative management and organ-preserving strategy, such as the “watch-and-wait” strategy, thereby avoiding unnecessary surgical morbidity without compromising oncological outcomes [6, 7]. Magnetic resonance imaging (MRI) is the preferred imaging modality for staging and image-based response assessment in rectal cancer [8, 9]. In clinical practice, multimodal MRI typically includes high-resolution T2-weighted imaging (T2WI), which provides detailed anatomical delineation of tumor extent and mesorectal involvement, and contrast-enhanced T1-weighted imaging following gadolinium administration (GD), which reflects tumor vascularity and treatment-induced perfusion changes. These complementary sequences capture distinct but interrelated aspects of tumor biology, including structural heterogeneity and vascular remodeling after chemoradiotherapy. However, conventional qualitative interpretation of these multimodal images remains subjective and lacks standardized criteria for accurately identifying pathological complete response [10]. Beyond conventional radiological interpretation, radiomics has emerged as a powerful tool that converts medical images into high-dimensional quantitative features, enabling the characterization of tumor heterogeneity and prediction of therapeutic outcomes [8, 11, 12]. Several studies have demonstrated that radiomic signatures derived from pre- or/and post-treatment MRI can serve as predictive biomarkers for pCR and survival in rectal cancer [8, 12–16]. In addition, recent deep learning-based frameworks have further explored automated tumor analysis on post-nCRT MRI, highlighting the growing role of advanced image modeling techniques in rectal cancer assessment [17]. In parallel, deep learning approaches using convolutional neural networks (CNNs) and three-dimensional models, such as Med3D, have been increasingly applied to medical imaging, offering automated feature extraction and improved predictive performance [18]. Despite the promise of radiomics and deep learning, existing studies have primarily assessed them separately or through basic fusion strategies, leaving the comprehensive integration of handcrafted radiomics and deep learning features across pre- and post-nCRT largely unaddressed. These limitations underscore the need for more advanced architectures capable of modeling complex multimodal and longitudinal imaging data. Recently, transformer architectures have shown remarkable success in computer vision and natural language processing by leveraging self-attention mechanisms to capture long-range dependencies [19]. In medical imaging, transformers have been adapted for tasks such as lesion detection, segmentation, and outcome prediction, demonstrating superior performance compared to conventional CNNs [20]. Moreover, Siamese networks, which allow the direct modeling of pairwise or temporal differences, are particularly suited for analyzing pre- and post-treatment imaging data, making them promising candidates for treatment response assessment [21]. Nevertheless, the comprehensive integration of handcrafted radiomics and deep learning features within a transformer-based Siamese multimodal framework across pre- and post-nCRT remains largely unaddressed. Therefore, in this study, we aimed to develop and validate a transformer-based framework that integrates handcrafted radiomics and deep learning features extracted from pre- and post-treatment multimodal MRI. Our findings highlight the potential of transformer-based Siamese approaches to improve individualized risk assessment and guide clinical decision-making in LARC. Methods Study participants This retrospective study included 440 patients with clinical stage II-III (AJCC 8th edition [22]) LARC treated across three tertiary hospitals in China. All patients received standard nCRT, consisting of radiotherapy delivered at 50–57.5 Gy to the gross tumor volume and 45 Gy to the clinical target volume in 25 fractions, combined with concurrent 5-fluorouracil-based chemotherapy administered orally or intravenously [23]. Six to eight weeks after completion of nCRT, patients underwent clinical restaging with digital rectal examination, endoscopy, and rectal magnetic resonance imaging, followed by curative surgical resection. Details of the inclusion and exclusion criteria are provided in Fig. 1. Flowchart depicting the patient selection process across three participating centers. Note: SAHZU, The Second Affiliated Hospital Zhejiang University School of Medicine; GDPH, Guangdong Provincial People’s Hospital; GZFPH, Guangzhou First People’s Hospital Pathological assessment and definition of pCR Pathological response was evaluated according to the tumor regression grade (TRG) system based on examination of the surgically resected specimens following neoadjuvant chemoradiotherapy. The pCR status was uniformly defined as the absence of residual viable tumor cells in both the primary tumor and regional lymph nodes (ypT0N0). All resected specimens were assessed by experienced gastrointestinal pathologists at each participating institution. To ensure consistency across centers, identical standardized pathological criteria were applied at all three institutions for TRG evaluation and pCR determination. MRI data preprocessing and lesion segmentation MRI examinations were performed using 3.0-T scanners. Image preprocessing and lesion segmentation were conducted with 3D Slicer (version 5.0.3) and Python (version 3.7.12). To harmonize multicenter MRI data, N4 bias field correction was applied to reduce intensity inhomogeneity, and voxel sizes were resampled to isotropic dimensions of 1 × 1 × 1 mm3 using B-spline interpolation. Spatial registration across sequences was achieved using the General Registration (Elastix) module in 3D Slicer. Regions of Interest (ROIs) of the tumor (pre-treatment) or primary tumor bed (post-treatment) were manually delineated slice by slice on GD and T2WI by 3 experienced radiologists in consensus, with reference to other MRI sequences, operative notes, and histopathological records to ensure accuracy. MRI radiomics feature extraction Radiomics features were extracted using PyRadiomics [24] in accordance with the Imaging Biomarker Standardization Initiative (IBSI) guidelines. Features were extracted from multiple image types, including Original, Laplacian of Gaussian (LoG; sigma = 2.0–5.0), and Wavelet-filtered images. The feature categories comprised shape features, first-order statistics, and texture features derived from gray-level co-occurrence matrix (GLCM), gray-level run-length matrix (GLRLM), gray-level size-zone matrix (GLSZM), gray-level dependence matrix (GLDM), and neighboring gray tone difference matrix (NGTDM). For each ROI, 1,197 features were computed, encompassing shape descriptors, first-order intensity statistics, texture features, and higher-order features. Feature extraction was performed separately for each MRI sequence (T2WI and GD), restricted to the manually delineated habitats of the primary lesion. Detailed parameter settings for radiomics feature extraction are available in the PyRadiomics documentation (https://pyradiomics.readthedocs.io/en/latest/). 3D deep learning feature extraction A 3D ResNet101 model pre-trained on the Med3D dataset [18] was employed to extract volumetric features. The 3D architecture preserves stereoscopic anatomical information, enabling richer contextual representations than conventional 2D CNNs. Specifically, the network follows a residual bottleneck design with stacked 3D convolutional layers, batch normalization, and ReLU activation functions, allowing effective hierarchical feature learning while mitigating gradient degradation in deep architectures. The model was implemented in PyTorch and initialized with publicly available Med3D pretrained weights. Single-channel volumetric inputs were used for each MRI sequence. For each patient, the tumor ROI was enclosed in a bounding cube, resampled to 64 × 64 × 105 voxels, and augmented by axis inversion. The input dimensions were chosen to preserve the full craniocaudal tumor extent while standardizing in-plane resolution. The pretrained network was used as a fixed feature extractor without additional fine-tuning. Deep representations were obtained from the penultimate average pooling layer, where 2,048 features were extracted and subsequently reduced to 256 principal components using Principal Component Analysis (PCA). The first 256 principal components retained > 99.8% of the total variance across all sequences (pre-GD: 99.97%, post-GD: 99.96%, pre-T2: 99.88%, post-T2: 99.82%), indicating minimal information loss. These high-dimensional representations were subsequently used for downstream integration with handcrafted radiomics features. Construction of early fusion-based machine learning model Feature-level fusion was performed by concatenating radiomics and deep learning features into a unified vector. Radiomics features of the primary tumor were extracted with PyRadiomics, while volumetric deep learning features were obtained from the pre-trained Med3D ResNet101 and reduced via PCA, as described above. All features were standardized using Z-score normalization. To avoid data leakage, feature selection procedures were conducted exclusively within the training cohort. Feature relevance and reproducibility were assessed using Spearman correlation analysis and intraclass correlation coefficients (ICCs), respectively. Subsequently, Least Absolute Shrinkage and Selection Operator (LASSO) regression was applied within the training set to select the most informative features, and the selected features and corresponding parameters were then applied to the validation cohort without refitting. Finally, Naïve Bayes, k-nearest neighbor (KNN), and ExtraTrees classifiers were implemented to construct feature-fusion models. Transformer-based Integration of Siamese multimodal features We adopted a transformer-based framework to integrate radiomics and deep learning features from pre- and post-treatment Siamese multimodal MRI. The model is built upon a multi-head attention (MHA) encoder [25], which efficiently captures global dependencies across longitudinal and cross-modal representations. Within this architecture, the self-attention mechanism adaptively emphasizes the most informative features, which are subsequently fused into a comprehensive representation of multimodal imaging data. The fused vector is then passed to the classification layer to predict pCR following neoadjuvant chemoradiotherapy. Further technical details of the MHA-based multimodal integration are provided in the Supplementary Methods. Model training and implementation details All deep learning models were implemented in PyTorch. The transformer-based framework was optimized using stochastic gradient descent (SGD) with an initial learning rate of 0.01. Binary cross-entropy loss was used for pCR prediction. The input feature dimension was 1,453, corresponding to the concatenation of 1,197 radiomics features and 256 deep learning features. A bag-based training strategy was adopted (bag size = 4), and feature normalization was applied prior to model input using Z-score normalization. Models were trained for 50 epochs with a batch size of 16. Model performance was monitored on the validation cohort to prevent overfitting. All experiments were conducted on a single NVIDIA A40 GPU. Model evaluation The predictive performance of the models was evaluated using multiple complementary approaches. Discrimination was assessed with receiver operating characteristic curves (ROC) and quantified by the area under the curve (AUC), with differences between models compared using the DeLong test. Sensitivity, specificity, and overall accuracy were also reported. To examine incremental predictive value, Integrated Discrimination Improvement (IDI) and Net Reclassification Improvement (NRI) indices were calculated. Clinical utility was evaluated using decision curve analysis (DCA), and calibration curves were plotted to assess concordance between predicted probabilities and observed outcomes. Collectively, these analyses quantified the added value of the proposed models in risk stratification and patient reclassification compared with reference models. Statistical analysis Categorical variables were assessed with the Chi-square or Fisher’s exact tests, and continuous variables were analyzed using the Mann-Whitney U test or independent t-test. A two-sided P-value of less than 0.05 was considered statistically significant. Statistical analyses were conducted using R (v4.4.0) and Python (v3.7.0). Data availability The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Code availability Other codes used in this study are available from the corresponding author upon reasonable request. Results Patient characteristics A total of 440 patients with LARC were included in this study underwent nCRT, comprising 305 patients in the training cohort and 135 patients in the internal validation cohort. Baseline clinical characteristics of the two cohorts are summarized in Table S1. There were no significant differences in age, sex, height, weight, baseline CEA, or CA19-9 levels (all P > 0.05). Patients in the training cohort had significantly higher clinical T and N stages compared with those in the validation cohort. The pCR rates were 20.0% in the training cohort and 11.9% in the validation cohort, with no statistically significant difference between groups (P > 0.05). Subgroup analyses by pCR status are presented in Tables S2 and S3. In both the validation cohort (n = 135) and the training cohort (n = 305), baseline clinical characteristics showed no significant differences between the pCR and non-pCR groups. Radiomics and deep learning feature extraction All patients underwent pre- and post-treatment multimodal MRI examinations, including T2WI and GD. Tumor ROIs were manually delineated by 3 experienced radiologists to ensure consistent boundary definition. To assess feature reproducibility, inter-observer reliability was evaluated using ICCs, and features with ICC values greater than 0.90 were considered highly reproducible and retained for subsequent analysis. From each ROI, a total of 1,197 handcrafted radiomics features-covering intensity, shape, and texture descriptors-were extracted. In parallel, 2,048 deep learning features were obtained from the fully connected layer of a Med3D-pretrained ResNet-101 network, and subsequently compressed into a 256-dimensional representation using PCA. Comparative analyses between patients achieving pCR and those without (non-pCR) revealed a substantial number of discriminative imaging biomarkers. Specifically, 291 features were significantly different in pre-treatment T2WI, 179 in post-treatment T2WI, 511 in pre-treatment GD, and 347 in post-treatment GD (Fig. 2A). Construction of an early fusion based machine learning model for pCR prediction. A Volcano plots showing the distribution of radiomic features between pCR and non-pCR groups across different imaging modalities (Pre-T2, Post-T2, Pre-GD, and Post-GD). Each color represents a distinct feature family. B LASSO regression analysis for feature selection. Left: tuning parameter (λ) selection via cross-validation. Middle: coefficient profiles of features along the λ path. Right: bar plot of the most informative features with corresponding coefficients. C Receiver operating characteristic (ROC) curves for three representative classifiers (KNN, Naïve Bayes, and ExtraTrees) using the selected features. Both training (solid line) and testing (dotted line) performance are shown with corresponding AUC values Early fusion-based machine learning model construction Based on the Siamese multimodal MRI framework, 1,197 handcrafted radiomics features and 256 compressed deep learning features (derived from the Med3D-pretrained ResNet-101) were extracted from each imaging modality. After concatenation of pre- and post-treatment features, an early fusion feature set comprising 5,812 variables was obtained. Following normalization, redundant features were removed by excluding one of any pair with a Pearson correlation coefficient > 0.9, yielding 2,350 non-redundant features. LASSO regression with pCR as the endpoint further reduced the feature set to 23 informative predictors (Fig. 2B). These selected features were then incorporated into multiple machine learning classifiers, including Naïve Bayes, KNN, and ExtraTrees. In the training cohort, all models demonstrated only modest predictive performance, with AUC values clustering around 0.60 (Fig. 2B; Table S4). Among them, the Naïve Bayes classifier achieved the highest discrimination (AUC = 0.629), which, although superior to the other algorithms, still reflected suboptimal predictive performance. Transformer-based framework for pCR prediction To further enhance model performance, we implemented a transformer-based framework for pCR prediction. In this approach, 1,197 handcrafted radiomics features and 256 compressed deep learning features were extracted from each imaging modality, including pre- and post-treatment T2WI and GD sequences. Features were concatenated and processed using a Siamese transformer encoder with shared weights to capture both temporal and cross-modal dependencies. The encoded representations were aggregated through max-pooling and subsequently fed into fully connected layers to generate pCR probabilities (Fig. 3). Workflow of the Transformer-based framework incorporating Siamese multimodal feature fusion for pCR prediction Among the 3 transformer-based strategies, the Pre-treatment model, the Post-treatment model, and the Siamese multimodal mode all demonstrated superior performance compared with traditional machine learning approaches (the Early fusion model). In the validation cohort, the Siamese multimodal transformer achieved the highest discrimination, with an AUC of 0.789, followed by the Pre-treatment (AUC = 0.739), Post-treatment (AUC = 0.728), and Early fusion (Naïve Bayes; AUC = 0.629) models (Table S5, Fig. 4A). Decision curve analysis (DCA) further confirmed the clinical utility of transformer-based frameworks, with the Siamese multimodal strategy providing the greatest net benefit across a broad range of threshold probabilities (Fig. 4B). Calibration analysis showed that all transformer-based models achieved good agreement between predicted and observed pCR probabilities, with the Siamese multimodal model exhibiting the closest fit (Fig. 4C). Comparable results were observed in the validation cohort, where the Siamese multimodal model retained the best predictive ability, significantly outperforming the Early fusion machine learning baseline (P < 0.01, DeLong test; Fig. 4D). Additional statistical comparisons using net reclassification improvement (NRI) and integrated discrimination improvement (IDI) demonstrated that transformer-based models, particularly the Siamese multimodal approach, improved classification accuracy relative to early fusion machine learning models (Fig. 4D). Performance evaluation of transformer-based models with different feature fusion strategies. A ROC curves of transformer-based models using Pre-treatment, Post-treatment, Siamese multimodal, and Early fusion features in the training and validation cohorts. B Decision curve analysis (DCA) showing the net clinical benefit across probability thresholds for each transformer-based strategy in both cohorts. C Calibration curves comparing predicted versus observed probabilities of pCR in the training and validation cohorts. D Statistical comparisons among transformer-based models using DeLong tests, net reclassification improvement (NRI), and integrated discrimination improvement (IDI) analyses in the training and validation cohorts Kaplan-Meier analysis showed that transformer-based risk stratification was associated with progression-free survival (PFS). The Post-treatment model achieved the most significant separation between high- and low-risk groups (p = 0.0004), followed by the Pre-treatment (p = 0.037) and Siamese multimodal models (p = 0.040) (Fig. 5). These results suggest that transformer-based models not only predict pCR but also provide prognostic information on long-term outcomes. Prognostic performance of transformer-based models for progression-free survival (PFS) prediction. Kaplan-Meier survival curves for PFS based on risk stratification using the best-cutoff approach. Patients were divided into high-risk and low-risk groups according to predicted scores from the transformer-based models: Post-treatment features (A), Pre-treatment features (B), and Siamese multimodal features (C) Discussion In this study, we developed and validated a transformer-based multimodal framework that integrates handcrafted radiomics and deep learning features extracted from pre- and post-treatment multimodal MRI to predict pCR in patients with LARC. Our results demonstrated that transformer-based models consistently outperformed traditional early fusion machine learning approaches, with the Siamese multimodal strategy achieving the highest predictive performance. Beyond accurate pCR prediction, transformer-derived risk stratification was also significantly associated with progression-free survival, underscoring the prognostic relevance of this framework. Radiomics and deep learning have independently shown promise in predicting therapeutic response in rectal cancer. Prior studies reported AUCs ranging from 0.65 to 0.75 for MRI-based radiomics signatures [16, 26], while more recent deep learning pipelines have modestly improved performance but remain limited by single-modality or single-timepoint analysis. By contrast, our Siamese transformer framework explicitly captured longitudinal and cross-modal dependencies, yielding an AUC of 0.789 in the validation cohort-superior to both the simple early fusion machine learning baseline (AUC 0.629) and transformer models that relied only on single-sided data (Pre-treatment AUC 0.739, Post-treatment AUC 0.728). This finding suggests that transformers are well suited to handle the high-dimensional, temporally heterogeneous data inherent to neoadjuvant treatment settings. Similar transformer-based strategies have recently shown improved prognostic performance in other tumor types: for instance, Gomaa et al. integrated MRI, molecular, and clinical data in glioblastoma, achieving higher C-index values (~ 0.70) than late-fusion CNN models [27]; and Kim & Park proposed a radiomics-guided multimodal self-attention network for breast MRI, successfully predicting pCR through enhanced cross-modal feature learning [28]. Recently, Zhang et al. developed a ResNet-Vision Transformer-based MRI-endoscopy fusion model in a multicenter cohort and achieved an AUC of 0.769 in the external test set for treatment response prediction, highlighting the potential of multimodal transformer-based fusion strategies, although their approach relied solely on pretreatment data, whereas our framework integrates both pre- and post-treatment multimodal MRI features through a Siamese architecture to capture longitudinal dynamics [29]. Moreover, Xia et al. proposed a fully automated large model-assisted multi-parametric MRI pipeline (Rectal-RadioSAM) that combined MedSAM-based segmentation with radiomics and achieved an AUC of 0.83 for pCR prediction, demonstrating the feasibility of large-model-driven automated assessment, whereas our framework focuses on longitudinal multimodal feature integration using a Siamese transformer architecture to explicitly model temporal and cross-modal dependencies [30]. These findings resonate with our results, supporting the broader applicability of transformer architectures in multi-timepoint and multimodal oncology imaging. Accurate prediction of pCR is of paramount clinical importance, as it can identify candidates for organ-preserving strategies such as “watch-and-wait”, thereby sparing patients from the morbidity of radical resection [31]. Equally, patients with a low predicted likelihood of achieving pCR with chemoradiotherapy alone may be considered for intensified neoadjuvant treatment to maximize therapeutic benefit. Our decision curve analysis confirmed that transformer-based models, particularly the Siamese multimodal framework, provided the greatest net clinical benefit across a range of thresholds. Furthermore, the significant association between model-derived risk stratification and PFS highlights the potential utility of this approach in prognostic assessment, which could aid in tailoring postoperative surveillance and adjuvant therapy [32]. Importantly, the use of noninvasive MRI-based biomarkers makes this strategy both clinically accessible and repeatable in routine practice [33]. In this study, tumor regions were manually delineated by experienced radiologists to ensure accurate representation of tumor boundaries. High inter-observer agreement further supported the robustness of this approach. Nevertheless, recent advances in automated and semi-automated segmentation, particularly deep learning–based methods [34–36], offer promising opportunities to improve efficiency and reproducibility. The integration of such techniques may further enhance the scalability and clinical applicability of our framework in future studies. Beyond methodological considerations, the clinical implications of the proposed model also warrant careful interpretation. Although the proposed model demonstrated a significant association with PFS, this analysis was conducted exclusively in the training cohort. As such, the observed prognostic relevance should be interpreted cautiously and considered exploratory. Future studies incorporating independent external cohorts are necessary to validate these findings and to further establish the clinical utility and generalizability of the model in survival prediction. Several limitations warrant consideration. First, although this was a multicenter study, the analysis was retrospective, which may introduce inherent selection and information bias; prospective studies are warranted to validate our findings. Second, despite the multicenter design, variability in scanner vendors, coil configurations, and acquisition protocols across institutions may introduce inter-center heterogeneity in radiomics features. Although the inclusion of data from multiple centers may improve model robustness to real-world clinical variability, such heterogeneity cannot be fully eliminated and may affect feature stability and model generalizability. Future studies incorporating harmonization strategies (e.g., ComBat or domain adaptation methods) are warranted to further mitigate these effects. Third, baseline imbalances were observed between the training and validation cohorts—specifically higher clinical T and N stages in the training set, which reached statistical significance, and an apparent difference in pCR rates (20.0% vs. 11.9%) that did not reach statistical significance but showed a clear trend. These disparities suggest potential distribution effects, and larger multicenter cohorts will be required to confirm the robustness, calibration, and clinical utility of our transformer framework. Fourth, although strong performance was observed in the training cohort, the possibility of model overfitting cannot be fully excluded, particularly given the high-dimensional feature space and the complexity of the transformer-based architecture. While performance was evaluated in the independent validation cohort, further validation using larger-scale datasets and external cohorts, along with more advanced regularization strategies, is warranted to ensure model robustness and generalizability. Fiveth, a direct head-to-head comparison with standard clinical imaging assessments, such as mrTRG, was not performed. Because standardized mrTRG evaluations were not uniformly available across participating institutions in this retrospective setting, the incremental value of the proposed model over routine radiologic assessment could not be formally quantified. Future prospective studies incorporating standardized mrTRG assessment and centralized imaging review are necessary to further clarify the added clinical benefit of our framework. Finally, integrating genomic or molecular biomarkers and the results from endoscopy and digital rectal examination with imaging features may enhance biological interpretability and robustness, paving the way for mechanism-based prediction and clinical translation. Conclusion In conclusion, we established and validated a transformer-based Siamese multimodal framework that integrates radiomics and deep learning features from pre- and post-treatment MRI for predicting pathological complete response in patients with LARC. Compared with early fusion machine learning and single-sided models, the Siamese multimodal transformer demonstrated superior discrimination and clinical utility. These findings highlight the potential of transformer-based multimodal imaging biomarkers to support individualized risk stratification and inform treatment decision-making. Supplementary Information Supplementary Material 1. Supplementary Material 2. Acknowledgements We thank all the workers who contributed to this study. Abbreviations Locally advanced rectal cancer Neoadjuvant chemoradiotherapy Pathological complete response Tumor regression grade Magnetic resonance imaging Convolutional neural networks Imaging biomarker standardization initiative Regions of interest Gadolinium T2-weighted imaging Principal component analysis Least absolute shrinkage and selection operator Multi-head attention Receiver operating characteristic curves Area under the curve Integrated discrimination improvement Net reclassification improvement Decision curve analysis K-nearest neighbor Progression-free survival Authors’ contributions Conceptualization: B.F., J.X.; Data curation: B.F., J.X, Y.H.; Formal analysis: B.F., J.X.; Funding acquisition: J.X.,X.C. ; Investigation: B.F.,J.X.; Methodology: B.F., J.X., S.L.,Q.W.; Project administration: B.F., S.D.; Resources: B.F., J.X.; Software: B.F., J.X., S.L.,Q.W.; Supervision: B.F., S.D.; Validation: B.F., S.D.; Visualization: B.F., J.X.; Roles/Writing - original draft: J.X., X.C.; Writing - review & editing: B.F., S.D.;. Funding This work was supported by the National Natural Science Foundation of China (Grant No. 82303742 to Jing Xu) and the Zhejiang Provincial Natural Science Foundation of China (Grant No. LQN26H160030 to Bohai Feng). Data availability The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Declarations Ethics approval and consent to participate This study complied with the principles of the Declaration of Helsinki and received approval from the ethics committees of the Second Affiliated Hospital, Zhejiang University School of Medicine (No:2023 − 0496), Guangdong Provincial People’s Hospital (No: KY-Q-2022-208-01), and Guangzhou First People’s Hospital (No: K-2024-172-01). Given its retrospective nature and reliance on de-identified data, the institutional review boards waived the requirement for informed consent. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Jing Xu, Xin Chen and Yeting Hu contributed equally to this work. References Associated Data Supplementary Materials Supplementary Material 1. Supplementary Material 2. Data Availability Statement The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.