A Generalizable Multimodal Model for Treatment‐Stratified Risk and Survival Assessment under Real‐World Constraints: A Multi‐Center Study of Colorectal Cancer Chuangjie Cao, Zheyi Ji, Chang He, Jiying Wang, Xiaoming Luo, Junyi Liu, Xiaohu Jing, Fang Yan, Yirong Chen, Linhao Qu, Sen Yang, Le Lu, Yuming Jiang, Xiyue Wang, Chengyun Dou, Junhan Zhao Abstract Integrating histopathology, genomic, and clinical phenotypes holds promise for improving prognostic patient stratification, including within treatment‐defined subgroups, in colorectal cancer (CRC). However, ensuring the availability of all modalities in routine clinical practice remains challenging. Comprehensive molecular profiling is often constrained by cost and turnaround time, leaving histopathology as the most consistently available modality across institutions. In addition, variability in staining protocols, scanning devices, patient demographics, and outcome distributions across centers undermines the generalizability of models trained under controlled conditions with complete inputs. To address these challenges, we developed the Domain‐Adaptive Incomplete Multimodal Stratification framework (DAIMS). DAIMS leverages paired histopathology, genomic, and clinical data during training to learn generalizable disease representations and uses histology‐derived surrogate latent representations when auxiliary modalities are unavailable, enabling flexible histology‐only inference. Trained on TCGA samples and validated on two independent European and East Asian cohorts comprising 842 patients, DAIMS consistently outperformed state‐of‐the‐art methods. Adapted DAIMS improved the C‐index by 5.6%–13.9% on F1CRC and 4.1%–9.0% on SURGEN compared with baseline methods. DAIMS improved within‐cohort prognostic discrimination and localized prognostically relevant morphological patterns that remained stable across institutions. Our evaluation showed that DAIMS achieved generalizable survival stratification across disease stages, molecular subgroups, and treatment‐defined patient groups. DAIMS is a multimodal model for treatment stratified risk and survival assessment in colorectal cancer. Training draws on pathology, genomics, and clinical reports, yet inference requires only whole slide images, and adaptation to new cohorts proceeds without target survival outcomes. Across multiple centers, stratification remains accurate when modalities are missing, bringing multimodal prognostication within reach of routine practice. Introduction Colorectal cancer (CRC) remains a major global health burden, accounting for approximately 1.93 million new cases and nearly 935,000 deaths each year [, , ]. Despite continued advances in staging systems and therapeutic strategies, survival outcomes remain highly heterogeneous even among patients with similar clinicopathological profiles [, , ]. This heterogeneity reflects the multifactorial nature of CRC progression, often shaped by the interplay of tumor stage, molecular alterations, and histopathological phenotypes [, , , ]. Improving prognostic modeling therefore requires integrative approaches that can capture complementary information across biological and clinical levels. In routine clinical practice, histopathological slides remain central to the diagnosis and assessment of CRC. Histology images encode critical morphological features [, , , ]. In contrast, sequencing‐based genomic profiles capture driver mutations (e.g., KRAS [] and BRAF []), microsatellite instability (MSI), tumor mutational burden (TMB), and copy number alterations, molecular features closely associated with disease progression and survival [, , ]. Clinical records add a longitudinal dimension, capturing family history, treatment course and response, and laboratory measurements that further inform individual risk assessment [, ]. Together, these complementary clinical data modalities offer a rich profile of patient disease status. Recent advances in prognostic modeling have been achieved across individual data modalities, including histopathological image analysis [, ], genomic profiling [, , ], and clinical text [, , ] modeling. Several pathology foundation models have advanced transferable histopathological representation learning through large‐scale pretraining and knowledge distillation, such as MUSK [], H‐Optimus‐1 [], and GPFM []. Motivated by the complementary nature of these heterogeneous data sources, multimodal deep learning frameworks have been proposed to integrate histopathological images, genomic features, and clinical information [, , , ]. However, most existing approaches assume that the training and deployment environments share the same modality set, acquisition pipeline, cohort composition, and outcome distribution [, ]. This assumption rarely holds in practice, where two interrelated challenges limit clinical translation. In real‐world clinical environments, multimodal inputs are often incomplete and heterogeneous, which limits the reliability of trained models at deployment []. First, visual covariate shift arises from staining protocols, scanners, microns‐per‐pixel resolution, image quality, and tissue‐processing differences across institutions [, , ]. Second, cohort and label‐space shift arise because patient populations, treatment pathways, censoring patterns, survival endpoints, and event‐time distributions can differ between source and external cohorts. These differences are especially important for survival models, because a risk head optimized on source‐domain event‐time bins may be poorly calibrated when the target cohort follows a different endpoint or time distribution. Third, modality availability itself varies across domains: many hospital datasets contain only histopathological slides and limited clinical metadata, whereas genomic profiles may be unavailable because of cost and long processing times [, , ]. Clinical reports are also frequently stored as unstructured narratives, complicating standardization and large‐scale computational analysis [, ]. Consequently, morphological, molecular, and clinical information are often fragmented and only partially observed rather than forming a unified patient representation. Modality incompleteness and multi‐source batch effects therefore jointly limit the robustness and clinical applicability of existing multimodal prognostic models, underscoring the need for frameworks that remain reliable under incomplete and heterogeneous real‐world conditions []. Here, we propose the Domain‐Adaptive Incomplete Multimodal Stratification framework (DAIMS). DAIMS is designed as a domain adaptation framework for real‐world CRC patient stratification. It uses multimodal TCGA data to learn transferable representations, then adapts a WSI‐centered deployment branch to external cohorts where only pathology slides are available. Within this design, prototype‐guided aggregation and fusion, cross‐modal representation imputation, and lightweight target‐domain fine‐tuning are not independent goals. They are complementary mechanisms for aligning patient representations across heterogeneous visual, modality‐availability, and outcome‐distribution shifts [, ]. DAIMS is trained on 351 TCGA samples and evaluated on two external cohorts (F1CRC, n=420; SURGEN [], n=422), consistently outperforming both multimodal and unimodal baselines across multiple encoders. Results Overview of the Domain‐Adaptive Incomplete Multimodal Stratification (DAIMS) Framework We present DAIMS (Figure ), a multimodal survival framework for colorectal cancer. In clinical practice, comprehensive molecular profiling is often delayed or unavailable. To address this, DAIMS is designed to operate on histology alone at inference, while leveraging multimodal training (histology, molecular profiles, and clinical notes) to enrich the model's learned representations. We formulate the resulting modality‐availability shift between training and inference as a domain adaptation problem [, , ]. DAIMS employs an imputation module to encourage WSI features to incorporate genomic and clinical information during training [, ] (Figure ). At transfer time, a light‐weight target‐domain adaptation step is performed where only the WSI adapter and survival head are updated based on pseudo risk labels obtained from target domain WSIs by the frozen source‐domain DAIMS model. Crucially, this adaptation step does not use the target‐domain survival time, censoring status, event labels, or risk‐group annotations. Target‐domain survival outcomes were reserved exclusively for final evaluation. We assessed DAIMS in terms of prognostic discrimination, transferability to external cohorts, treatment‐stratified analysis, and molecular‐subgroup stratification. Multi‐Center and Multi‐Modal Dataset and Cohort Characteristics We collected and analyzed three independent colorectal cancer (CRC) cohorts: TCGA (n=351), F1CRC (n=420), and SURGEN (n=422) (Table ). TCGA served as the source‐domain cohort and provided matched WSIs, genomic pathway profiles, and structured clinical reports. In contrast, F1CRC and SURGEN provided WSIs as model inputs for external validation, reflecting real‐world modality‐availability shift, while additional clinical or molecular annotations were used only for subgroup analyses [, ]. For downstream subgroup analysis, TCGA and F1CRC additionally contained detailed treatment annotations, including radiation therapy, pharmaceutical treatment, chemotherapy regimens, and surgical procedures. SURGEN further provided molecular biomarker information, including BRAF [], KRAS exon 2 [], and mismatch repair (MMR) status [], together with complete pathological staging information. The three cohorts exhibited substantial demographic, clinical, and technical heterogeneity. Molecular annotation also varied across cohorts: TCGA provided comprehensive molecular profiling, whereas F1CRC did not provide equivalent molecular profiling and SURGEN included selected biomarkers only. TCGA comprised a racially diverse population and was predominantly composed of White patients (71.2%), whereas F1CRC consisted entirely of East Asian patients. In contrast, SURGEN was derived from a United Kingdom cohort predominantly representing Caucasian patients. Clinical characteristics also differed substantially across cohorts. For example, rectal tumors accounted for 48.6% of cases in F1CRC but 10.0% in TCGA, while SURGEN contained a larger proportion of Stage III disease (45.5%) than TCGA (29.6%). In addition, differences in slide preparation, staining protocols, and scanner platforms across institutions introduced substantial inter‐center variation in histopathology appearance []. Together, these complementary cohorts provide a realistic benchmark for evaluating multimodal survival models under heterogeneous cross‐cohort settings. DAIMS Achieves Robust Risk Stratification Across Diverse Cohorts The prognostic performance of DAIMS was first evaluated on the TCGA source cohort under both fully observed and controlled modality‐availability conditions. The framework was then assessed for cross‐domain generalization on the F1CRC and SURGEN cohorts, where only WSI data were used for model inference. Source‐Domain Evaluation Under Modality‐Availability Shifts DAIMS was trained with four pathology foundation encoders to assess whether the observed performance gains were preserved across heterogeneous WSI backbones. Under fully observed multimodal inputs, DAIMS consistently separated predicted high‐risk and low‐risk patients in Kaplan–Meier analysis (Figure ). CONCH v1.5 achieved the strongest stratification (log‐rank p=1.20×10−3), while UNI v1, UNI2‐h, and Virchow2 showed the same overall trend. This pattern was reproduced across multiple pathology encoders, arguing against the gain being restricted to a single backbone configuration [, , , ]. DAIMS was further evaluated under three controlled modality‐availability conditions, in which genomic and clinical reports data were removed at missing ratios of (30%, 30%), (20%, 40%), and (40%, 20%) for (genomic, clinical reports), respectively, while always retaining WSI data. Complete fold‐wise results are provided in Table . Across all four encoders and all missing‐ratio settings, Kaplan–Meier analysis continued to demonstrate significant survival stratification. Using CONCH v1.5 features, the (30%, 30%) setting yielded p=7.68×10−4, the (20%, 40%) setting yielded p=7.07×10−4, and the (40%, 20%) setting yielded p=4.62×10−6. In general, performance degradation relative to the fully observed setting was limited across these auxiliary‐modality imbalance settings, indicating that DAIMS can retain informative prognostic signals even when auxiliary modalities are partially unavailable. Notably, the relative contribution of genomic and report availability varied across encoders and missing‐ratio settings, but the overall results suggest that auxiliary genomic and clinical‐report supervision provides complementary information for risk stratification [, ]. Quantitative comparative experiments further supported these observations (Figure ; Table ). DAIMS outperformed pathology‐only baselines across all evaluated settings while maintaining stable performance under incomplete modality conditions. Using CONCH v1.5 features, the mean C‐index reached 0.761 under fully observed inputs, exceeding ABMIL (0.681), ILRA (0.718), and WIKG (0.703), and remained at 0.744 under the (20%, 40%) missing‐ratio setting. Similar trends were observed with UNI v1, UNI2‐h, and Virchow2, whose pathology‐only baseline performance ranged from 0.62 to 0.68 but increased to 0.685–0.733 after integration into DAIMS. Overall, DAIMS maintained prognostic discrimination across heterogeneous pathology encoders and under reduced auxiliary‐modality availability. To directly assess inference‐time patch sensitivity, we evaluated fixed DAIMS models from all four pathology foundation encoders. Patient‐level prognostic discrimination was reduced at low patch budgets but broadly maintained from 5,000 to 10,000 patches per WSI, without a consistent gain beyond the default 4,096‐patch setting (Figure and Table ). A complementary analysis varying the maximum patch budget during model development and held‐out evaluation showed the same non‐monotonic pattern across source missingness settings. Cross‐Domain Generalization to Independent Cohorts The ability of DAIMS to generalize from TCGA to independent external cohorts was evaluated on F1CRC (n=420) and SURGEN (n=422). These cohorts differ from TCGA in patient demographics, specimen preparation, staining protocols, scanning hardware, image quality, and survival‐time distributions [], providing a stringent setting for cross‐domain and cross‐population evaluation. Because survival endpoints and event‐time distributions differed across cohorts, high‐ and low‐risk groups were defined separately within each cohort using the cohort‐specific median predicted risk score. Across both F1CRC (Figure ) and SURGEN (Figure ), Kaplan–Meier analysis showed significant separation between predicted high‐risk and low‐risk groups under all encoder configurations and all missing‐ratio settings (all log‐rank p<0.05). In SURGEN, the strongest separation was observed with UNI2‐h under the (20%, 40%) setting (p=3.98×10−17), while Virchow2 achieved strong stratification under the (40%, 20%) setting (p=2.20×10−15). In F1CRC, CONCH v1.5 under the (20%, 40%) setting yielded the most significant separation (p=3.62×10−9), and UNI v1 also achieved strong stratification under the (20%, 40%) setting (p=6.53×10−8). No single encoder or missing‐ratio configuration clearly outperformed across both cohorts, indicating stable prognostic stratification across heterogeneous deployment conditions. C‐index analysis was further performed to assess prognostic performance across external cohorts (Figure ; Table ). After pseudo‐label‐based adaptation on each target cohort, DAIMS generally outperformed the corresponding pathology‐only baselines across encoders. In F1CRC, the best performance reached 0.810 with CONCH v1.5, followed by Virchow2 (0.786), UNI2‐h (0.752), and UNI v1 (0.734). In SURGEN, DAIMS achieved up to 0.734 (UNI2‐h), 0.712 (CONCH v1.5), 0.708 (Virchow2), and 0.711 (UNI v1), exceeding the corresponding pathology‐only baselines within each encoder group. The relative ranking of encoders was slightly different between cohorts, but DAIMS retained gains over pathology‐only models in all settings. Importantly, these results were obtained without access to target‐domain genomic or structured clinical reports data during inference. This suggests that multimodal training may enhance the robustness of pathology‐based representations under external deployment settings, particularly when only WSI data are available. DAIMS Preserves Prognostic Stratification Across Treatment‐Defined and Clinical‐Molecular Subgroups Treatment Subgroups in TCGA Patients were grouped by treatment modality (Radiation, Pharmaceutical therapy, Chemotherapy, and Surgery) and stratified into predicted high‐ and low‐risk subgroups within each category (Figure , top; Table ). Kaplan–Meier analysis showed statistically significant survival separation in the Radiation, Pharmaceutical therapy, Chemotherapy, and Surgery groups. DAIMS captured residual prognostic risk within patient groups defined by broad treatment categories. Patients undergoing the same treatment modality can differ significantly in tumor differentiation, disease burden, microenvironment composition, and treatment sensitivity, all of which may leave morphological imprints on routine pathology slides [, , ]. Treatment and Surgical Subgroups in F1CRC Patients were grouped by treatment strategy into Surgery + Chemotherapy vs. Surgery‐only cohorts (Figure , middle; Tables ). In both groups, DAIMS produced clear and statistically significant separation between predicted high‐ and low‐risk patients, supporting transfer of the TCGA‐trained risk signal to an East Asian external cohort. Patients were further stratified by chemotherapy regimen into those who did not receive chemotherapy and those treated with doublet regimens (e.g., XELOX, CAPEOX, FOLFOX) []. DAIMS successfully distinguished survival outcomes within both subgroups. Surgical stratification was further refined into four clinically meaningful categories: rectal resection with sphincter preservation (Dixon/AR), colon resection, palliative procedures (ostomy, exploratory surgery, bypass, stenting), and rectal resection without sphincter preservation (Miles/APR) []. DAIMS consistently stratified patients into high‐ and low‐risk groups with statistically significant survival differences across all four surgical subgroups, suggesting that morphology‐linked prognostic signals remain informative across clinically heterogeneous surgical subgroups. Stage and Molecular Subgroups in SURGEN DAIMS achieved statistically significant separation between predicted high‐ and low‐risk patients within both early‐stage (I–II) and advanced‐stage (III–IV) disease in the SURGEN cohort (Figure , bottom; Tables ), indicating that prognostic discrimination was preserved even after controlling for this dominant clinical determinant [, ]. Beyond disease stage, we examined three molecular markers that define clinically distinct CRC subtypes. For BRAF mutation status, a defining alteration of a biologically aggressive CRC subtype [], patients were stratified as WT vs. Mutant; DAIMS separated high‐ vs. low‐risk patients within both subgroups, including BRAF‐mutant tumors. For KRAS exon 2 mutation status [, ], patients were stratified as WT vs. Mutant, and DAIMS separated high‐ and low‐risk patients within each subgroup. For mismatch repair (MMR) status [], patients were categorized as MMR‐proficient (pMMR) vs. MMR‐deficient (dMMR), and DAIMS separated high‐ vs. low‐risk patients in both subgroups []. Together, these analyses test whether the prognostic signal captured by DAIMS primarily recapitulates known clinical and molecular stratifiers or instead provides additional prognostic information that remains stable across clinically defined subgroups. The stable stratification across subgroups defined by BRAF, KRAS, MMR, stage, and treatment argues against the learned risk signal being solely explained by the examined clinical or molecular factors. This indicates that histopathologic patterns provide prognostic information not fully captured by canonical clinical classifications, which is reflected in the representations learned by DAIMS through multimodal source‐domain training. Exact sample sizes for all treatment‐, stage‐, and molecularly defined subgroups are provided in Tables . Domain Adaptation Improves WSI‐Derived Risk Organization Across External Cohorts To examine the effect of lightweight adaptation on external deployment, we analyzed the organization of WSI‐derived semantic representations across TCGA, F1CRC, and SURGEN cohorts before and after adaptation. During WSI‐only inference, DAIMS generates histology‐derived surrogate tokens for multimodal fusion. The lower clinical‐report R2 reflects the heterogeneity of free‐text embeddings while these tokens retain task‐relevant information for survival prediction. These cohorts differ in population composition, batch effects from fixation, staining, and scanning, and the survival prediction head optimized on TCGA‐specific event‐time bins, leading to distribution mismatch in external cohorts [, ]. Before adaptation, t‐SNE projections (Figure ) showed that TCGA and F1CRC samples were largely mixed, without an obvious low‐to‐high predicted‐risk organization, whereas SURGEN samples formed a distinct cluster shifted relative to the other cohorts. Within each cohort, high‐risk and low‐risk cases were poorly separated. In F1CRC, risk predictions were generally underestimated, whereas SURGEN predictions were concentrated around the median, indicating limited transferability of the TCGA‐trained survival mapping under unadjusted conditions. After adaptation, cohort‐specific structure remained visible, while risk gradients became clearer within each cohort. This pattern is consistent with task‐oriented adaptation that improves prognostic organization while preserving meaningful cross‐cohort differences. As t‐SNE provides a qualitative visualization, generalizability was evaluated by prognostic discrimination across independent cohorts. We further quantified this improvement by comparing Gaussian‐fitted predicted‐risk distributions with observed event‐time distributions (Figure ). Their agreement was measured using the overlap area (OA), defined as OA=∫min(p(x),q(x))dx, where p(x) and q(x) denote normalized predicted‐risk and event‐time densities. The OA increased from 0.744 to 0.858 in F1CRC and from 0.887 to 0.925 in SURGEN. The increase in OA was larger in F1CRC (0.114) than in SURGEN (0.038). OA characterizes the distributional alignment of relative prognostic risk. The observed OA increases indicate more coherent within‐cohort risk organization after adaptation. Complementing this analysis, IPCW Brier scores decreased after adaptation at all six evaluated horizons in both F1CRC and SURGEN, indicating lower time‐dependent survival‐probability prediction error (Table ). Together, these analyses support improved risk organization and outcome‐based survival prediction after adaptation. We evaluated pseudo‐label quality through its downstream effect on model performance, averaging over all four pathology encoders and all missing modality scenarios at each of the eight adaptation iterations. In F1CRC, the mean C‐index gain increased steadily across iterations and reached 0.076 at iteration 8. In SURGEN, the mean gain increased over the first five iterations and then plateaued, ending at 0.039 at iteration 8. Median split risk group separation increased in both cohorts (Figure ). Since pseudo‐labels are regenerated at every iteration, compounding pseudo‐label error would produce a decline in later iterations, which we do not observe in either cohort. A focused sensitivity analysis evaluated adaptation iterations, learning rate, update scope, and the target‐domain case selection ratio. Learning‐rate changes produced modest effects. Joint prediction‐head/WSI‐adapter updating and the default selection ratio of 0.5 provided balanced performance across cohorts. Eight adaptation iterations improved within‐cohort discrimination relative to four iterations in both cohorts (Figure ). Together, these results indicate that lightweight adaptation improves within‐cohort risk organization and supports prognostic discrimination across heterogeneous multicenter CRC cohorts. DAIMS provides cohort‐relative prognostic rankings and adapts to a new institution using only target‐domain WSIs and model‐generated pseudo‐risk labels [, , , ]. Labeled target‐domain outcomes can further support site‐specific absolute risk calibration. Modality Representation Imputation Enables Flexible Inference Under Missing Auxiliary Modalities Reconstructed genomic and clinical‐report features exhibited clear linear correspondence with their original representations (Figure ). Genomic pathway features were recovered with high fidelity (cosine similarity 0.966, r=0.964, R2=0.854), whereas clinical‐reports embeddings showed moderate agreement (cosine similarity 0.78, r=0.738, R2=0.422). This asymmetry may partly reflect the more structured nature of pathway‐level genomic programs compared with the heterogeneous, partially observed nature of clinical narratives [, ]. These results indicate that the imputation modules function primarily as a regularizer, enforcing cross‐modal associations where they are biologically grounded while tolerating imperfect recovery where they are not []. Importantly, the module reconstructe latent token embeddings used by the frozen LLM backbone. Prototype‐Guided Attention Reveals Interpretable Histopathologic Risk Patterns The attention patterns of DAIMS were examined for pathological plausibility in representative cases from all three cohorts. Built on Virchow2 [], the PGAF module learns prognostic prototypes (high‐risk and low‐risk) that aggregate WSI patches into slide‐level attention heatmaps, which can be overlaid on the original image. Attention Patterns in TCGA Source‐Domain Cases In representative TCGA cases, DAIMS separated localized atypical foci from the broader tissue context rather than treating any isolated malignant patch as sufficient for a high‐risk call (Figure ). In the favorable‐outcome case (Figure ), high‐risk attention was mainly concentrated on well‐differentiated colorectal adenocarcinoma with crowded back‐to‐back glands, mild‐to‐moderate nuclear atypia, chronic inflammatory cells, and mild desmoplastic stromal reaction, whereas low‐risk activation preferentially covered benign colonic wall tissue with smooth muscle bundles and connective tissue. This pattern suggests that limited malignant morphology did not dominate the slide‐level risk estimate when embedded within a broader tissue landscape that remained largely non‐neoplastic [, , ]. In the poor‐outcome case (Figure ), the qualitative pattern reversed. High‐risk prototype activation became spatially extensive and concentrated on poorly differentiated adenocarcinoma with infiltrative sheets, nests, and cords of pleomorphic malignant cells, near‐complete loss of glandular architecture, and prominent desmoplastic stromal reaction, whereas low‐risk activation was relatively sparse. Thus, the key difference between the two samples is not only the presence of tumor, which can be found in both samples, but the predominance of high‐grade invasive morphology in the tissue landscape. This is consistent with the view that CRC outcomes are influenced by the global balance between malignant epithelial proliferation, stromal remodeling, and residual non‐neoplastic structures, rather than by any single isolated field of view [, , ]. Attention Patterns Transfer Across Cohorts Despite Domain Shift Cross‐domain heatmap analysis qualitatively supported the generalization results. When applied to representative cases from F1CRC and SURGEN, DAIMS preserved the same prototype logic observed on TCGA, with limited high‐risk activation concentrating on invasive adenocarcinoma regions and low‐risk activation preferentially covering fibroadipose stroma, reactive inflammatory background, necrosis, or only limited residual tumor components in regions with less aggressive morphology []. Across both external cohorts, surviving patient examples (Figure ) showed limited high‐risk prototype activation over gland‐forming adenocarcinoma regions, while low‐risk activation predominantly highlighted fibroadipose stroma, vessels, chronic inflammatory infiltrates, and other reactive non‐neoplastic tissue. In deceased patient examples (Figure ), DAIMS exhibited broader high‐risk activation across morphologically aggressive tumor regions. In F1CRC, these included cribriform or back‐to‐back malignant glands with necrosis, while in SURGEN they corresponded to more heterogeneous moderate‐to‐poorly differentiated adenocarcinoma. In contrast, low‐risk activation was largely restricted to benign background, extensive necrotic debris, or small residual lower‐grade tumor areas. These examples show no exact color or texture matching across institutions due to differences in staining protocols and scanners, but consistent morphologic relationships. Across cases, DAIMS contrasts limited gland‐forming tumor with reactive stroma in lower‐risk samples vs. necrotic, pleomorphic, or disorganized carcinoma in higher‐risk samples [, , ]. The high‐ and low‐risk prototypes reflect differences in intratumoral heterogeneity, with higher‐risk cases showing greater spatial predominance of invasive tumor regions relative to benign architecture and gland‐forming components. Similar patterns are observed across three cohorts with distinct staining and demographic profiles. Expanded prototype‐level examples across predicted high‐ and low‐risk patients from F1CRC and SURGEN, with top‐attention patches shown for all four pathology foundation encoders, are provided in Figure . Ablation Analysis Characterizes Modular Contributions Four targeted ablation analyses were conducted to identify the components of DAIMS responsible for the observed prognostic performance, as well as their individual contribution to source‐domain discrimination and cross‐cohort generalization (Figure ; Table ). Effect of the Prototype‐Guided Attention Fusion Module We replaced the PGAF module with mean pooling or attention pooling while keeping all other components unchanged. Both alternatives consistently underperformed the full DAIMS across all four encoders on TCGA (Figure ). With CONCH v1.5, the full PGAF module achieved a C‐index of 0.761, compared with 0.701 for mean pooling and 0.651 for attention pooling. Similar gaps were observed for UNI v1 (0.685 vs. 0.638 and 0.640), UNI2‐h (0.724 vs. 0.611 and 0.642), and Virchow2 (0.713 vs. 0.681 and 0.623). The attention‐pooling variant performed worse than mean pooling in most cases, suggesting that unstructured token aggregation loses prognostic information that the prototype‐guided mechanism preserves. These results indicate that PGAF contributes more than a generic compression step: the learned prototypes provide a structured space for selecting and organizing risk‐relevant tokens across modalities. Effect of Auxiliary Modalities This analysis varied the modality combination during source‐domain training, while all held‐out‐fold evaluations used pathology WSIs as the only direct input. Across the four encoders, Full DAIMS (trained on complete modalities) achieved the highest C‐index of 0.761±0.114, compared with 0.711±0.024 for w/o G, 0.694±0.022 for w/o R, and 0.719±0.017 for w/o G+R (Figure ). Full DAIMS matched or exceeded the corresponding ablated configuration in all paired encoder‐by‐ablation comparisons. Overall, Full DAIMS provided the strongest and most consistent performance across encoders. Effect of the Frozen LLM Backbone Across all four encoders, the frozen LLM backbone outperformed two simpler fusion alternatives (Figure ). Relative to the learnable transformer, the frozen backbone achieved higher C‐index values for CONCH v1.5, UNI v1, and UNI2‐h, whereas the learnable transformer was slightly higher for Virchow2 (0.720 vs. 0.713). These results support the frozen backbone as an effective parameter‐efficient fusion strategy in three of the four evaluated encoder configurations, rather than establishing universal superiority over trainable fusion. A concatenation‐MLP baseline achieved substantially lower C‐index values (0.661, 0.636, 0.674, 0.679), while a learnable transformer partially closed the gap (0.737, 0.678, 0.659, 0.720) and exceeded the frozen backbone on Virchow2 (0.720 vs. 0.713) but underperformed on the remaining three encoders. The consistent advantage of the frozen backbone indicates that pretrained token‐level interaction modeling generalizes effectively to heterogeneous multimodal sequences without additional training. Pathology‐Only Branch on External Cohorts Finally, we evaluated the pathology‐only DAIMS branch on the two external cohorts to assess whether the multimodal‐trained backbone carries transferable risk information without target‐domain adaptation (Figure ). On F1CRC, the pathology‐only branch achieved C‐index values of 0.675, 0.683, 0.643, and 0.644 across the four encoders, ranking above most dedicated pathology‐only baselines under the same backbones. On SURGEN, it reached 0.647, 0.618, 0.623, and 0.625, remaining competitive with pathology‐only models. After lightweight adaptation, DAIMS further improved external performance across all encoders, reaching up to 0.810 on F1CRC and 0.734 on SURGEN. The gap between the full adapted DAIMS and the pathology‐only branch reflects the combined contribution of multimodal source‐domain supervision and target‐domain adaptation for cross‐cohort generalization. Interestingly, the pathology‐only branch, trained with multimodal supervision but deployed without adaptation, already outperforms pathology‐only models, indicating that multimodal training provides a lasting representational advantage. Conclusions In this study, we developed DAIMS, a domain‐adaptive incomplete multimodal stratification framework for colorectal cancer. By curating and analyzing whole‐slide images from three independent cohorts (TCGA, n=351; F1CRC, n=420; SURGEN, n=422), we assembled a heterogeneous multicenter dataset comprising more than 1,100 patients with variable acquisition conditions, outcome distributions, and molecular and clinical modality availability. DAIMS uses multimodal source‐domain supervision, PGAF, cross‐modal imputation, and lightweight WSI‐branch adaptation to address these real‐world shifts as a single transfer problem. Across evaluations spanning cohorts, treatment settings, disease stages, and molecular subtypes, the framework generally preserved prognostic stratification under heterogeneous deployment conditions. These findings suggest that histopathology can serve as a stable anchor modality, capturing clinically meaningful risk signals that remain informative despite cross‐cohort heterogeneity and incomplete multimodal inputs [, , , ]. Our results extend prior work showing that deep learning on histopathology images can support outcome‐aware patient stratification in colorectal cancer. However, many existing studies have been evaluated within single cohorts or under fully observed multimodal settings, which limits their utility in real‐world scenarios where target cohorts differ in acquisition pipeline, patient composition, and modality availability. DAIMS explicitly treats these factors as a domain adaptation problem. Incomplete multimodal learning is therefore not a parallel objective, but one mechanism that allows knowledge transfer from a fully paired source domain to WSI‐only external domains with minimal fine‐tuning and reduced dependence on cohort‐specific supervision. The consistent performance observed in both domain‐internal and domain‐external cohorts indicates that DAIMS learns shared prognostic representations rather than relying on dataset‐specific artifacts. From a biological and pathological perspective, the robustness of DAIMS across treatment‐, stage‐, and genotype‐defined subgroups suggests that histopathology images encode latent prognostic information beyond conventional clinical labels. Morphological features reflecting tumor architecture, stromal composition, immune infiltration, and tumor–microenvironment interactions are known to influence disease progression and therapeutic response in colorectal cancer. DAIMS appears to capture such signals that remain informative within stage‐ and treatment‐defined strata, as evidenced by its ability to stratify patients within standardized chemotherapy regimens and genetically defined subgroups, including BRAF‐mutant, KRAS‐mutant, and MMR‐deficient tumors []. These observations support the view that histopathology contains rich prognostic information that not only complements molecular profiling but may also partially compensate for it when molecular assays are unavailable. Across all tested encoder architectures, including CONCH v1.5 [], UNI v1, UNI2‐h [], and Virchow2 [], DAIMS outperformed pathology‐only models in most evaluated settings, including ABMIL [], ILRA [], TransMIL [], and WIKG [], as well as previously reported multimodal methods, including CMTA [], MOTCAT [], and SurvPath []. This advantage was preserved under modality‐availability shifts, where DAIMS maintained robust discrimination and yielded absolute C‐index gains of 0.04–0.05 over the strongest baselines in TCGA internal evaluation and mean gains of 10.4% and 6.6% on F1CRC and SURGEN, respectively. The frozen LLM backbone served as a fixed sequence‐level fusion to organize heterogeneous pathology, genomic, and clinical tokens without adding extra trainable parameters, and PGAF improved risk‐relevant token aggregation. The validity of this interpretation is corroborated by the performance drop when replacing the frozen backbone with a concatenation‐MLP or a learnable transformer. For target‐domain adaptation, DAIMS restricts pseudo‐label‐based updates to the WSI adapter and prediction head, while fixing the pathology encoder, imputation modules, and LLM backbone. This limited update space reduces the risk of pseudo‐label drift. The improved post‐adaptation risk‐distribution overlap shows that adaptation can recalibrate relative‐risk organization rather than amplifying unstable pseudo‐label noise. The clinical implications of these findings are substantial. DAIMS offers a practical approach for risk stratification using routinely acquired pathology slides, enabling timely prognostic assessment even in resource‐limited settings or when molecular profiling is unavailable or delayed. By identifying overall prognostic risk within stage‐ and treatment‐defined groups, DAIMS has supported risk stratification across real‐world care settings that may include concurrent procedures and treatments, such as surgery, chemotherapy, radiation therapy, and other pharmacological treatments. Future studies with standardized treatment‐response endpoints will extend this framework toward treatment‐benefit assessment. Importantly, DAIMS is not intended to replace molecular testing, but rather to complement existing diagnostic pipelines by providing an additional layer of risk information. DAIMS has provided cohort‐relative prognostic rankings, consistent with recent pathology‐based prognostic studies that have used model‐derived scores for risk stratification across cohorts [, , , ]. Site‐specific calibration with target‐domain outcomes can further support absolute risk estimation and clinical deployment. Several limitations should be acknowledged. The study cohorts differed in demographic composition: TCGA was predominantly White, F1CRC was entirely East Asian, and SURGEN was predominantly Caucasian. In addition, the representation of some molecular subgroups, including BRAF‐mutant and dMMR patients, was limited. Future validation in larger and more demographically diverse cohorts with broader representation of molecular subtypes will help further characterize the generalizability of DAIMS across these patient populations. External validation also involves endpoint heterogeneity, as TCGA‐based survival definitions may not fully align with those in external cohorts, introducing both covariate and label‐space shifts. In addition, the WSI‐only deployment setting creates an information bottleneck, as pathology cannot fully substitute for molecular or clinical variables in all cases. The contribution of modality ablation analysis showed that complete multimodal supervision provided the strongest overall performance across pathology encoders for WSI‐only inference. Future work will investigate adaptive cross‐modal alignment and modality weighting across encoder architectures. Finally, the observational design precludes causal interpretation of the learned features. Future work will focus on harmonizing endpoints across larger multi‐center datasets, incorporating emerging modalities such as spatial transcriptomics [], improving interpretability, and extending DAIMS to other cancer types. In summary, DAIMS provides an explainable, domain‐adaptive, and modality‐agnostic framework for colorectal cancer patient stratification, capable of delivering robust prognostic assessment under heterogeneous real‐world conditions. These findings underscore the potential of histopathology‐centered multimodal learning, organized around domain adaptation, to advance precision oncology and enable potentially useful prognostic risk assessment, subject to prospective validation and site‐specific calibration in real‐world practice. Experimental Methods Overview of DAIMS Framework We introduce DAIMS, a lightweight domain‐adaptive framework for real‐world CRC patient stratification under cross‐cohort modality and modality‐availability shift. DAIMS is trained on the fully paired TCGA‐CRC dataset, which includes WSIs, structured clinical reports, and genomic pathway features. WSIs were processed using TRIDENT, a scalable WSI‐processing toolkit, to perform tissue segmentation, patch tiling, and patch‐level feature extraction with pathology foundation models including UNI v1, UNI2‐h, Virchow2, and CONCH v1.5 [, ]. Encoder‐specific preprocessing, native feature dimensions, and downstream harmonization are summarized in Table . Clinical reports are embedded using BioBERT [] after LLM‐based structuring (See Table ), and pathway activity profiles are transformed via self‐normalizing networks []. DAIMS employs three Prototype‐Guided Aggregation and Fusion (PGAF) modules to select discriminative tokens from each modality and map them into a shared latent representation. To handle incomplete modalities as a deployment‐domain shift, Prototype‐Guided Cross‐Modal Representation Imputation modules learn to reconstruct clinical and genomic token embeddings directly from WSI features, enabling consistent multimodal supervision even in pathology‐only cohorts. When evaluated on the external F1CRC and SURGEN cohorts, DAIMS performs efficient domain adaptation by updating only a small subset of parameters, including the WSI adapter and the final prediction head. Iterative pseudo‐labeling further aligns latent representations to cohort‐specific staining and scanner characteristics. Prototype‐Based Feature Selection To distill informative morphological patterns from multimodal embeddings, we employ a prototype‐based feature selection module, consisting of (i) a learnable Gaussian prototype memory bank and (ii) a differentiable feature aggregation mechanism. Given input tokens x∈RB×L×d, a projection layer encodes them into latent representations z∈RB×L×D. Unlike conventional class‐specific prototypes, we define Gaussian prototypes corresponding to distinct risk stratification, each parameterized by (μk,σk) to capture the distribution of morphological patterns associated with a predicted risk level. During training, proxy samples are drawn as to provide distributional regularization in the contrastive loss. During inference, the deterministic means μk are used directly, ensuring stable and reproducible aggregation. To compress the L raw tokens into a compact set of K semantically structured representations, we introduce K learnable aggregation centroids C={c1,...,cK}, which serve as trainable slots for the downstream LLM. Token‐centroid similarity determines the attention weights: and the aggregated features are obtained as The global sequence‐level representation is then defined as the mean of the aggregated slots z ̄agg. The aggregated representation is aligned with its corresponding risk‐stratified prototype through a contrastive objective: where p(i) maps each sample to the prototype of its predicted risk label derived from clinical event y and survival status c. This loss encourages clustering around the appropriate prototype, improving discriminability and stability under domain shift. Modality Representation Imputation In real‐world clinical datasets, genomic and clinical modalities are often unavailable in the target domain, while WSIs are nearly always available. We therefore treat modality incompleteness as a modality‐availability shift within the domain adaptation setting. To enable pathology‐only inference while still leveraging multimodal supervision during training, we introduce the Prototype‐Guided Cross‐Modal Representation Imputation module. The core idea is to use the stable target‐modality prototypes learned in the previous stage as semantic anchors to guide the generation of surrogate latent tokens for unavailable modalities, rather than explicitly reconstructing the original signals. Given a source modality feature sequence s∈RB×S×ds (e.g., WSI features), DAIMS first projects it into the target latent space via a linear mapping zs=Wss. A two‐stage cross‐attention stack then transfers these source features into the structural space of the target modality. The first attention layer aligns zs with the frozen target prototypes {μk}k=1P: where γ1 is a learnable residual scaling parameter. Subsequently, K learnable query slots q∈RK×dt query the aligned representation h1 to construct target‐shaped latent tokens: A decoder then refines h2 to produce the reconstructed target tokens t^∈RB×K×dt. To ensure the fidelity of the imputation when ground‐truth target tokens tgt are available, we employ a supervised loss based on Hungarian matching. Specifically, we solve a linear sum assignment problem to find the optimal permutation π that minimizes the bipartite matching cost: which accounts for potential semantic permutations among the learnable slots. Finally, DAIMS combines the reconstructed tokens with a source‐derived baseline using an adaptive gating mechanism computed from the variability of the source features. This enables the model to generate high‐quality surrogate latent representations for missing modalities during inference using only WSIs, without requiring explicit recovery of the original modality signals. Cross‐Cohort Validation and Domain Adaptation For external validation, we benchmarked DAIMS against pathology‐only baselines (ABMIL, TransMIL, ILRA, and WIKG) on independent cohorts with differences in staining, scanning resolution, patient composition, survival distributions, and modality availability. We initialized adaptation with TCGA‐trained checkpoints and adapted each external cohort independently using unlabeled target WSIs. DAIMS produced continuous cohort‐relative risk scores. We evaluated within‐cohort discrimination using the C‐index and defined high‐ and low‐risk groups using the median predicted risk score within each cohort. Target‐domain event times and censoring indicators were not used during adaptation or score fusion. At each adaptation iteration, the pre‐update survival model generated a pseudo‐event bin y∼ and a continuous relative risk score for each target WSI. A bootstrap WSI view assessed prediction consistency and guided target‐case selection. In parallel, a frozen censor predictor trained on TCGA generated a soft censoring probability p^(c=1) for each target case. We used eight adaptation iterations with four epochs per iteration, a learning rate of 2×10−5, a WSI batch size of one, and up to 4,096 patch embeddings per slide. We updated the prediction head and WSI adapter, while keeping the pathology encoder, modality‐imputation modules, and frozen LLM backbone fixed. At each iteration, we selected up to 50% of target cases with consistent risk predictions and optimized the following soft‐censoring survival loss: The final target‐domain score was obtained through equal‐weight cohort‐rank fusion. We ranked the adapted survival risk and frozen soft‐censoring probability within each target cohort and averaged the two ranks. Model Visualization and Interpretation We further validated that DAIMS captures biologically meaningful information in colorectal cancer by developing an interpretation pipeline based on the PGAF module. Specifically, cross‐attention weights aligning raw pathology tokens with risk‐stratified prototypes were used to identify the most informative morphological patterns. By mapping attention scores from the highest‐ and lowest‐risk prototypes back to the original WSI coordinates, we generated high‐resolution heatmaps that highlight tumor regions, stromal structures, and immune cell arrangements associated with poor or favorable prognosis. These visualizations, grounded in the Virchow2 foundation model, allow us to verify that DAIMS encodes transferable morphological signals rather than institution‐specific appearance. Computational Trade‐Off To estimate practical deployment cost, we summarized model size, trainable parameters, GPU memory usage, and inference latency under a unified evaluation setting (see Table ). Each modality is passed through a lightweight adapter and fused using a frozen BioBERT‐Base v1.2 backbone []. This encoder‐only Transformer comprises 12 layers, 12 attention heads, a hidden dimension of 768, and approximately 108.3 million parameters in our implementation. All backbone parameters were frozen during source‐domain training and target‐domain adaptation, and the model was used as a fixed sequence‐level fusion module rather than as a generative language model. Across the evaluated encoders, the total parameter count remained approximately 171M, while only about 63M parameters were trainable because the language backbone was frozen. Memory use was about 0.64 GB. Inference latency was measured with a batch size of one on NVIDIA L20 (48 GB), NVIDIA RTX 6000 (96 GB), and NVIDIA H200 GPUs, using up to 4,096 pre‐extracted WSI patch embeddings per slide. The reported latency reflects patient‐level DAIMS inference after patch features are available and excludes tissue segmentation, patch tiling, and pathology‐foundation‐model feature extraction. Across the evaluated model configurations, mean per‐slide latency ranged from approximately 30 to 34 ms. For the same configuration, mean latency differed by less than 1 ms across GPU platforms. Ethics Statement This study analyzed de‐identified human data from three colorectal cancer cohorts. The TCGA and SURGEN datasets were obtained from publicly available repositories and used in accordance with their respective data access policies. The F1CRC cohort was provided by The First Affiliated Hospital of University of South China as a retrospective, de‐identified dataset. Ethical approval for the use of the F1CRC cohort was granted by the Institutional Review Board of The First Affiliated Hospital of University of South China (approval number: 2025LL1009001). Given the retrospective nature of the study and the use of de‐identified data, the requirement for informed consent was waived by the Institutional Review Board. All research was conducted in accordance with the principles of the Declaration of Helsinki. Consent Not applicable because the manuscript does not contain identifiable individual data. Contributor Information Chengyun Dou, Email: ashleydou@163.com. Junhan Zhao, Email: junhan_zhao@hms.harvard.edu.