A disease-centric vision-language foundation model for precision oncology in kidney cancer Authors: Yuhui Tao, Zhongwei Zhao, Zilong Wang, Xufang Luo, Feng Chen, Kang Wang, Chuanfu Wu, Xue Zhang, Shaoting Zhang, Jiaxi Yao, Xingwei Jin, Xinyang Jiang, Yifan Yang, Dongsheng Li, Lili Qiu, Zhiqiang Shao, Jianming Guo, Nengwang Yu, Ying Xiong, Shuo Wang Published: 2026 ABSTRACT AbstractThe non-invasive assessment of renal masses remains a critical challenge in urologic oncology, where diagnostic uncertainty frequently causes overtreatment. Here, we develop RenalCLIP, a vision-language foundation model for precision oncology in kidney cancer. Utilizing 27,866 computed tomography scans from 8809 patients across diverse multi-center cohorts, we employ a two-stage pre-training strategy to align domain-specific visual and textual representations. RenalCLIP achieves enhanced performance and generalizability across ten core clinical tasks, spanning anatomical assessment, diagnostic classification, and survival prediction, significantly outperforming state-of-the-art general-purpose foundation models. Furthermore, RenalCLIP demonstrates strong data efficiency in diagnostic classification, achieving peak baseline performance using only 20% of the training data. The model also exhibits robust zero-shot diagnostic capabilities, effective image-text retrieval, and high-quality medical report generation. Our findings establish RenalCLIP as a powerful, generalizable tool to enhance diagnostic precision, refine prognostic stratification, and personalize the management of renal masses.Subject terms: Machine learning, Renal cell carcinoma, Image processing, Diagnostic markers, Cancer imaging Introduction The widespread adoption of medical imaging has dramatically increased the incidental detection of renal masses, posing new clinical dilemmas in kidney cancer management. Despite a surge in radiologically identified renal tumors, kidney cancer-specific mortality has not declined in parallel, underscoring a substantial burden of overtreatment for benign or indolent lesions1,2. Notably, up to 20% of surgically resected renal masses are ultimately confirmed as benign, subjecting patients to unnecessary surgical risks, healthcare costs, and potential loss of renal function3,4. This clinical dilemma is most pronounced for small renal masses (≤4 cm), driven by the size-dependent prevalence of malignancy. As demonstrated by Bhindi et al.4, the probability of malignancy is 68% for tumors ≤1 cm and steadily increases to over 90% for those > 4 cm. Therefore, active surveillance is primarily considered for smaller masses and is rarely contemplated for larger tumors (>4 cm) unless dictated by limited life expectancy or high surgical risk. Current diagnostic pathways are limited: radiological interpretation remains highly subjective with considerable inter-observer variability5, while percutaneous biopsy is invasive, carries a significant non-diagnostic rate, and frequently fails to capture the true biological aggressiveness of tumors due to intratumoral heterogeneity6–9. A critical unmet need therefore exists for a non-invasive tool that can precisely and comprehensively assess a tumor’s malignant potential, aggressiveness, and prognosis from preoperative CT scans10–16. While artificial intelligence (AI) has shown promise in medical image analysis, conventional models for kidney cancer have been constrained by critical limitations that have prevented their clinical translation. Most are trained on image data alone, overlooking the rich semantic context within radiology reports that is essential for expert-level reasoning17. Furthermore, the prevailing one-model, one-task paradigm is inefficient and results in models that lack generalizability, often failing when tested on external data18,19. Recent breakthroughs in general-purpose vision-language foundation models20–24 offer a path forward but possess a critical flaw for oncology: their broad training across all pathologies lacks the necessary depth for the nuanced, high-stakes decisions required in cancer care19,25–28. A general CT model may identify an organ, but it cannot be expected to master the subtle radiological features that distinguish an indolent from an aggressive renal tumor. We reasoned that to solve a specific clinical problem, a disease-centric strategy is essential. Such an approach, which revolves entirely around the clinical and pathological complexities of a single disease, is necessary to bridge the gap between generalist AI and the demands of precision medicine. Here, we present RenalCLIP, a multi-task, disease-centric vision-language foundation model designed for comprehensive assessment of kidney cancer. RenalCLIP is pre-trained on a large-scale, multi-institutional dataset comprising 21,819 CT series and their corresponding radiology reports from 6867 patients, collected across four leading medical centers in China. By aligning visual representations from CT scans with clinical-pathological semantics extracted from textual reports, RenalCLIP learns robust, context-aware embeddings of renal tumors. We systematically evaluate its performance across a broad spectrum of clinically relevant tasks, including automated anatomical scoring, benign-versus-malignant discrimination, aggressiveness stratification, multi-endpoint survival prognosis, and cross-modal retrieval. In all benchmarks, RenalCLIP consistently outperforms existing state-of-the-art models. Moreover, the model demonstrates advanced capabilities in zero-shot learning and radiology report generation, underscoring its potential as a foundation model for precision kidney cancer imaging and decision support. Results Overview of RenalCLIP A comprehensive assessment of renal masses from CT imaging, spanning their anatomical complexity, diagnosis, and long-term prognosis, is essential for guiding personalized patient management. However, achieving this in a consistently accurate and objective manner remains a significant clinical challenge. To address this, we developed RenalCLIP, a disease-centric vision-language foundation model for renal cancer, by pre-training with a large-scale dataset of 21,819 CT scans from 6867 patients. The model’s performance on a suite of downstream clinical tasks was then rigorously evaluated on a multi-center dataset of 1942 patients (6047 CT scans). This dataset was partitioned into: (1) an internal cohort (n = 500; split into n = 400 for fine-tuning and n = 100 for test); (2) a combined external testing cohort (n = 1017), which aggregated data from five independent centers: Xiamen (n = 251), Shandong (n = 250), Ruijin (n = 229), Lianyun (n = 220), and Zhangye (n = 67); and (3) the public Cancer Imaging Archive (TCIA)29,30 cohort for independent evaluation (Fig. 1a, b, see Supplementary Tables 1–2 for detailed cohort characteristics and task-specific class distributions). Overview of the RenalCLIP design. a, b The study was developed using a large-scale, multi-center dataset. For detailed sample sizes and class distributions for each specific downstream task across all cohorts, please refer to Supplementary Table 2. a Distribution of patients and CT volumes across the pre-training, fine-tuning, and test cohorts (dots, patients; bars, CT volumes). b The cohorts encompass a diverse range of pathologic subtypes and CT phase availability, ensuring comprehensive model training and evaluation. c Schematic of the RenalCLIP framework illustrating the two-stage pre-training pipeline (left and middle panels) and its clinical application (right panel). The left panel depicts uni-modal knowledge enhancement: the image encoder is pre-trained via multi-task learning using structured labels, while the text encoder (Llama3-8B) concurrently undergoes a three-step adaptation (Phase I–III, utilizing LLM2Vec technology) to transform LLM into a text feature extractor. The middle panel illustrates cross-modal alignment, where the pre-trained image and text encoders are aligned using a CLIP-based contrastive learning framework. The right panel demonstrates the subsequent application to the three primary clinical downstream domains: characterization, diagnosis, and prognosis. See “Methods” for details. d–f Summary of model performance via radar plots, demonstrating the consistently high performance of RenalCLIP against baseline models across these primary domains. The comparison is shown for the internal test cohort (d), the combined external test cohort (e), and the TCIA cohort (f). In these plots, performance on characterization and diagnosis tasks is quantified by the area under the receiver operating characteristic curve (ROC AUC), while prognosis is quantified by Harrell’s C-index. Icons in c for the report and kidney are by Iconjam and Futuer via www.flaticon.com. Icons representing the expert and prognosis are from Microsoft PowerPoint. Source data are provided as a Source Data file. The pre-training of RenalCLIP is designed to explicitly embed disease-centric clinical knowledge into its encoders before their cross-modal alignment (Fig. 1c, Supplementary Fig. 2c–e). This is achieved through a crucial initial knowledge-enhancement stage. The image encoder is pre-trained within a multi-task learning framework, supervised by a rich set of structured attributes that were systematically extracted from radiology reports. Concurrently, the text encoder, built upon the Llama3 large language model31, is transformed into a specialized medical language expert using LLM2Vec techniques32,33. In the second stage, these knowledge-infused encoders are jointly optimized through a vision-language contrastive objective34, forcing a deep alignment between fine-grained visual features and their clinical semantics (see Methods for a detailed description of the entire pre-training pipeline). Following the completion of vision-language pre-training, we then systematically benchmarked RenalCLIP’s capabilities across a comprehensive suite of 13 distinct downstream tasks. These ranged from a core set of 10 applications across three major clinical domains—renal mass characterization (five R.E.N.A.L. score components), diagnosis (malignancy and aggressiveness), and prognosis (prediction of recurrence-free survival (RFS), disease-specific survival (DSS), and overall survival (OS))—to 3 tests of advanced capabilities, namely zero-shot classification, cross-modal retrieval, and report generation. As summarized in the performance radar plots (Fig. 1d–f, see Supplementary Fig. 1 for adjusted-scale versions), RenalCLIP consistently demonstrates improved or highly competitive performance against a range of baseline models, including a standard 3D CNN35 trained from scratch (termed CNN (Rand Init) in figures) and three general-purpose CT foundation models (CT-CLIP36, Merlin37, and CT-FM38), across the three primary task domains. The detailed evidence for its leading performance across all 13 tasks is presented in the subsequent sections (Figs. 2–5). Performance of RenalCLIP on renal mass characterization. a Rose plots summarizing the performance on the five R.E.N.A.L. score components show that RenalCLIP outperforms baseline models on most features in both the internal test cohort and the combined external test cohort (comprising the Xiamen, Shandong, Ruijin, Lianyun, and Zhangye cohorts). b Bar charts detailing the performance across the five proprietary external cohorts and the public TCIA cohort further demonstrate the model’s generalizability. c, Radiology report generation performance on the internal test cohort. Top, RenalCLIP achieves the highest scores across all standard language metrics when benchmarked against several established generative models. Bottom, Expert evaluation of generated reports. Box plots display the distribution of accuracy scores averaged across three radiologists. For the box plots, the center line indicates the median, the box limits represent the 25th and 75th percentiles, and the whiskers extend to the minimum and maximum values. The scoring criteria (0–5 scale) were adapted from HistoGPT44 (see “Methods” for details). The jittered data points represent individual case scores, showing that RenalCLIP achieves the highest median rating and consistent clinical alignment compared to baselines. d A representative qualitative example illustrates the high concordance between a report generated by RenalCLIP and the ground-truth report written by a human expert (see Supplementary Fig. 3 for comprehensive comparisons). Unless otherwise stated, all downstream evaluations utilized a standardized input protocol (single aligned contrast-enhanced CT volume, prioritizing the arterial phase). Performance on anatomical characterization (a, b) is quantified by ROC AUC. Capped error bars in (b and c) (top) represent 95% CIs estimated via 1000 bootstrap resamples and the centers correspond to the point estimates of each metric. Statistical comparisons (including significance tests) for R.E.N.A.L. scoring are detailed in Supplementary Tables 4–8, while comprehensive results for report generation are provided in Supplementary Tables 12–17. Source data are provided as a Source Data file. Performance of RenalCLIP on zero-shot transfer and data-efficient fine-tuning. a Cross-modal retrieval performance on five cohorts, quantified by Recall@K (K = 1, 3, 5). The plots for text-to-image (t2i, top) and image-to-text (i2t, bottom) retrieval show that RenalCLIP achieves higher performance compared to other vision-language models. Zero-shot classification performance for malignancy (b) and aggressiveness (c) prediction on three test cohorts. Bar plots show the mean ROC AUC from a stochastic prompt sampling strategy, while star markers indicate the performance using a deterministic maximum similarity strategy. Data-efficient fine-tuning performance on the internal (d, f) and combined external (e, g) cohorts for malignancy (d, e) and aggressiveness (f, g) prediction. The plots show model ROC AUC as a function of the fraction of training data used for fine-tuning. Dashed horizontal lines indicate the zero-shot performance of each model for reference. The results highlight both RenalCLIP’s high data efficiency (achieving strong performance with only 20–40% of data) and the benefit of its pre-training, as its zero-shot performance frequently surpasses the fully fine-tuned performance of baseline models. Error bars in (a–c) represent 95% CIs estimated via 1,000 bootstrap resamples. The centers correspond to the exact point estimates in a, and the average ROC AUC from the resamples in (b, c). In d–g, data points for training fractions less than 100% represent the mean ROC AUC from five random sampling runs, with capped error bars indicating the standard deviation; the 100% data point represents a single run. Source data are provided as a Source Data file. Automated anatomical characterization and report generation The R.E.N.A.L. nephrometry score serves as a standardized framework for quantifying the anatomical complexity of renal tumors, facilitating preoperative risk stratification and surgical planning39. We systematically evaluated RenalCLIP’s ability to predict the five constituent anatomical features—R (Radius), E (Exophytic/Endophytic), N (Nearness to collecting system), A (Anterior/Posterior), and L (Location) using ROC AUC as the primary metric (see Supplementary Table 3 for R.E.N.A.L. grading details). RenalCLIP’s leading discriminative performance was first evident in the internal test cohort, where it outperformed all comparative models on features R (ROC AUC = 0.908), E (0.646), A (0.857), and L (0.747), while achieving a competitive ROC AUC of 0.713 on feature N (vs. 0.723 for the baseline CNN). This strong performance was largely replicated in the combined external test cohort, where it again attained the highest ROC AUC for R (0.902), E (0.610), N (0.715), and L (0.727), while remaining competitive for feature A (0.754 vs. 0.757 for CT-FM) (Fig. 2a). Further analysis of individual external cohorts confirmed its robustness, with RenalCLIP achieving the highest ROC AUC in the majority of head-to-head comparisons across most features. Taken together, these results establish RenalCLIP’s superior and generalizable capability for automated, multi-component anatomical scoring across diverse clinical settings (Fig. 2b; see Supplementary Tables 4–8 for comprehensive results). To assess the clinical utility of these anatomical descriptors for surgical planning, we stratified performance by renal mass size (Supplementary Tables 9–10) and benchmarked the model against six independent board-certified experts (Supplementary Table 11). RenalCLIP demonstrated a competitive advantage in the clinically actionable cT1a (≤4 cm) subgroup, the primary candidates for nephron-sparing surgery. In this critical population, the model achieved the highest discriminative performance across all four anatomical descriptors (E, N, A, and L), consistently outperforming baseline models (Supplementary Table 10). This robust performance extended to the cT1b (4–7 cm) subgroup, confirming reliable support for complex surgical decision-making. Although performance in cT2 (>7 cm) masses exhibited variability—likely due to the substantial anatomical distortion inherent to large tumors, the blinded multi-reader study confirmed that RenalCLIP’s scoring fell well within the range of inter-observer variability (Supplementary Table 11). Notably, the model achieved top-tier consistency in objective features (R, A) and matched expert consensus in complex ones (E, N, L). This indicates that the observed performance constraints likely reflect the intrinsic subjectivity of characterizing complex renal masses rather than model-specific failure. Beyond discrete anatomical descriptors, the radiology report is the primary communication tool for translating complex imaging findings into actionable clinical insights40. We therefore evaluated RenalCLIP on the advanced task of automated report generation, which tests its ability to synthesize learned anatomical features into a coherent narrative, moving beyond the prediction of discrete scores. To this end, we benchmarked its report generation quality against state-of-the-art models, including GPT-4o41, MedGemma42, RadFM43, and CT-CHAT36. Across all standard language generation metrics (BLEU-1, BLEU-2, BLEU-4, METEOR, and ROUGE-L), RenalCLIP achieved the highest scores on the internal test cohort, indicating a stronger alignment with ground-truth reports annotated by expert radiologists (Fig. 2c, top). Crucially, this performance translated into superior clinical validity: in a blinded expert evaluation by board-certified radiologists (see Methods for details), RenalCLIP achieved the highest expert ratings for report accuracy (median score 2.67/5), establishing it as the only model to surpass the threshold for clinical acceptability (score > 2; adapted from HistoGPT44; Fig. 2c, bottom and Supplementary Table 12). A representative qualitative example comparing a RenalCLIP-generated report to the human-annotated ground truth is provided in Fig. 2d, while comprehensive comparisons against all baseline models and the quantitative results are detailed in Supplementary Fig. 3 and Supplementary Tables 13–17. Robust diagnosis of malignancy and aggressiveness stratifies patient outcomes We first assessed RenalCLIP’s capability in diagnosing the malignancy of renal masses, a critical prerequisite for determining treatment strategies45. Across both the internal and combined external test cohorts, RenalCLIP consistently achieved the highest ROC AUC among all models (Fig. 3a). Its diagnostic precision was further evidenced in the combined external cohort, where RenalCLIP attained leading performance across five key metrics: ROC AUC (0.841), PR AUC (0.941), sensitivity (0.827), specificity (0.735), and F1-score (0.876) (Fig. 3b). This superiority was maintained across the majority of the six individual external cohorts (5 out of 6, Fig. 3c and Supplementary Table 18). Crucially, the performance gap between RenalCLIP and the baseline models widened on the external cohorts, providing compelling evidence for its extended generalizability. Specifically, RenalCLIP outperformed the top baseline (CNN) by 8.2% in the internal test cohort (0.856 vs. 0.791), this margin increased to 17.3% against CNN (0.841 vs. 0.717) and 13.8% against the leading baseline (CT-FM, 0.841 vs. 0.739) in the combined external test cohort (visualized in ROC curves in Fig. 3d–e, and Supplementary Fig. 4). However, we observed a performance degradation in the public TCIA cohort (ROC AUC = 0.680). This decline is attributable to the extreme class imbalance within the TCIA dataset, where benign lesions constitute only 3.3% of the population (Supplementary Table 2), posing a statistical challenge for discrimination tasks46. Notably, this trend was consistent across all benchmarked models (e.g., CT-FM dropped to ROC AUC 0.61), yet RenalCLIP continued to outperform these baselines, highlighting its relative stability even under adverse data distributions. To dissect the drivers of this robust generalizability, we conducted controlled ablations on the malignancy diagnosis task, focusing on pre-training data scale and diversity. External performance improved consistently as pre-training volume increased from 0% to 100% (Fig. 3i). Moreover, under matched data volume (∼45%), multi-center pre-training yielded better external generalization than single-center pre-training (Fig. 3j), despite the latter’s dominance on the internal set. These results confirm that both the scale and multi-center diversity of our pre-training strategy are pivotal for ensuring robust generalizability. Performance of RenalCLIP on diagnosis of renal masses. a Rose plots summarizing diagnostic performance demonstrate RenalCLIP’s superiority in both malignancy and aggressiveness prediction on the internal test and combined external test cohorts. Sample sizes (n independent patients) are as follows: n = 100 (internal test) and n = 1017 (combined external test) for malignancy prediction; n = 99 (internal test) and n = 1005 (combined external test) for aggressiveness prediction. b A comprehensive comparison across five key performance metrics in the combined external cohort further confirms RenalCLIP’s robust diagnostic precision. c Bar charts comparing the ROC AUC for malignancy (top) and aggressiveness (bottom) prediction across the five proprietary external cohorts highlight RenalCLIP’s consistent performance advantage. Representative ROC curves visualizing the performance margin for malignancy diagnosis on the internal test cohort (d) and the combined external test cohort (e). Kaplan-Meier analysis in the TCIA cohort validates the clinical relevance of the aggressiveness prediction, showing significant RFS stratification for patients based on RenalCLIP’s predictions (f) but not for those based on the baseline model, CNN (g). h Subgroup analysis on the combined external test cohort, stratified by tumor size (Small Renal Masses (SRMs), ≤4 cm vs. large masses, >4 cm). The left panel displays the performance for malignancy classification, while the right panel shows the performance for aggressiveness classification. i Impact of pre-training data scale. External generalization improves monotonically with increasing pre-training data volume (0–100%), contrasting with the substantial performance drop on external data observed at 0% (CNN (Rand Init) baseline). j Impact of pre-training data diversity. The multi-center (controlled at ∼45% data volume) pre-trained model outperforms the single-center (Zhongshan-only) model on external test sets, validating the benefit of source diversity for robust generalization. Hazard ratios (HR) and their 95% CIs were derived from univariate Cox proportional hazards models, while P-values for overall curve comparison were calculated using a two-sided log-rank test. Shaded areas in the survival curves and capped error bars in the bar charts represent 95% CIs. For bar charts, CIs were estimated via 1000 bootstrap resamples, and the centers of the bars correspond to the exact ROC AUC point estimates. Statistical comparisons (including significance tests) for malignancy and aggressiveness diagnosis are detailed in Supplementary Tables 18–19. Source data are provided as a Source Data file. Moving beyond binary malignancy, we evaluated the model’s performance on the more nuanced task of aggressiveness stratification, which is paramount for personalized treatment selection47. Here again, RenalCLIP demonstrated superior accuracy compared to all baseline models in the internal cohort (Fig. 3a). In the combined external test cohort, RenalCLIP achieved leading performance metrics, including an ROC AUC of 0.703, PR AUC of 0.460, sensitivity of 0.713, specificity of 0.613, and an F1-score of 0.506 (Fig. 3b). A detailed cohort-level analysis further underscored its robust generalizability, showing that RenalCLIP consistently outperformed other models across most external cohorts (Fig. 3c and Supplementary Table 19). To explicitly verify that these predictions rely on intrinsic imaging phenotypes rather than tumor size, we performed a focused subgroup analysis on small renal masses (SRMs, ≤ 4 cm) within the combined external test cohort (see Supplementary Table 20 for sample sizes and class distributions stratified by tumor size). In this clinically critical subgroup, RenalCLIP maintained superior performance for malignancy diagnosis (ROC AUC = 0.762), substantially outperforming the baseline CNN (0.630), other foundation models, and the classic R.E.N.A.L. score-based nomogram48 (0.476) (Fig. 3h). Furthermore, for the challenging task of aggressiveness stratification in SRMs, RenalCLIP demonstrated strong robustness (ROC AUC = 0.655), performing comparably to CT-FM (0.653) while outperforming the baseline CNN (0.571), which exhibited performance degradation in smaller lesions. These results confirm that RenalCLIP captures size-independent features essential for early-stage diagnosis (more details can be found in Supplementary Table 21). To validate the clinical relevance of this aggressiveness classification, we investigated whether the prediction of the model could stratify patients into distinct prognostic groups, as oncologic outcomes are the ultimate manifestation of tumor aggressiveness. In the internal test cohort, tumors classified as aggressive by RenalCLIP exhibited significantly worse RFS outcomes compared to those predicted to be indolent (P = 0.005, HR = 10.5), a capability shared by most baseline models in this less challenging setting (Supplementary Fig. 5a–e). In contrast, within the independent TCIA cohort, only the predictions made by RenalCLIP retained statistically significant prognostic power (P = 0.010, HR = 2.23) (Fig. 3f). Notably, tumors stratified by any of the baseline models, including CNN (P = 0.225, HR = 1.40), CT-CLIP (P = 0.590, HR = 1.26), Merlin (P = 0.178, HR = 1.44), and CT-FM (P = 0.094, HR = 1.64), showed no significant difference in RFS (Fig. 3g, Supplementary Fig. 5f–h). This unique ability to retain prognostic significance in a challenging external cohort provides powerful, clinically meaningful evidence for the robust generalizability of RenalCLIP’s aggressiveness characterization. Non-invasive prognostic biomarker for kidney cancer survival Accurate prognosis prediction is crucial for guiding personalized follow-up strategies and adjuvant therapy decisions in renal cancer. With a lack of reliable non-invasive biomarkers12, CT-based indicators offer a promising alternative49,50. We therefore evaluated whether a risk score derived from RenalCLIP could serve as a reliable prognostic biomarker. Initial analysis showed that RenalCLIP achieved the highest Concordance Index (C-index) for predicting RFS across the internal test cohort (0.864), the combined external test cohort (0.671), and the TCIA cohort (0.726). Acknowledging the limited follow-up periods in some external sites, we focused our subsequent in-depth evaluation primarily on the TCIA cohort. Within this cohort, RenalCLIP demonstrated a substantial advantage over baseline models: it outperformed the top-performing baseline (CT-FM) by 22.6% for RFS (0.726 vs. 0.592), by 6.3% for disease-specific survival (DSS; 0.690 vs. 0.649), and by 4.3% for overall survival (OS; 0.650 vs. 0.623) (Fig. 4a). We refer the reader to Supplementary Tables 22–24 for more detailed results of model performance. Performance of RenalCLIP on prognosis of renal tumors. a Rose plots comparing Harrell’s C-index for recurrence-free survival (RFS), disease-specific survival (DSS), and overall survival (OS) demonstrate the high prognostic accuracy of RenalCLIP across the internal test cohort, the combined external test cohort, and the TCIA cohort. The total number of independent patients evaluated across these endpoints were n = 52 (internal test), n = 224 (combined external test), and n = 420 (TCIA). For the exact sample sizes per specific survival endpoint, please refer to Supplementary Table 2. b–e Kaplan-Meier analyses for RFS in the TCIA cohort validate the clinical relevance of the derived risk scores. While patient stratification based on risk scores from baseline models, including CNN (b), Merlin (c), and CT-FM (d) failed to show statistically significant survival differences, the stratification by RenalCLIP provided clear and significant prognostic separation between high- and low-risk groups (e). Time-dependent analyses in the TCIA cohort further confirm RenalCLIP’s durable prognostic power, showing its sustained superiority over baseline models across all evaluated time points as measured by ROC AUC (f), C-index (g), and Brier score (h). i, Forest plot illustrating the prognostic performance for RFS specifically within the SRMs subgroup of the TCIA cohort. In the forest plot (i), the markers represent the exact hazard ratios (HR), and the error bars represent the 95% CIs. HR and their 95% CIs shown on survival curves (b–e) and forest plot (i) were derived from univariate Cox proportional hazards models; P-values for (b–e) were from a two-sided log-rank test. Shaded areas in the survival curves (b–e) represent 95% CIs of survival probability. Source data are provided as a Source Data file. To assess its clinical utility in risk stratification, we categorized patients into high- and low-risk groups based on the cohort-specific median score (see Supplementary Table 25 for specific thresholds). Kaplan-Meier analysis in the TCIA cohort revealed that RenalCLIP provided the most significant and clearest risk stratification for RFS. Patients in the RenalCLIP-defined high-risk group had significantly worse survival than the low-risk group, exhibiting the highest hazard ratio among all models (P < 0.001, HR = 3.7) (Fig. 4b–e). Furthermore, RenalCLIP demonstrated strong and statistically significant prognostic power for both DSS and OS (Supplementary Fig. 6a–j). We further validated the model’s prognostic value for RFS specifically within the SRMs subgroup of the TCIA cohort, an indolent population where risk stratification is difficult. Despite the limited number of recurrence events, RenalCLIP successfully identified high-risk patients with statistical significance (P = 0.032, HR = 2.28), whereas baseline models failed to generalize to this subgroup (Fig. 4i, Supplementary Fig. 6k–o, and Supplementary Table 26). To confirm that this prognostic value was independent of established clinical standards, we next performed multivariate Cox regression analysis. After adjusting for standard pathological indicators such as TNM stage and WHO/ISUP grade, the RenalCLIP risk score remained an independent adverse prognostic factor for RFS in the TCIA cohort (P = 0.016, HR = 2.27), while the risk scores derived from all baseline models failed to retain statistical significance (see Supplementary Table 27 for details). Furthermore, we performed time-dependent analyses to evaluate the stability of the model’s predictive accuracy over time. For RFS, RenalCLIP consistently outperformed all baseline models at all evaluated time points, as measured by ROC AUC (Fig. 4f), C-index (Fig. 4g), and Brier score for prediction error (Fig. 4h). Similar improvements over baselines were observed for DSS and OS across the same time range (Supplementary Fig. 7a–f). Collectively, these comprehensive analyses establish RenalCLIP as a robust and independent prognostic biomarker. Its ability to non-invasively predict patient outcomes from pre-operative imaging offers significant potential to refine clinical trial enrichment, guide personalized management, and improve patient counseling. Zero-shot generalization and data efficiency A critical barrier to the widespread clinical deployment of deep learning models is their heavy reliance on large, expert-annotated datasets, which are scarce and labor-intensive to create51,52. To overcome this, we designed RenalCLIP to excel in low-data regimes. We rigorously evaluated two of its hallmark capabilities: first, its ability to generalize to novel tasks via zero-shot transfer without any fine-tuning34, and second, its data efficiency when adapted to new tasks with limited labeled samples. To quantitatively validate the quality of the shared embedding space learned by RenalCLIP, we assessed its cross-modal retrieval performance. This task serves as a direct probe of the model’s vision-language alignment and has practical applications in cohort identification and medical education. We benchmarked RenalCLIP against other vision-language models (CT-CLIP and Merlin) using Recall@K (K = 1, 3, 5) on five cohorts containing radiology reports. RenalCLIP substantially outperformed both baselines across all cohorts in both text-to-image (t2i) and image-to-text (i2t) retrieval (Fig. 5a and Supplementary Tables 28–32). For instance, in the t2i task, RenalCLIP achieved a mean Recall@5 of 0.120 across the five cohorts. This performance exceeds the random guess baseline (0.039) and represents an improvement over existing state-of-the-art medical VLMs, such as Merlin (0.051) and CT-CLIP (0.004). Building on this strong cross-modal alignment, we next evaluated the model’s zero-shot classification performance on malignancy and aggressiveness diagnosis. To ensure a robust evaluation against linguistic variations, we employed a prompt ensemble methodology and assessed performance under two distinct strategies: a deterministic approach using maximum similarity and a stochastic prompt sampling strategy (see Supplementary Tables 33–35 for prompts used for each class in zero-shot classification and Methods for details). Our results show that RenalCLIP is robust to the choice of evaluation strategy and consistently outperforms baseline models. Using the primary maximum similarity ensemble strategy, RenalCLIP substantially outperformed all baselines on both tasks across all three test cohorts (Fig. 5b–c and Supplementary Tables 36–37). On the challenging TCIA cohort, for example, RenalCLIP achieved a ROC AUC of 0.730 for malignancy prediction, significantly outperforming Merlin (0.664) and CT-CLIP (0.605). A similar margin of performance gap was observed for aggressiveness prediction, where RenalCLIP (ROC AUC = 0.657) again surpassed both Merlin (0.591) and CT-CLIP (0.412). Notably, in a stratified analysis of challenging SRMs (≤ 4 cm), RenalCLIP retained its lead in zero-shot malignancy detection (ROC AUC 0.681 vs. 0.490–0.622 for baselines, Supplementary Table 38), though zero-shot capability for aggressiveness was limited across all models due to the lack of macroscopic correlates. This underscores that while disease-centric pre-training captures fundamental diagnostic features, explicit supervision remains indispensable for resolving the subtle pathological grading of small lesions, as evidenced by the substantial performance recovery achieved with fine-tuning (Malignancy ROC AUC 0.762; Aggressiveness ROC AUC 0.655, Supplementary Table 21). Finally, we investigated the value of RenalCLIP’s pre-trained features for downstream data efficiency. When fine-tuned, RenalCLIP demonstrated strong data efficiency, achieving performance comparable to fully-trained baseline models while using only 20–40% of the labeled images on both malignancy and aggressiveness tasks (Fig. 5d–g). Exceptions were observed in the TCIA malignancy task (Supplementary Fig. 8a) due to extreme class imbalance (Supplementary Table 2). Additionally, for aggressiveness prediction, external evaluation (combined external test and TCIA) performance peaked at intermediate data fractions (40–80%). This suggests that utilizing the full source dataset may induce overfitting to source-specific characteristics, whereas a curated subset yields better generalization53. However, even with the observed performance drop at 100% data usage, RenalCLIP consistently outperformed all comparative baselines. The power of its pre-trained representations was most strikingly evident when comparing its zero-shot performance directly against fine-tuned models. For malignancy prediction, RenalCLIP’s zero-shot model surpassed the peak performance of all baseline models—even after they were fully fine-tuned on 100% of the data—across both internal and combined external cohorts. When compared against itself on the external cohort, its zero-shot generalization was so effective that it outperformed the RenalCLIP model fine-tuned on up to 80% of the data and was comparable to the fully fine-tuned version (Fig. 5d–e). For the more challenging aggressiveness task, although fine-tuning on RenalCLIP improved performance over its zero-shot baseline (even with only 10% of the data), this zero-shot performance itself remained highly competitive against the other fine-tuned models; it exceeded the peak fine-tuned results of the CNN, CT-CLIP, and Merlin models in the internal cohort, and its performance in the external cohort was comparable to that of the CNN, Merlin, and CT-FM models when fine-tuned on 40% of the data (Fig. 5f–g). The detailed analysis for the TCIA cohort, which showed similar trends of strong zero-shot performance, is provided in Supplementary Fig. 8. We refer readers to Supplementary Tables 39–40 for more detailed reporting of model performance. Discussion In this study, we developed and validated RenalCLIP, a disease-centric vision-language foundation model that presents a framework for the comprehensive, non-invasive assessment of renal masses from CT imaging. By uniquely integrating deep visual features with the rich semantic context of radiological reports through a specialized knowledge-enhancement strategy, RenalCLIP demonstrates robust performance, generalizability, and data efficiency across a wide spectrum of clinical tasks compared to both conventional AI models35 and existing general-purpose CT foundation models36–38. The management of renal masses presents a cascade of clinical challenges where precision is paramount5–16. For younger, otherwise healthy patients with a long-life expectancy, the non-invasive differentiation of benign from malignant tumors remains the critical diagnostic objective to prevent the overtreatment of benign lesions. However, the concept of overtreatment is highly dependent on patient demographics. For older patients with significant comorbidities and limited life expectancy, overtreatment frequently involves the unnecessary management of frankly malignant but indolent neoplasms. In this demographic, distinguishing aggressive neoplasms from benign and indolent malignant masses is arguably the more consequential clinical priority54. Furthermore, these clinical challenges extend to the accurate prediction of postoperative prognosis to personalize follow-up and adjuvant treatment decisions. Current imaging interpretation struggles to meet these needs, and the utility of percutaneous biopsy is limited by significant non-diagnostic rates and potential complications. Aiming to address these challenges, prior AI research has largely focused on radiomics and conventional deep learning. For instance, in malignancy diagnosis, one of the largest CT-based radiomics models achieved an ROC AUC of 0.75 in a cohort of 735 patients55. Similarly, for aggressiveness assessment, a notable radiomics signature developed to distinguish low-grade from high-grade clear cell renal cell carcinoma (RCC) reached an ROC AUC of 0.70 in 587 patients56. However, the clinical translation of such models faces multifaceted challenges, including data governance and privacy constraints, regulatory compliance, and integration into clinical workflows. In addition, from a methodological standpoint, the field has often been limited by single-task development paradigms, relatively small cohorts, and insufficient independent external validation, which together can hinder robustness and real-world generalization; indeed, in the few studies where external evaluation was performed, these models often exhibited poor generalizability. More recent general CT foundation models, while powerful, lack the disease-centric focus required for high-stakes, nuanced clinical questions in oncology. Although models like CT-FM38 and Merlin37 demonstrate versatility in broad tasks such as whole-body segmentation, and CT-CLIP36 excels in chest imaging, their pre-training corpora are not enriched with sufficient, high-quality examples of renal cancer. Our head-to-head comparisons confirmed this limitation: their performance on fine-grained tasks like aggressiveness prediction and prognostication was significantly inferior to RenalCLIP. In contrast, RenalCLIP is explicitly designed to fill this gap by integrating deep visual features with renal-specific clinical-pathological semantics. This disease-centric design, which infuses the model with specialized knowledge through a unique pre-training strategy, enables RenalCLIP to excel in nuanced oncology tasks. It offers a level of precision in aggressiveness stratification and prognostication that general-purpose foundation models, which lack this targeted training, cannot match. It is also important to position RenalCLIP relative to highly optimized, task-specific deep learning models, such as the recent work by Xiong et al. on CNN-based classification of renal mass malignancy and aggressiveness57. While fully supervised specialist models trained on extensive downstream datasets (e.g., ∼2400 cases in prior work) can achieve high peak performance on singular endpoints, direct numerical comparisons are confounded by differing evaluation protocols. The present study intentionally adopts a more rigorous, clinically realistic setting—incorporating highly heterogeneous multi-center cohorts and strictly retaining cases with missing contrast phases that were previously excluded. Under these challenging real-world conditions, RenalCLIP achieves comparable diagnostic accuracy (ROC AUC 0.84 vs. 0.87 for malignancy diagnosis) while utilizing less than 17% of the fine-tuning data (∼400 cases). This highlights the exceptional data efficiency and feature transferability of our disease-centric pre-training paradigm, overcoming the common bottleneck of data scarcity in clinical AI. Ultimately, aligning with the emerging consensus on generalist medical AI58,59, RenalCLIP represents a transition from a specialist to a generalist: it consciously trades marginal peak optimization on a single task for broad generalizability, zero-shot inference capabilities, and multi-modal proficiency (e.g., complex report generation and cross-modal retrieval) that traditional CNNs fundamentally cannot achieve. A key advantage of RenalCLIP is its robust generalizability, which we attribute to our disease-centric strategy coupled with the large scale of our multi-center pre-training dataset. It was demonstrated through three key findings. First, the performance gap between RenalCLIP and baseline models widened substantially in external cohorts. For malignancy diagnosis, RenalCLIP’s margin over the top CNN baseline expanded from 8.2% in the internal cohort to 17.3% in the combined external evaluation, underscoring its robustness against domain shifts. Second, while predictions from both RenalCLIP and most baseline models could significantly stratify patient RFS in the internal test cohort, only those from RenalCLIP retained this statistically significant prognostic capability in the independent TCIA cohort, where all baselines failed. Finally, despite being pre-trained predominantly on data from Chinese medical centers, RenalCLIP consistently outperformed all baseline models across all tasks in the independent TCIA cohort, supporting its generalizability across datasets with different population compositions. Beyond its robust generalizability, the power of RenalCLIP’s pre-trained representations was most evident in its strong data-efficient learning capability, which addresses a fundamental barrier to the translation of AI in medicine: the scarcity of large, expert-annotated datasets. As observed, conventional 3D CNNs trained from random initialization stagnated even when the sample size increased tenfold (40 to 400 subjects, Fig. 5f). Interestingly, while 400 subjects begin to trigger a discernible learning trend for the relatively macroscopic task of malignancy diagnosis (Fig. 5e), this volume falls short of the data threshold required to extract the subtle features necessary for aggressiveness stratification, resulting in a flat learning curve. In contrast, RenalCLIP effectively circumvents this bottleneck, achieving peak baseline performance with minimal annotations and demonstrating strong zero-shot capabilities. This is not merely a technical achievement; it signifies a potential paradigm shift in how specialized medical AI tools are developed. This capability opens a feasible pathway for rapidly extending the model to other critical but data-scarce problems in renal oncology. For instance, developing classifiers for rare but aggressive histologic subtypes (e.g., collecting duct carcinoma, medullary carcinoma), predicting response to targeted therapies, or identifying genomic markers from imaging currently suffers from a lack of sufficient training examples. The strong few-shot and zero-shot performance of RenalCLIP demonstrates its capability to learn robust representations from limited data. While specific evaluation on rare subtypes remains a subject for future work, these results suggest that RenalCLIP holds promise as a foundational platform to address data-scarce, long-tail clinical challenges where collecting large-scale training datasets is impractical. Finally, the potential clinical value of these advanced capabilities was explored in the evaluation of prognostic endpoints. For aggressiveness stratification, many models, including our baselines, could separate prognostic groups in the internal cohort. However, the independent TCIA cohort provided a more rigorous test of generalizability, where RenalCLIP’s predictions retained statistically significant prognostic power. These findings highlight the potential future utility of such approaches. If further validated and refined, a non-invasive imaging marker for aggressiveness, available at the time of initial diagnosis, could eventually support patient management by assisting in the consideration of active surveillance for low-risk tumors, and informing decisions on adjuvant therapy trials for high-risk patients post-nephrectomy. Despite these promising results, our study has several limitations that must be acknowledged. First, while RenalCLIP demonstrated improved performance over baseline models in characterizing SRMs, arguably the most challenging yet clinically impactful subgroup, its current diagnostic accuracy in this specific scenario remains insufficient to independently guide clinical decisions. Further algorithmic refinements and larger, targeted training datasets are necessary to achieve the reliability required for routine clinical deployment in SRMs. Second, the analysis was retrospective, which may introduce selection and spectrum biases; prospective validation in a real-world clinical workflow is essential to confirm its utility. Third, although the independent TCIA cohort provided external evaluation beyond the proprietary Chinese cohorts, our pre-training data were predominantly derived from Chinese medical centers. Further training and evaluation using more geographically and demographically diverse datasets are warranted. Fourth, while RenalCLIP’s report generation outperformed baselines, it is not yet at human expert level and can still exhibit minor factual inconsistencies, highlighting that high-fidelity medical report generation remains a significant challenge. Finally, our evaluation, while comprehensive, did not extend to predicting genomic profiles (e.g., BAP1, PBRM1 mutations) from imaging, which is a critical frontier in radiogenomics and an important avenue for future work. In conclusion, RenalCLIP provides a robust, multi-task foundation for renal cancer assessment that bridges the gap between radiological imaging and clinical semantics. This work establishes the necessity and effectiveness of a disease-centric pre-training strategy for building high-performing, generalizable, and trustworthy medical foundation models. This disease-centric paradigm serves as a blueprint for developing similar specialized AI systems in other areas of oncology, potentially accelerating the integration of artificial intelligence into the core of personalized patient care. Methods The study followed the Declaration of Helsinki and complied with all relevant ethical regulations. The study was approved by the ethics committees of Zhongshan Hospital, Fudan University; the First Affiliated Hospital of Zhejiang University; Qilu Hospital of Shandong University; Linyi People’s Hospital; Xiamen Branch of Zhongshan Hospital; Shandong Cancer Hospital & Institute; Ruijin North Hospital of Shanghai Jiao Tong University; the First People’s Hospital of Lianyungang; and Zhangye People’s Hospital of Hexi University. The requirement for written informed consent was waived by the ethics committees due to the retrospective nature of the analysis, which involved pre-existing anonymized CT images and posed no additional risk to the patients. Information on race, ethnicity and ancestry was not collected for the proprietary cohorts and was not used for model development, validation or testing. Study cohorts and data preprocessing Study population We screened patients who underwent nephrectomy or biopsy for suspicious renal masses between January 1, 2009, and October 30, 2023, from nine medical centers in China and the public Cancer Imaging Archive (TCIA). The key inclusion criterion was the availability of preoperative multi-phase CT scans. Patients were excluded if they had received neoadjuvant therapy, had a diagnosis of hereditary renal cell carcinoma (RCC), or if their CT scans exhibited severe artifacts. Sex information, reported as male sex (%) in Supplementary Table 1, was determined based on self-report. Sex was summarized as a cohort characteristic but was not used as a stratification variable in the study design. Sex-stratified performance analyses were not performed because this study was designed to evaluate model performance across predefined renal imaging tasks and cohorts rather than to assess sex-specific effects. This screening process yielded a final study population of 8809 patients providing a total of 27,866 preoperative CT scans, which were subsequently divided into pre-training and downstream task datasets. Crucially, this division was conducted strictly at the patient level. This ensured that all scans associated with a single patient were assigned exclusively to either the training, validation, or test cohort, thereby preventing any data leakage. The pre-training dataset was sourced from four major medical centers (Zhongshan Hospital of Fudan University, The First Affiliated Hospital of Zhejiang University, Qilu Hospital of Shandong University, and Linyi People’s Hospital) and comprised a total of 6867 patients (21,819 CT scans). This dataset was further partitioned into a training set of 6367 patients (20,278 scans) and a validation set of 500 patients (1541 scans). The validation set was used exclusively for selecting the best-performing model checkpoint during the pre-training phase. The downstream task dataset, used for fine-tuning and evaluation, consisted of a separate cohort of 1942 patients (6047 CT scans). This included an internal cohort sourced from the pre-training institutions, except for Linyi People’s Hospital, which was partitioned into a training set of 400 patients (1240 scans) and a test set of 100 patients (316 scans). For comprehensive external evaluation, we assembled five proprietary cohorts from the following institutions: Xiamen Branch of Zhongshan Hospital (termed the Xiamen cohort; 251 patients, 796 scans), Shandong Cancer Hospital & Institute (termed the Shandong cohort; 250 patients, 991 scans), Ruijin North Hospital of Shanghai Jiao Tong University (termed the Ruijin cohort; 229 patients, 727 scans), the First People’s Hospital of Lianyungang (termed the Lianyun cohort; 220 patients, 671 scans), and Zhangye People’s Hospital of Hexi University (termed the Zhangye cohort; 67 patients, 259 scans). These five cohorts were aggregated to form the combined external test cohort used for pooled analyses. The public TCIA cohort (425 patients, 1047 scans) was used as an additional, independent evaluation set. Radiology reports were unavailable for the Lianyun and TCIA cohorts, while such reports were accessible for all other cohorts. The detailed clinicopathological characteristics of all cohorts are provided in Supplementary Table 1. Histologic classification Renal tumors were classified into three categories—benign indolent, malignant indolent, and malignant aggressive—based on previous research4,57. Benign indolent renal masses included angiomyolipoma, complex renal cyst, oncocytoma, mixed epithelial and stromal tumor, metanephric adenoma, among others. Malignant indolent renal tumors encompassed indolent clear cell renal cell carcinoma (ccRCC, defined as ccRCC without invasion into major veins or perinephric tissues, grade 3–4 components, necrosis, or sarcomatoid differentiation), indolent papillary RCC (papillary RCC without invasion into major veins or perinephric tissues, grade 3–4 components, or sarcomatoid differentiation), chromophobe RCC, clear cell papillary renal cell tumors, multilocular cystic renal neoplasm of low malignant potential, epithelioid angiomyolipoma, and others. Malignant aggressive subtypes included aggressive ccRCC (non-indolent ccRCC), aggressive papillary RCC (non-indolent papillary RCC), TFE3-rearranged renal cell carcinoma, renal cell carcinoma NOS, and other high-risk variants. Tumor stage at the time of surgery was determined according to the 8th edition of the AJCC cancer staging system. For cases with available slides, three pathologists re-evaluated tumor subtypes in accordance with the 2022 WHO classification for renal tumors and tumor nuclear grades based on the WHO/ISUP grading system for cases with available slides. The detailed distribution of these histologic subtypes and tumor grades for the entire cohort is provided in Supplementary Table 1. CT image acquisition and preprocessing To ensure spatial correspondence across the multi-phase CT scans (non-contrast (N), arterial (A), venous (V), and delayed (D)), we performed inter-phase rigid alignment within each patient based on DICOM geometry. Specifically, all available phases were mapped to the coordinate system of the arterial phase and resampled onto the arterial reference grid using linear interpolation with the Resample Image (BRAINS) module in 3D Slicer (v4.11.20210226). If the arterial phase was unavailable, the venous phase was used as the reference (all cases included at least one contrast-enhanced phase). Following this alignment, we employed a semi-automated pipeline to localize the region of interest (ROI) (Supplementary Fig. 2a). A pre-trained nnU-Net60 model was first utilized to automatically segment the kidneys, renal cysts, and tumors on the contrast-enhanced phase (arterial, or venous if arterial was unavailable), where lesion boundaries are most distinct. The resulting high-quality segmentation masks were then propagated to the spatially corresponding non-contrast phase of the same patient. The subsequent localization strategy differed by task. This distinction was intentional, reflecting the different objectives of each stage: the pre-training phase was designed to maximize data diversity and learn robust features from the entire kidney, where precise lesion-centric localization was less critical, whereas the downstream tasks required highly accurate ROI definition centered on the lesion to learn task-specific diagnostic features. For the pre-training dataset, the ROI was automatically defined as the geometric center of the segmented foreground; in the rare event that the segmentation model produced no foreground mask, the kidney center was then manually annotated with a single point by an expert radiologist. For the downstream task datasets, an initial ROI center was proposed by identifying the axial slice with the largest cross-sectional area of the primary lesion (tumors prioritized over cysts) and calculating the lesion’s geometric center on that slice. To ensure localization accuracy, an expert radiologist reviewed the quality of the underlying nnU-Net segmentations for all 1,942 cases in the downstream datasets. The automatically calculated center point was used for all accurately segmented lesions (approximately 90% of cases). In the remaining 10% of cases with suboptimal segmentation, the radiologist instead manually placed a single point to define the lesion center. All manual point annotations were performed using 3D Slicer. Following the established protocol57,61, a 3D volume centered on the localized ROI was cropped to a fixed physical size of 140 × 140 × 160 mm. These volumes were then resampled to an anisotropic spacing of 1.0 × 1.0 × 5.0 mm, yielding a uniform input size of 140 × 140 × 32 voxels. Finally, image intensities were normalized by windowing with a window level of 50 Hounsfield Units (HU) and a window width of 500 HU. The strategy for selecting input volumes also varied by task. For pre-training, cropped volumes from both kidneys of each patient were utilized. For downstream tasks, only the kidney containing the primary lesion was included; in cases with bilateral lesions, the lesion determined to be most aggressive via pathology reports was selected. During model training, a 128 × 128 × 32 voxel patch was randomly cropped from the larger 140 × 140 × 32 volume for data augmentation. During inference, a deterministic center crop of the same size was used. For all comparison models, we utilized their officially provided preprocessing functions and default configurations. The input for these models was based on the same lesion-centered ROIs generated by our pipeline to ensure a fair comparison. Radiology report processing The original free-text radiology reports, sourced in Mandarin Chinese, underwent a two-step processing pipeline to align with the single-kidney image inputs. First, given that our image encoder processes individual kidneys, we developed a large language model (LLM)-based method41 to parse each report and partition the descriptive text into left kidney and right kidney components. Second, following this anatomical separation, the partitioned Chinese texts were programmatically translated into English to facilitate model development (Supplementary Fig. 2b). The specific prompts designed for both the report partitioning and translation tasks are detailed in the Supplementary Tables 41–42. Multi-phase CT data handling and fusion strategy Our core strategy for handling multi-phase CT scans treats each available phase from a patient’s scan as an independent 3D sample. This approach serves two primary objectives: first, to substantially augment the effective size of our training dataset, and second, to implicitly regularize the model by exposing it to a wide variety of imaging contexts. During the pre-training stage, our data loader randomly samples a single existing phase (selected from all available phases: non-contrast, arterial, venous, or delayed) for each patient within a batch. This forces the model to learn a robust feature representation that is agnostic to the specific enhancement characteristics of any single phase. For the primary downstream validation experiments, however, we adopted a standardized single-phase input to ensure fair comparison against baseline models, which typically utilize a single 3D volume (Supplementary Fig. 10a). Specifically, we prioritized the arterial phase, followed by the venous phase if the former was unavailable. This hierarchy reflects established clinical practice, as the arterial phase optimally demonstrates tumor vascularity and the venous phase is crucial for assessing both tumor margin and contrast washout patterns. Finally, to quantify the performance benefit of integrating multi-phase information, we conducted an ablation study. For this study, a model was first fine-tuned on the downstream tasks using a random single-phase sampling strategy, analogous to the method used in pre-training stage. At inference time, this trained model was then evaluated using a late-fusion approach: each of a patient’s available 3D phase volumes was independently passed through the model to generate phase-specific logits. These logits were then combined via element-wise averaging to produce a single, fused prediction. RenalCLIP pre-training RenalCLIP is a self-supervised learning model specifically designed for the RCC CT imaging domain. It builds upon the success of established language-image pre-training (e.g., CLIP34) in the natural image domain. RenalCLIP integrates two core components: an image encoder f⋅;θf and a text encoder g⋅;θg. The image encoder f⋅;θf is composed of a backbone fbackbone⋅, specifically a 3D ResNet1835 adapted for volumetric CT data that yields a 512-dimensional feature vector, and a projection head fproj⋅, where θf collectively denotes the parameters of both modules. The projection head maps the high-dimensional features extracted by the image backbone directly to a single global image representation of 4096 dimensions, specifically designed with the same dimensionality as the text features to enable cross-modal alignment through contrastive learning. The text encoder g(⋅;θg) is built upon the Llama3 8B architecture31 and leverages LLM2Vec32 technology. It comprises 32 transformer layers, featuring an embedding dimension of 4096 and a hidden dimension of 14,336. A global representation of each caption is extracted by taking the mean of the last_hidden_state outputs across all its tokens. The text encoder does not employ an additional projection head and remains frozen during this phase, motivated by recent evidence62,63 that text embeddings from a fixed pre-trained text encoder are highly effective for guiding visual representation learning. Uni-modal pre-training via knowledge enhancement Previous research22,64 suggests that sequentially pre-training uni-modal modules prior to joint visual-language pre-training, significantly improves zero-shot transfer performance. Following this strategy, both our image and text encoders undergo independent pre-training. Image encoder pre-training We pre-trained the image encoder using a multi-task learning approach, leveraging structured labels derived from original radiology reports. Clinical experts designed 14 critical questions, each with several options, focusing on key aspects relevant to kidney cancer imaging interpretation. An LLM (GPT-4o41) then parsed the free-text reports based on these question-option pairs to extract the required structured attributes, capturing the semantic hierarchy. To validate the fidelity of this automated extraction, a board-certified radiologist manually audited a random subset of 100 kidneys, yielding an accuracy of 93.6% (786/840 verifiable items). This high concordance confirms that the extracted labels are reliable for pre-training. These extracted results were converted into a one-hot encoding format for model training. The image backbone was pre-trained by minimizing a classification loss (categorical cross-entropy), defined as: where N denotes the number of samples, and C represents the number of classes for each question. 1[·] is the indicator function, which equals 1 if the specified condition is satisfied and 0 otherwise. y(i) is the ground truth label of the i-th sample, and y^j(i) denotes the predicted probability that sample i belongs to class j. This process was crucial for injecting domain-specific clinical knowledge into the image encoder, enhancing its ability to capture features vital for RCC diagnosis and characterization from CT images. Specific questions, options, and prompts for structured attribute extraction are in Supplementary Tables 43-44. Text encoder pre-training Our text encoder is built upon the Llama3 8B architecture and leverages LLM2Vec technology, which transforms state-of-the-art LLMs into potent text feature extractors. Its core methodology involves three key steps: first, replacing the LLM’s causal attention mechanism with a bidirectional structure to enhance contextual semantic understanding. Second, masked language modeling65,66 (MLM) is applied. In this unsupervised framework, parts of words in reports are masked, and the model is trained to predict their original identity based on their bidirectional context, thereby capturing rich contextual representations. This step fine-tunes the LLM using LoRA67 (Low-Rank Adaptation) on the MIMIC-CXR dataset68 (227,973 chest X-ray reports) to significantly improve general medical domain comprehension. The MLM loss is formulated as: where N denotes the number of samples, M represents the set of masked tokens, M is the number of elements in M, V is the vocabulary size, yt is the ground truth label for token t, and p^v(t) denotes the predicted probability that token t corresponds to the v-th vocabulary item. Third, contrastive learning with SimCSE69 further refines the text encoder’s understanding of kidney cancer-related terminology. Here, an input sentence is passed twice through the model with independently sampled dropout masks to generate representations hi and its distinct positive representations hi+, which are then optimized to maximize their similarity, while simultaneously minimizing their similarity with negative samples (representations of other sentences in the batch). This SimCSE objective function is formulated as follows: where N is the number of samples in a mini-batch, τ is a temperature hyperparameter, and simhi,hj denotes the cosine similarity between hi and hj, which is defined as: where ∣∣hi∣∣ and ∣∣hj∣∣ denote the L2 norms of hi and hj, respectively. This fine-tuning is also conducted using LoRA and is applied to the training set of our internal pre-training dataset. Cross-modal visual-language pre-training Following uni-modal pre-training, RenalCLIP undergoes cross-modal visual-language pre-training34. A mini-batch is constructed from N image-report pairs, denoted as xi,rii=1N, where xi is the i-th CT volume and ri is a sequence of T word tokens representing its corresponding radiological report. For each pair xi,ri, the L2-normalized image representation ui is obtained from the cascaded backbone and projection head of the image encoder f⋅;θf, and the L2-normalized report representation vi is derived from the text encoder g⋅;θg. The model is trained to align positive (matching) image-report pairs and push apart negative (non-matching) pairs within the shared embedding space using a symmetric InfoNCE loss. The complete objective is given by: where simu,v denotes the cosine similarity between representations u and v, and τ is a learnable temperature parameter that scales the similarity scores. This dual-sided objective ensures that both image-to-text and text-to-image alignments are optimized simultaneously, driving the model to learn semantically rich and aligned representations for renal CT volumes and their corresponding reports. Pre-training configuration Our uni-modal pre-training involved separate stages for the image backbone and the text encoder. The image backbone was pre-trained using an AdamW optimizer70 on a single NVIDIA A100 80GB GPU. Training spanned up to 200 epochs with a batch size of 300 and a learning rate of 5 × 10−4. After each epoch, the model’s performance was validated on the pre-training validation set, and the best-performing model checkpoint was saved based on the highest macro-averaged ROC AUC across 14 key attributes, reflecting its strong multi-task learning capabilities. The text encoder pre-training, utilizing LLM2Vec, consisted of two sequential steps, both optimized with AdamW on a single NVIDIA A100 80GB GPU. To reduce computational overhead, both steps incorporated gradient checkpointing71 and mixed-precision training (bfloat16). Each step was trained for 1000 iterations with a batch size of 32. The initial phase, leveraging MLM, used a learning rate of 5 × 10−5 and an MLM probability of 0.2. This was followed by the SimCSE phase, which employed a learning rate of 3 × 10−5 and a SimCSE dropout rate of 0.2. Detailed hyperparameters for uni-modal pre-training are provided in Supplementary Tables 45–47. Our visual-language pre-training experiments were conducted on four NVIDIA A100 80GB GPUs, utilizing a local batch size of 1024 per GPU and mixed-precision training (bfloat16) for enhanced efficiency. We employed AdamW as the optimizer. The image encoder’s backbone was trained with an initial learning rate of 1 × 10−5, while its projection head used 5 × 10−4. The model was trained for 100 epochs, incorporating a 10-epoch warmup phase. Following each training epoch, we evaluated cross-modality retrieval performance on the pre-training validation set. The checkpoint demonstrating the best retrieval performance was selected as the final model weights. For more detailed cross-modal pre-training hyperparameter configurations, please refer to Supplementary Table 48. For image data, augmentation strategies included random crop, random affine transformations, and variations in signal intensity. For more details about data augmentation techniques for 3D CT volumes, please refer to Supplementary Table 49. Regarding text data, features derived from the frozen text encoder were pre-computed and saved offline to disk to minimize computational overhead during training. For text data augmentation, we used sentence shuffling: for each original radiological report, we generated and saved four shuffled versions alongside the original. Evaluation of downstream tasks Baseline models and comparisons We benchmarked RenalCLIP against a set of state-of-the-art baseline models. The specific models used for comparison varied by the downstream task, reflecting a comprehensive evaluation against both vision-only architectures and vision-language models (VLMs). Please refer to Supplementary Table 50 for a comprehensive summary of model sources, architectures, and training protocols across all tasks. For end-to-end fine-tuning on characterization, diagnosis, and prognosis tasks, we included four 3D models. A standard 3D ResNet-1835, trained from scratch with random weight initialization (CNN (Rand Init)), served as a baseline to quantify the performance gains attributable to pre-training. We also included CT-FM38, a 3D vision-only foundation model with a Seg-ResNet backbone72, which was pre-trained on 148,000 whole-body CT scans from the Imaging Data Commons. The set of baselines was completed with two 3D VLMs: CT-CLIP36, a 3D VLM that pairs a Vision Transformer73 encoder with a text encoder and was pre-trained on over 50,000 chest CTs, and Merlin37, a 3D VLM designed for abdominal imaging that was pre-trained on approximately 15,000 abdominal CTs paired with EHR data and reports. For zero-shot cross-modal retrieval and classification, comparisons were limited to the VLMs CT-CLIP and Merlin evaluated in inference-only mode, as these tasks require pre-trained, paired image and text encoders. The vision-only 3D CNN and CT-FM models were not applicable for these evaluations. For the image-to-text report generation task, we selected four established models evaluated in inference-only mode to serve as robust baselines. RadFM43 is a VLM specifically designed for medical report generation, pre-trained on a large-scale multi-modal medical dataset of 16 million medical scans with paired text. CT-CHAT36 is another specialized VLM that combines the CT-CLIP vision encoder with a Llama 3.1 8B model, fine-tuned on over 2.7 million question-answer pairs from chest CTs. Additionally, we included two leading 2D VLMs: OpenAI’s GPT-4o41, a general-purpose multi-modal model, and Google’s MedGemma42, a healthcare-specialized model. For these 2D models, the center axial slice of each 3D CT volume was used as the image input. For these baseline models, official pre-trained weights were used directly for inference. In contrast, since RenalCLIP is natively a contrastive model lacking text generation capability, we integrated it with a multi-modal projector and a BioMistral-7B LLM, which were then trained (via projector alignment and LoRA fine-tuning) on the training set (see Methods, Radiology report generation, for details). Human expert benchmarking R.E.N.A.L. nephrometry scoring To benchmark the model against human inter-observer variability, we conducted a blinded multi-reader study. An expert panel comprising six board-certified clinicians (three urologists and three radiologists) independently evaluated a randomly selected subset of 50 cases from the internal test cohort. Raters were strictly blinded to both the original consensus ground truth and the model’s predictions. Clinical evaluation of generated reports To assess the clinical validity of the generated text, we conducted an independent evaluation on the full internal test cohort. Three board-certified radiologists evaluated the reports. To prevent assessment bias, the reports (generated by models alongside the human ground truth) were presented in a randomized order for each case, with raters strictly blinded to the source. Evaluation metrics For the renal mass characterization task (i.e., the five R.E.N.A.L. score components), which are multi-class (three-class) classification tasks, we primarily recorded the Area Under the Receiver Operating Characteristic curve (ROC AUC). For this, we calculated the macro-averaged ROC AUC by treating each class as a positive against the others and then averaging the results. For the human benchmarking study, inter-rater agreement against the consensus ground truth was quantified using accuracy and Cohen’s kappa. The relative rank of each rater was determined by their respective kappa value to contextualize the model’s performance within the spectrum of human variability. For the binary-class diagnosis tasks (malignancy and aggressiveness), a more comprehensive set of metrics was used: ROC AUC, Precision-Recall Area Under the Curve (PRAUC), sensitivity, specificity, and the F1-score. For these threshold-dependent metrics, the optimal classification threshold for each cohort was determined by maximizing Youden’s J statistic. For prognosis tasks, we used Harrell’s concordance index (C-index)74 to evaluate how well a model’s predicted risk aligned with the actual order of events. We also employed time-dependent ROC AUC, C-index, and the Brier score75 to assess predictive accuracy at various follow-up times. The Brier score specifically measures the accuracy of probabilistic predictions, with lower scores indicating better calibration. For zero-shot and retrieval tasks, we used ROC AUC for zero-shot classification and Recall@K (with K = 1, 3, 5) for retrieval, which measures whether a correct item is found among the top K retrieved results. For the report generation task, we reported BLEU76, METEOR77, and ROUGE-L78 scores to evaluate different aspects of generation quality. Specifically, we calculated BLEU-1/-2/-4 to measure the precision of n-gram overlap against human-written references; METEOR, which incorporates semantic matching (e.g., synonyms) to better capture meaning; and ROUGE-L, which measures the longest common subsequence to reflect sentence-level structure and fluency. For the expert qualitative evaluation, reports were scored using a 6-point clinical accuracy scale adapted from HistoGPT44: (5) beyond expectation; (4) highly accurate; (3) generally accurate with minor variations lacking clinical impact; (2) partially accurate with variations of potential clinical impact; (1) minimally accurate; and (0) completely inaccurate. As defined by this protocol, a score > 2 indicates a diagnosis that is considered correct or within an acceptable range of subjectivity. The final score for each case was calculated as the average of the ratings from the three experts and inter-rater reliability was assessed using Fleiss’ kappa. Fine-tuning for clinical tasks To evaluate RenalCLIP’s utility in key renal cancer clinical applications, we fine-tuned the model on various downstream tasks related to RCC characterization, diagnosis, and prognosis. These experiments were conducted using our internal downstream dataset, comprising 400 training samples and 100 evaluation samples. Model performance was subsequently tested on independent external and TCIA cohorts. The AdamW optimizer was consistently employed, and each model was trained for 100 epochs on a single A100 80GB GPU. Optimal hyperparameters (learning rate and batch size) for each model and task were determined through a comprehensive grid search. The best-performing model for each comparison was identified based on its performance on the internal test set: macro-averaged AUC for characterization and diagnosis tasks (specifically macro-AUC for multi-class classification), and C-index for prognosis tasks. These hyperparameter sets were then used to evaluate the models on the external and TCIA cohorts. Due to variations in GPU memory footprint and model architecture, the hyperparameter search ranges differed across models. For 3D ResNet18 (randomly initialized) and RenalCLIP’s backbone (35,087,424 parameters), batch sizes were searched from [50, 100, 150], and learning rates from [5 × 10−4, 1 × 10−4, 5 × 10−5, 1 × 10−5]. CT-CLIP (176,885,144 parameters) used a fixed batch size of 8, with its learning rate mirroring that of Merlin due to their comparable parameter counts. Merlin (121,882,204 parameters) used a fixed batch size of 16, and its learning rates were searched from [1 × 10−4, 5 × 10−5, 1 × 10−5]. For CT-FM (77,760,992 parameters), batch sizes were searched from [50, 100], and learning rates from [5 × 10−4, 1 × 10−4, 5 × 10−5, 1 × 10−5]. For characterization and diagnosis tasks, features extracted by the model’s backbone were passed through a Multi-Layer Perceptron (MLP) classification head to map them to the class dimension. These models were trained using a cross-entropy loss function. For prognosis tasks, the same backbone architecture was used, but the extracted features were mapped to a single continuous risk score (representing the log-hazard ratio, or linear predictor) via an MLP. Prognostic models were trained using the Cox Proportional Hazards (CoxPH) model79, leveraging a partial log-likelihood loss function to account for censored data. While our dataset included three prognostic labels (recurrence-free survival (RFS), defined as the interval from surgery to local recurrence, distant metastasis, or renal cancer-related death; disease-specific survival (DSS), the interval from surgery to death from renal cancer; and overall survival (OS), the interval from surgery to death from any cause), we exclusively utilized the RFS label for model training. This decision was driven by the fact that RFS events directly reflect intrinsic tumor aggressiveness, whereas OS is confounded by non-cancer competing risks and DSS events are limited by effective systemic therapies. The trained RFS model was then used to calculate risk scores across all three survival endpoints (RFS, DSS, and OS) for quantitative evaluation. Consequently, the predicted risk scores are identical for a given patient across these tasks. For risk stratification, patients were categorized into high- and low-risk groups based on the cohort-specific median risk score (see Supplementary Table 25 for specific thresholds). Radiology report generation We developed a caption generation model, adapted from the LLaVA framework80,81, designed to interpret 3D renal CT volumes. This model integrates the frozen RenalCLIP image backbone fbackbone⋅, a multi-modal projector, and an LLM. The image backbone processes 3D renal CT volumes, compressing them into a feature space. These features are then passed to our multi-modal projector, which is implemented as a simple linear layer. This layer linearly transforms the image features, converting them from their original dimension (e.g., 512) into a dimension compatible with the LLM’s hidden size (e.g., 4096). This effectively functions as a visual tokenizer, aligning the image features with the LLM’s word embedding space. For the LLM component, we employed BioMistral-7B82, a version of Mistral-7B further pre-trained on medical textual data from PubMed Central Open Access, making it well-suited for the medical domain; its architecture includes 32 transformer layers, 32 attention heads, an embedding dimension of 4096, a hidden dimension of 14,336, and a vocabulary size of 32,000. The captioning model undergoes a two-step training strategy. In both steps, the model is guided by a consistent instruction prompt structured as follows: [INST] Generate a radiology report for the given CT scan of suspected renal cancer. Use the following format: Findings: [Detailed observations based on the CT scan.] Impression: [Summary of the diagnostic conclusions.] [/INST] First, in the initial alignment step, only the multi-modal projector is trained while both the image backbone and the LLM remain frozen, utilizing the train set of our internal pre-training dataset. This step is crucial for aligning the multi-modal projector in biomedical concept space and enhancing subsequent fine-tuning performance. Second, in the fine-tuning step, training continues for the multi-modal projector (image backbone remains frozen), but the LLM is fine-tuned using LoRA67, utilizing the internal training cohort. All training for the captioning model, across both steps, was conducted on a single A100-80GB GPU with a batch size of 30. In the initial alignment step, the projector was trained for 20 epochs using a learning rate of 1 × 10−3. After each epoch, performance was validated on the validation set of the pre-training dataset, and the checkpoint with the lowest validation loss was saved. For the subsequent fine-tuning step, the projection layer’s learning rate was set to 5 × 10−4, while the LoRA weights for the LLM were fine-tuned with a learning rate of 1 × 10−6 over 2 epochs. More hyperparameters for both pre-training and fine-tuning in radiology report generation are detailed in Supplementary Tables 51–52. Zero-shot classification For zero-shot classification, we employed a method that adapts CLIP’s original framework34. For each class, we generated a text prompt by combining a class prompt (e.g., benign for malignancy assessment) with a pre-defined sentence template. For example, a template might be “A soft tissue mass that is {…} is visible in the kidney.” In total, 20 distinct templates were used and each class was associated with five different class prompts, resulting in 100 unique text prompts generated for every class. A comprehensive list of all class names and templates used is provided in Supplementary Tables 33–35. To rigorously evaluate model performance while accounting for the variability arising from this diverse prompt set, we implemented and compared two distinct zero-shot evaluation strategies. First, to determine a robust, high-performance metric, we employed a maximum similarity ensemble strategy. For each image embedding ui, we computed its cosine similarity against the entire set of 100 pre-computed text embeddings for each class c∈{0,1}, denoted as {vc,k}k=1100, where k indexes the unique prompts for a given class. The maximum similarity score within each class set was then taken as the definitive logit for that class, i.e., sc=maxksimui,vc,k. An image was assigned to the class with the highest logit score. Second, to specifically quantify the model’s sensitivity to prompt selection, we employed a stochastic prompt sampling strategy. In this analysis, we performed B bootstrap iterations (e.g., B = 1000). For each iteration, we randomly sampled a single prompt embedding for the positive class and one for the negative class to form a classifier pair. This pair was then used to evaluate the model’s ROC AUC across the entire test cohort. This procedure yielded a distribution of performance scores, from which we report the mean and 95% CI to visualize the model’s sensitivity. Cross-modal retrieval For zero-shot retrieval, we employed a method analogous to our zero-shot classification approach, operating within the same aligned latent space. For text-to-image retrieval, a specific text query (e.g., a radiology report) was used to find the top-K closest CT volumes. Conversely, image-to-text retrieval was performed by querying with a CT volume to find the nearest text entries. Ablation studies Impact of uni-modal pre-training components We next sought to dissect the contributions of the uni-modal pre-training stages that precede the final cross-modality alignment in RenalCLIP. We conducted an ablation study comparing the RenalCLIP (Full) model against four variants, all trained using the same CLIP contrastive learning framework (except the random baseline): (1) a model using the CLIP framework initialized with our domain-specific image encoder only (termed RenalCLIP (Image Pre-train)); (2) a model using the CLIP framework initialized with our domain-specific text encoder only (termed RenalCLIP (Text Pre-train)); and (3) a baseline model using the standard CLIP framework initialized with generic weights (BioClinicalBERT83 text encoder and random image encoder, termed CLIP Baseline), and (4) a 3D ResNet-18 initialized with random weights (termed CNN (Rand Init)). The comprehensive comparison across cross-modality retrieval, zero-shot classification, and fine-tuning tasks is presented in Supplementary Fig. 9. We found that for cross-modality retrieval, RenalCLIP (Full) achieved the best performance, followed by RenalCLIP (Text Pre-train) and RenalCLIP (Image Pre-train), which demonstrated comparable results with cohort-specific advantages, and finally the CLIP Baseline. For fine-tuning tasks, which are primarily driven by visual features, a different trend was observed. The RenalCLIP (Full) and RenalCLIP (Image Pre-train) performed comparably and were superior to the other models. Crucially, the CLIP Baseline achieved performance comparable to or only marginally better than the CNN (Rand Init). This demonstrates that generic pre-training pipelines are insufficient without domain-specific adaptation, whereas our disease-centric image pre-training provides the decisive performance gain for visual tasks. The zero-shot classification results presented a more nuanced picture that was dependent on the evaluation strategy. Using the deterministic maximum similarity ensemble strategy, RenalCLIP (Full) again achieved the highest performance, followed by RenalCLIP (Image Pre-train). Conversely, when evaluated using the stochastic prompt sampling strategy to assess prompt sensitivity, the RenalCLIP (Image Pre-train) slightly outperformed the full RenalCLIP (Full). This finding suggests that while our domain-specific text pre-training is highly effective for the deterministic ensemble approach, it may also introduce an increased sensitivity to prompt phrasing variations, a phenomenon that warrants further investigation. Overall, these findings establish that domain-specific image pre-training provides the most substantial and consistent performance foundation across all tasks. Our renal-specific text pre-training offers a crucial synergistic benefit, particularly for fine-grained retrieval and robust zero-shot classification. Impact of multi-phase CT inputs To investigate the marginal benefit of integrating multi-phase data, we conducted an ablation study evaluating four cumulative input settings: single-phase (A), dual-phase (A and V, termed AV), tri-phase (N, A, and V, termed NAV), and the full quad-phase (N, A, V, and D, termed NAVD) (Supplementary Fig. 10c–e). Our results reveal a nuanced trade-off. While multi-phase fusion strategies generally yielded a modest performance gain over the single-phase input across R.E.N.A.L scoring, diagnosis, and survival prognosis, this single-phase model itself established a remarkably strong baseline. Consequently, we observed no universally optimal phase combination; the best-performing configuration varied depending on the specific downstream task and patient cohort. For instance, while the quad-phase input occasionally yielded the highest performance, the dual-phase (AV) or tri-phase (NAV) inputs proved more effective in other settings. We hypothesize that this complex relationship stems from a trade-off between accumulating diagnostic signal and introducing potential noise, a dynamic amplified by our simple logit-averaging fusion method. Although additional phases offer supplementary information (e.g., washout patterns), they can also introduce confounding features that dilute the primary signal, particularly when imaging characteristics vary across cohorts. These findings suggest that while multi-phase data holds clear value, the optimal strategy is not a simple aggregation of all available information. Instead, future work on more sophisticated, learnable fusion mechanisms is warranted, and for current applications, the selection of CT phases should be tailored to the specific clinical question. Statistical analysis To assess statistical uncertainty, 95% confidence intervals (CIs) for all reported performance metrics were estimated using a non-parametric bootstrap method with 1000 resamples, unless otherwise specified. Specifically, for experiments evaluating model performance with varying proportions of training data, results are presented as mean ± standard deviation (SD) derived from n = 5 independent runs with different random seeds. A two-sided P-value of less than 0.05 was considered statistically significant for all hypothesis tests. To evaluate the statistical significance of observed differences in model performance across diagnostic and scoring tasks (including malignancy, aggressiveness, and R.E.N.A.L. score predictions), a two-sided paired permutation test with 1000 permutations was employed. In each iteration, patient-level predictions from the two compared models were randomly swapped to generate a null distribution of performance differences. The P-value was calculated as the proportion of permutations where the absolute simulated difference exceeded the observed absolute difference. The analyses focused on three prognostic endpoints: RFS, DSS, and OS. For all survival analyses, the cohort was restricted to patients with malignant tumors. For risk stratification, each model’s continuous risk score was first dichotomized into high- and low-risk groups based on the cohort-specific median. Kaplan-Meier curves were plotted to visualize survival differences, which were assessed for significance using the log-rank test. These univariate survival analyses were performed using the lifelines library (version 0.30.0) in Python. To determine the independent prognostic value of each derived risk score, a multivariate Cox proportional hazards regression model was constructed using SPSS Statistics 21.0. The significance of each variable in the model was assessed using the Wald test. In this multivariate analysis, each model’s risk score was included as a continuous variable, and the model was adjusted for established clinical and pathological risk factors such as TNM stage and WHO/ISUP grade. Computing hardware and software Scripts for 3D CT pre-processing were written in Python (version 3.10.14) using the libraries Nibabel (version 5.3.1), Numpy (version 1.26.4), and Scipy (version 1.13.1). The multi-stage pre-training of RenalCLIP was conducted on NVIDIA A100 80GB GPUs using PyTorch (version 2.3.0) with CUDA 12.1. The initial image encoder pre-training utilized a single GPU, with data augmentation implemented via MONAI (version 1.3.1). The subsequent text encoder pre-training was also performed on a single GPU, leveraging the LLM2Vec (version 0.2.2), Transformers (version 4.44.2), flash-attn (version 2.7.2), and PEFT (version 0.10.0) libraries. The final cross-modal pre-training was performed on four GPUs configured for multi-GPU, single-node training using PyTorch’s Distributed Data Parallel (DDP) framework. All downstream task fine-tuning experiments were conducted on a single 80GB NVIDIA A100 GPU. The software environment was consistent with pre-training (Python 3.10.14, PyTorch 2.3.0, CUDA 12.1, MONAI 1.3.1). Specific tasks required additional libraries: PyCox (version 0.3.0) was used for prognosis tasks, while PEFT (version 0.10.0) and Transformers (version 4.40.2) were used for report generation. Performance metrics were calculated using several Python libraries. Standard classification metrics, including ROC AUC, PRAUC, sensitivity, specificity, and the F1-score, were computed using Scikit-learn (version 1.5.0). C-index was calculated using PyCox (version 0.3.0), while the time-dependent versions of the ROC AUC, C-index, and Brier score were calculated using scikit-survival (version 0.24.1). Natural language generation metrics were evaluated using NLTK (version 3.9.1) for BLEU and METEOR, and rouge-score (version 0.1.2) for ROUGE. Implementation of the comparative baseline models was based on the official configurations provided in their respective repositories: CT-CLIP, Merlin (https://github.com/StanfordMIMI/Merlin), CT-FM, RadFM (https://github.com/chaoyi-wu/RadFM), CT-CHAT, GPT-4o (https://ai.azure.com/catalog/models/gpt-4o), and MedGemma (https://huggingface.co/google/medgemma-4b-it). Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Supplementary information Supplementary Information Peer Review File Reporting Summary Source data Source Data Acknowledgements We thank pathologists Q. Sun (Fudan University) and H. Lin (the First People’s Hospital of Lianyungang) for re-evaluating pathologic slides. We thank radiologists X. Li (Fudan University) and L. Yao (Zhejiang University) for assessing R.E.N.A.L. score. The computations in this research were performed using the CFFF platform of Fudan University. This work was conducted in part during Y.T.’s internship at Microsoft Research Asia. Author contributions Y. T., Z. Z., Z. W., X. L. and F. C. contributed to data acquisition, analysis, interpretation, statistical analysis, and manuscript drafting. K. W., C. W., X. Z., S. Z., J. Y., X. Jin, X. Jiang, Y. Y., D. L. and L. Q. provided technical and material support and participated in statistical analyses. Y. T., K. W., Z. W., X. L. and S. W. were responsible for algorithm development and statistical analyses. Z. S., J. G., N. Y., Y. X. and S. W. designed the study, interpreted data, drafted and revised the manuscript, secured funding, and supervised the study. All authors critically reviewed the manuscript, approved the final version, and agree to be accountable for all aspects of the work. Peer review Peer review information Nature Communications thanks Carsten Marr, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available. Funding S.W. discloses support for the research of this work from the National Key Research and Development Program of China [2024YFF1207500] and the International Science and Technology Cooperation Program under the 2023 Shanghai Action Plan for Science [23410710400]. Y.X. discloses support for the research of this work from the Excellent Young Scientists Fund of Fujian Provincial Natural Science Foundation [2026D016]. J.G. discloses support for the research of this work from the National Natural Science Foundation of China [81974393]. J.Y. discloses support for the research of this work from the Hexi University President Fund Innovation Team Project [CXTD2022012]. The funders had no role in the study design, data collection, data analysis, data interpretation or writing of the manuscript. Data availability The public TCIA data used in this study are available in the Cancer Imaging Archive [https://www.cancerimagingarchive.net/collections/] and the GDC Data Portal [https://portal.gdc.cancer.gov/]. The multi-center imaging datasets analyzed in this study are available under restricted access due to patient privacy laws and institutional review board policies. Access can be obtained by submitting a formal request to the corresponding author, S.W. (shuowang10@fudan.edu.cn). Access to the data is restricted exclusively to researchers for non-commercial academic purposes. Requests will be reviewed and forwarded to each respective center for consent, with an expected response timeframe of within 4 weeks. Once approved, access to the requested data will be granted for a period of 1 year. Source data are provided with this paper. Code availability The source code for all experiments, pre-trained model weights for RenalCLIP, and tutorial Jupyter notebooks for key applications are publicly available on GitHub to support reproducibility and further research (https://github.com/dt-yuhui/RenalCLIP)84. The complete codebase and models are released under the MIT License. For reproducibility, we archived the exact version used in this study on Zenodo with the [10.5281/zenodo.19565967] and tagged release [v1.0.0]. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. These authors contributed equally: Yuhui Tao, Zhongwei Zhao, Zilong Wang, Xufang Luo, Feng Chen. Contributor Information Zhiqiang Shao, Email: shaozq2005@163.com. Jianming Guo, Email: guo.jianming@zs-hospital.sh.cn. Nengwang Yu, Email: qiluyunengwang@hotmail.com. Ying Xiong, Email: xiong.ying@zs-hospital.sh.cn. Shuo Wang, Email: shuowang10@fudan.edu.cn. Supplementary information The online version contains supplementary material available at 10.1038/s41467-026-74175-w. References Associated Data Supplementary Materials Supplementary Information Peer Review File Reporting Summary Source Data Data Availability Statement The public TCIA data used in this study are available in the Cancer Imaging Archive [https://www.cancerimagingarchive.net/collections/] and the GDC Data Portal [https://portal.gdc.cancer.gov/]. The multi-center imaging datasets analyzed in this study are available under restricted access due to patient privacy laws and institutional review board policies. Access can be obtained by submitting a formal request to the corresponding author, S.W. (shuowang10@fudan.edu.cn). Access to the data is restricted exclusively to researchers for non-commercial academic purposes. Requests will be reviewed and forwarded to each respective center for consent, with an expected response timeframe of within 4 weeks. Once approved, access to the requested data will be granted for a period of 1 year. Source data are provided with this paper. The source code for all experiments, pre-trained model weights for RenalCLIP, and tutorial Jupyter notebooks for key applications are publicly available on GitHub to support reproducibility and further research (https://github.com/dt-yuhui/RenalCLIP)84. The complete codebase and models are released under the MIT License. For reproducibility, we archived the exact version used in this study on Zenodo with the [10.5281/zenodo.19565967] and tagged release [v1.0.0].