A clustering-stratified cross-validation framework for validating omics survival models: application to head and neck cancer Authors: Antoine Dubray-Vautrin, Olivier Choussy, Constance Lamy, Grégoire Marret, Joey Martin, Jerzy Klijanienko, Sophie Vacher, Ladidi Ahmanache, Ivan Bieche, Célia Dupain, Christophe Le Tourneau, Jimmy Mullaert Published: 2025 ABSTRACT AbstractBackgroundThis study tackles the challenge of developing reliable prognostic models for time-to-event (TTE) outcomes using high-dimensional omics data in head and neck cancers. Resampling methods, particularly nested cross-validation, are considered as standard for model hyperparameter selection and performance evaluation. When handling clustered data, balancing the random partition of the cross-validation folds to minimize optimism bias and instability could be tested. This work compares the performance of three nested cross-validation implementations, including random assignment of the folds, clustering-based resampling, and internal-external validation using an hold out approach.MethodWe analyzed two head and neck squamous cell carcinoma (HNSCC) cohorts: The Cancer Genome Atlas (TCGA) and SCANDARE (NCT03017573), with clinical data and transcriptomic data normalized as log-transcripts per million. Three model selection methods LASSO, IPF-Lasso, and Priority-LASSO were evaluated within five nested cross-validation frameworks: Standard nested cross-validation, Clustering-based nested-cross validation, nested-cross validation with Combat correction, Nested cross-validation for optimization combined with hold-out for validation, Nested cross-validation for optimization combined with hold-out and ComBat correction for validation. Predictive performance was assessed using 3-year AUC and Integrated Brier Score (IBS).ResultsWe analyzed data from 581 patients (mean age 61.0 years, 33.6% female) across TCGA-HNSC (n = 505) and SCANDARE (n = 76). Clustering analyses, using UMAP and k-means, identified three transcriptomic clusters. Validation strategies demonstrated reduced instability for Lasso (p < 0.001), IPF-Lasso (p < 0.001) and Priority-lasso (p < 0.001) without apparent optimism in discrimination and calibration metrics with stratified nested cross-validation (SNCV), supporting its utility. As an application using IPF-Lasso Cox models with SNCV, we integrated clinical and transcriptomic data, selecting 35 prognosis variables of head and neck carcinomas. This model achieved a 3-year AUC of 0.71 and IBS of 0.08.ConclusionClustering-based nested cross-validation combined with stratified cross-validation offers a robust compromise for developing high-dimensional survival models and evaluating their predictive performance. This approach leverages clustering-derived stratification to balance heterogeneity in the dataset within cross-validation folds, although the training and test sets remain derived from the pooled dataset rather than fully independent cohorts.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12874-025-02709-9.Keywords: Cross-validation, Machine learning, Cox penalization, Survival, Internal validation, ROC Introduction Prognosis models using time-to-event (TTE) are particularly developed in medical research and epidemiology for prediction [1]. These models are essential for analyzing survival data and estimating the risk of specific events occurring over time. However, the increasing availability of high-dimensional data as genomic, transcriptomic, DNA methylation, proteomic or metabolomics sets presents both opportunities and challenges for developing robust and generalizable time-to-event models for personalized predictions and therapeutic [2]. Integrating omics data into a coherent model is still challenging because of the multicollinearity between the layers, the heterogeneity of the methods between the centers [3] when disposing of different sources of data. There are various approaches in the literature for omics data integration, encompassing supervised, semi-supervised, and unsupervised integrative analyses. These methods often employ advanced statistical techniques such as penalization approaches (e.g., LASSO, elastic net, hierarchical regression [4–6]) to handle the high dimensionality and complex relationships within omics datasets [7]. External validation is hindered by the limited availability of independent datasets that include all relevant omics layers. Addressing these challenges requires the development of advanced integration methods, standardization efforts and omics-specific validation strategies. In the context of model validation, internal and external validation strategies have to assess model performance across different settings and populations. The internal validation evaluates model performance within the development data set, and external validation have to assess generalizability to new populations [8]. Internal-external validation strategy are described in the literature validating the model in a distinct dataset [9]. The cross-validation approach for assessing predictive performance in high-dimensional settings [10], can be affected by varying levels of correlation when integrating multi-platform data, especially if the data are clustered. The challenges posed by both intra-platform and inter-platform correlations in multi-omic datasets can be addressed through model selection using sparse methods such as Lasso, and/or hierarchical integration approaches like Priority Lasso or IPF Lasso. To account for clustered data structures, stratification of cross-validation folds is a promising strategy, particularly valuable when handling time-to-event data in high-dimensional contexts. The problematic of source or cluster correlation between the samples could be deal by modifying the structure of the random cross-validation adding a clustering stratification of the folds. The Stratified cross-validation for time-to-event models addresses several key challenges in model development and validation [11]. It helps maintain the balance of event rates across folds, ensures proper handling of censored observations, and accounts for potential temporal dependencies in the data. Stratification in the cross-validation process ensures that each fold maintains a balanced representation of the internal and external datasets, reflecting their proportions within the entire dataset. This approach is particularly relevant in high-dimensional settings, where the risk of overfitting is heightened, and the need for robust validation strategies is paramount when the effect of the center have not to be measured in the statistical process. The impact of clustering based-stratification of the nested cross-validation is not described in the literature and need further consideration to improve the assessment of omics models to avoid overfitting [12]. The objective the study is to assess the performance of stratification on cross validation strategy for omics model assessment using penalized regression with application on head and neck squamous cell carcinomas (HNSCC). Methods Implementation and analysis All statistical analyses and figure generation were performed using R software (version 4.4.2; R Core Team, 2024, Austria). Data manipulation, survival analysis, and visualization were conducted within the R environment to ensure reproducibility and transparency. We summarized the process of the study on Fig. 1. Flow chart of the study design with comparison between internal validation strategies Data acquisition and processing The data sources of the study were obtained from the samples from The Cancer Genome Atlas (TCGA) (https://tcga-data.nci.nih.gov/tcga/, accessed 12 october 2024) and SCANDARE head and neck cohort (NCT03017573||https://www.clinicaltrials.gov/, registration date: January 11, 2017). From the TCGA cohort We downloaded samples from bulk tumor tissue with the following data: RNAseq and count data expression in FPKM normalization, clinical characteristics, and survival data. The FPKM value from TCGA was converted to transcripts per kilobase and per million reads (TPM) and combined as a data matrix. After merging the matrix of clinical data, Count data, we included a total of n = 505 complete samples in the analysis. We represented in the statistical analysis the count data in TPM excluding transcripts below 1 TPM for > = 50% of the samples and we used a logarithmic transformation of TPM with continuous values in the models. From the SCANDARE cohort We included a total of n = 76 complete samples in the analysis using clinical data, transcriptomic data with TPM normalization. As processed for the TCGA HNSC database we normalized the count data in TPM excluding the transcript variables when < 1 TPM for > = 50% of the samples and we used a logarithmic transformation of TPM with continuous values in the models. Data merging After merging the clinical and transcriptomic datasets, we retained 581 complete cases with available survival data, time to event measured in years, a non-specific event of death, and demographic variables including age (continuous, per year), biological sex (male = 1, female = 2),, tumor staging according to the 7th edition of TNM (stages 1–4) and Sublocation (Oral Cavity, Oropharynx, Pharyngo-larynx, Others). Gene expression values were integrated as log-transformed transcript per million (logTPM), providing a continuous measure of transcript abundance for each case. All samples underwent identical normalization and log-transformation protocols to minimize technical variability. We assessed two approaches to account for batch effects. The first involved a clustering approach without applying count transformation based on the negative binomial distribution. The second combined clustering with data transformation using the ComBat R package to limit batch effects between the two cohorts; both approaches were included in the validation strategies assessed. To further evaluate cohort comparability and control for batch effects, principal component analysis (PCA) was performed on the combined expression matrix. The top four principal components (PCs) were inspected to ensure that the major sources of transcriptomic variation did not simply reflect cohort origin (efigure 1). Outcome In this study, we selected overall survival (OS) as the outcome of interest for our statistical analysis. We defined OS as the time from the start of treatment until death from any cause, which was coded as event 1 when it occurred. Notation Let denote T the survival time and C the censoring time. We defined the observed time and be the censoring indicator. The partial likelihood function of Cox regression is detailed: Where Z𝑖 = (Z𝑖1,…,Z𝑖𝑝) the vector of covariates for patient i and 𝜷=(𝛽1,…,𝛽𝑝) the vector of associated coefficients. >Model selection To select the best model with performance assessment we combined different Cox penalized regressions using three different approaches : (i)The first method was the lasso cox regression using glmnet package with fixed value of α = 1 tuning the value by cross-validation approach [13]. The first method was the lasso cox regression using glmnet package with fixed value of α = 1 tuning the value by cross-validation approach [13]. Where is a tuning parameter of the convexity shape. (ii)The second method was the IPF-lasso [4] using the following estimation of coefficients using IPFlasso R package. The second method was the IPF-lasso [4] using the following estimation of coefficients using IPFlasso R package. Where M (for M = 1,., M) represents the variables corresponding to different data modalities, such as clinical, genomic, transcriptomic and methylation data. The tuning of is tailored to each modality of the omics data with a chosen modality as a reference. We tuned the best model with with LASSO tuning approach of each block of predictors and with a second optimization parameter corresponding to regularization of the modality of omics data. The penalty factor of the M = 1 is normalized to 1 and the penalty factors for other modalities are defined relatively to the reference. (iii)The third method was a Priority-lasso regression [5] using Priority-lasso R package with a tuned by block of modalities. Priority Lasso fits Lasso models sequentially, block by block, respecting a priority order respecting a three steps approach: The third method was a Priority-lasso regression [5] using Priority-lasso R package with a tuned by block of modalities. Priority Lasso fits Lasso models sequentially, block by block, respecting a priority order respecting a three steps approach: A lasso cox regression only on the highest priority block of variables where are estimated by minimizing the penalized loss function. A lasso cox regression only on the highest priority block of variables where are estimated by minimizing the penalized loss function. A Lasso regression on the second priority block , using the linear predictor from Step 1 as an offset (fixed component). A Lasso regression on the second priority block , using the linear predictor from Step 1 as an offset (fixed component). The procedure is repeated for each remaining block with the final modeling represented the sum of the contributions from all blocks: The procedure is repeated for each remaining block with the final modeling represented the sum of the contributions from all blocks: Where is the linear predictor after each block modality M optimization and π the priority order of the blocks and the value for included variables j of each block. We defined two blocks (clinical covariable and transcriptomic covariable). Clustering methods We determined clusters of samples using dimensional reduction approaches of matrix of predictors merging the TCGA and the SCANDARE cohorts with two crossmatch approaches: (i) UMAP (Uniform Manifold Approximation and Projection) [14] and (ii) K-means clustering [15] approaches. We determined the K-clustering trade-off using Elbow method [16] and gap statistic method [17] of factoextra R package (efigure 2). We fixed K = 3 after K-optimization to ensure a sufficient event rate within each fold. The interaction variable “cluster”/“outcome” was used to balance the folds through stratification into six levels. Validation strategies We proponed as an internal validation strategy a cross-validation approach [18] splitting the available sample into K subsamples of equal size. K − 1 subsamples are used as the training sample, while the remaining subsample is used as the assessment of the metrics. We introduced the nested optimization with a second partition of the sample in J subsamples of equal size in the inner loop to tune the hyper parameters of the evaluated models. To evaluate the discriminative and calibration performance of the best model we performed a J nested k-folds-cross validation approach with and without stratification with the fixed parameters K and J: K outer loop segmentation with K = 5 K outer loop segmentation with K = 5 J inner loop segmentation with J = 5. J inner loop segmentation with J = 5. We tuned the best model with their parameters in the inner loop segmentation of training data corresponding to K-1 partitions of all the data. We calculated the predictive performance of the best model using the test data at each iteration. The final discriminative and calibration performance corresponded to the mean of the performance of the best model over the outer-loop iteration. We considered as stratification factor the interaction variable between the clusters of the samples and the outcome of the study and we applied a J nested k-folds cross validation. We defined five implementations of internal validation approaches: i.A standard nested cross-validation method (NCV) where the outer folds and inner folds were randomly splitted. A standard nested cross-validation method (NCV) where the outer folds and inner folds were randomly splitted. ii.A clustering stratified based nested-cross validation procedure (SNCV). During both outer and inner fold splitting, the dataset is divided into training and testing sets such that each fold contains a proportional representation of the clusters. A clustering stratified based nested-cross validation procedure (SNCV). During both outer and inner fold splitting, the dataset is divided into training and testing sets such that each fold contains a proportional representation of the clusters. iii.A clustering stratified based nested-cross validation procedure with corrected batch effect using combat package (NCV-combat). During both outer and inner fold splitting, the dataset is divided into training and testing sets such that each fold contains a proportional representation of the clusters. A clustering stratified based nested-cross validation procedure with corrected batch effect using combat package (NCV-combat). During both outer and inner fold splitting, the dataset is divided into training and testing sets such that each fold contains a proportional representation of the clusters. iv.An internal-external validation approach using hold-out partition (NCV-HO) with 70% of the dataset to train the data with the nested-cross validation approach and 30% of the dataset with a random split without stratification of the folds to assess the prediction performance of the best model. An internal-external validation approach using hold-out partition (NCV-HO) with 70% of the dataset to train the data with the nested-cross validation approach and 30% of the dataset with a random split without stratification of the folds to assess the prediction performance of the best model. v.An internal-external validation approach using hold-out partition with corrected batch effect using combat R package (NCV-HO-combat) with 70% of the dataset to train the data with the nested-cross validation approach and 30% of the dataset with a random split without stratification of the folds to assess the prediction performance of the best model. An internal-external validation approach using hold-out partition with corrected batch effect using combat R package (NCV-HO-combat) with 70% of the dataset to train the data with the nested-cross validation approach and 30% of the dataset with a random split without stratification of the folds to assess the prediction performance of the best model. We compared the performance and stability of these three implementations of nested-cross validation. We detailed the nested-cross-validation structure on Fig. 2. Representation of stratified-nested cross-validation process of statistical validation model Estimands and performance metrics We measured the rate of non-convergence according to the cox penalized regression and the stratified and non-stratified approaches. We evaluated the quality of model optimization and sparsity by calculating the mean and standard deviation of the selected non-zero coefficients for each selection method (IPF Lasso, PL, and Lasso) under both stratified and non-stratified NCV. Pairwise comparisons were performed using a non-parametric Wilcoxon test, with statistical significance defined as p < 0.05. We determined a discriminative approach of the predictive performance using the Area Under the Curve receiving operating Characteristics (AUC) at a 3 years horizon using timeROC R package [19]. We considered for each dynamic analysis of AUC, a cumulative/dynamic method for calculation of AUC-time dependent [20, 21]. We determined the predictive calibration approach calculating the Brier Score at 3 years and the Integrated Brier Score using pec R package estimated with inverse probability of censoring weights (IPCW) [22, 23] calculating the individual survival probabilities using pec package from time 0 to 3 years for all the tested internal validation methods. We determined AUC and IBS dispersion assessment by using 100 replications of the experiments with different seed. Where corresponded to AUC or IBS estimates obtained by outer folds estimation. We obtained the density of probability of the outer fold AUC distribution using the gaussian function of the ggplot2 R package. To compare the optimism of the performance assessment we used an ANOVA comparison with a p-value < 0.05 considered as statistically significant. If the ANOVA comparison was significant a post hoc pairwise Tukey comparison was conducted. To compare the stability of the performance assessment we performed homogeneity test using Bartlett test for variance comparison between the validation strategies performance according to the model selection, a p-value < 0.05 was considered as statistically significant rejecting the null hypothesis of equal variances. Results Population of the study A total of 596 patients were initially included in the study. However, 15 patients were excluded due to incomplete survival data, leaving 581 patients for final analysis. The mean age of the cohort was 61.0 years (SD ± 11.7). Of the patients included, 33.6% were female. Regarding TNM staging, the distribution was as follows: 5.5% of patients were classified as Stage I, 18.0% as Stage II, 22.0% as Stage III, and 54.5% as Stage IV. We detailed on Table 1 the baseline characteristics of TCGA and SCANDARE cohorts. Table 1Baseline characteristics of the population TCGA-HNSC and SCANDARETCGA-HNSC (n = 505)SCANDARE (n = 76)Age60.9 (11.9)63.4 (11.0)Sex (male)430 (85.2)54 (71.1)Current smokers343 (68)43 (56.6)Sublocation Oral cavity310 (61.3)34 (44.7) Oropharynx81 (16.0)29 (38.2) Hypopharynx/larynx114 (22.7)13 (17.1)TNM staging I20 (3.9)15 (19.8) II97 (19.3)9 (11.8) III106 (20.9)20 (26.3) IV282 (55.8)32 (42.1)Numbers indicate “mean (SD)” for quantitative variable or “N (%)” qualitative variables Baseline characteristics of the population TCGA-HNSC and SCANDARE Numbers indicate “mean (SD)” for quantitative variable or “N (%)” qualitative variables Clustering analyses We clustered the population using the matrix of transcriptomic data with UMAP and k-means algorithm determining 3 clusters represented on Fig. 3. After merging TCGA HNSC and SCANDARE HNSC, n = 15,811 transcript variables were matched between the two cohorts. We confirmed the high degree of correlation between the transcriptomic data according to the cohort. The cluster n°1 corresponded to the data originating from the SCANDARE data. Umap and K-means clustering of the transcriptomic data. A: After ComBat transformation of the transcriptomic matrix; B: Without ComBat transformation of the transcriptomic matrix Performance assessment Regarding the issue of non-convergence, no instances of non-convergence were observed with the IPF Lasso, PL, and Lasso methods for both stratified and non-stratified nested cross-validation (NCV) strategies. Table 2 presents the non-zero coefficients selected by the lasso, IPF Lasso, PL methods using stratified, non-stratified NCV and NCV associated to hold-out approaches and incorporing the ComBat correction of batch effect. According to validation strategies the non-zero coefficients were statistically heterogeneous for Lasso models (p < 0.001), IPF lasso (p < 0.01) but not statistically significant PL (p = 0.98). According to validation strategies the non-zero coefficients were statistically heterogeneous for Lasso models (p = 0.01) but not statistically significant for IPF lasso (p = 0.08) and PL (p = 0.99). The ANOVA comparison based on the validation strategy showed statistically significant results for all models (Lasso, IPF-Lasso, and PL) with p-values < 0.001. The mean differences obtained from pairwise comparisons were as follows: for Lasso, SNCV vs. NCV = 1.05 [95% CI −0.72/2.82, padj = 0.34], HO vs. NCV = −3.82 [95% CI −5.58/−2.05, padj < 0.001], and HO vs. SNCV = −4.87 [95% CI −6.63/−3.10, padj < 0.001]; for IPF-Lasso, SNCV vs. NCV = 0.92 [95% CI −1.00/2.83, padj = 0.50], HO vs. NCV = −4.73 [95% CI −6.64/−2.81, padj < 0.001], and HO vs. SNCV = −5.64 [95% CI −7.56/−3.73, padj < 0.001]; and for PL, SNCV vs. NCV = 0.96 [95% CI −0.56/2.48, padj = 0.30], HO vs. NCV = −4.01 [95% CI −5.53/−2.49, padj < 0.001], and HO vs. SNCV = −4.97 [95% CI −6.49/−3.45, padj < 0.001]. For Lasso, IPF-Lasso, and PL, the model selection using NCV or SNCV provided greater complexity in model development with better stability. Table 2Representation of selection characteristics according to model and validationModelValidationNon zero coefficientsaANOVAVariance homogeneity **LassoNCV32.9 (11.1)< 0.001< 0.001SNCV34.0 (11.8)NCV-combat32.9 (10.8)NCV-HO29.1 (12.8)NCV-HO combat29.1 (12.8)IPF lassoNCV39.8 (12.4)< 0.001< 0.001SNCV40.7 (12.6)NCV-combat37.6 (10.5)NCV-HO35.1 (13.7)NCV-HO combat33.3 (12.3)Priority lassoNCV33.6 (10.2)< 0.0010.98SNCV34.6 (10.3)NCV-combat33.7 (10.2)NCV-HO29.6 (10.3)NCV-HO combat30.0 (10.5)aNon zero-coefficients according the outer folds and replications represented by “Mean (SD)”**p-value of homogeneity test calculated using Bartlett test of homogeneity of variances Representation of selection characteristics according to model and validation aNon zero-coefficients according the outer folds and replications represented by “Mean (SD)” **p-value of homogeneity test calculated using Bartlett test of homogeneity of variances We summarized on Table 3 the discriminative and calibration assessment of best models developed into the experiments according to model selection methods, validation strategy, outer folds performances and replication performances. Table 3Discriminative and calibration assessment in outer folds and replication according to model and validationModelValidationDiscriminationCalibrationAUC*ANOVAHomogeneity**IBS*ANOVAHomogeneity**LassoNCV0.69 (0.015)p=0.25p<0.0010.09 (4.4x10-4)0.99p<0.001 SNCV0.70 (0.014)0.09 (3.2 x10-4)NCV-combat0.70 (0.014)0.09 (2.9x10-4)NCV-HO0.69 (0.039)0.09 (1.1x10-2)NCV-HO-combat0.69 (0.039)0.09 (1.1x10-2)IPF-lassoNCV0.70 (0.016)p=0.11p<0.0010.09 (4.3x10-4)p=0.99p<0.001SNCV0.70 (0.015)0.09 (3.3x10-4)NCV-combat0.70 (0.013)0.09 (2.9x10-4)NCV-HO0.69 (0.039)0.09 (1.1x10-2)NCV-HO-combat0.69 (0.039)0.09 (1.1x10-2)Priority-lassoNCV0.69 (0.015)p=0.16p<0.0010.09 (4.4x10-4)p=0.10p<0.001SNCV0.69 (0.014)0.09 (3.3x10-4)NCV-combat0.69 (0.0.140.09 (2.9x10-4)NCV-HO0.69 (0.040)0.09 (1.1x10-2)NCV-HO-combat0.69 (0.039)0.09 (1.1x10-2)* AUC and IBS are represented with their mean (SD) across the replications** p-value of homogeneity test calculated using Bartlett test of homogeneity of variances Discriminative and calibration assessment in outer folds and replication according to model and validation * AUC and IBS are represented with their mean (SD) across the replications ** p-value of homogeneity test calculated using Bartlett test of homogeneity of variances Discrimination According to the repetitions of the experiment, the mean 3 years AUC of the best models assessed by stratified NCV with their dispersion were represented on Fig. 4. The homogeneity test of the variance between the internal validation methods were statistically significant for Lasso (p < 0.001), IPF-Lasso (p < 0.001) and PL (p < 0.001) in favor of a reduction of the instability of the AUC assessment using NCV or SNCV. The ANOVA comparison of the 3 years AUC between NCV, SNCV, NCV-combat, HO, HO-combat were not statistically different according to Lasso (p = 0.25), IPF-Lasso (p = 0.11) and PL (p = 0.16). Violin plot representing the 3 years AUC according to the statistical model and the internal validation method Calibration According the repetitions of the experiments, the mean 3 years IBS of the best models assessed are represented on Table 3. We represented the calibration approaches of the internal validation strategy across the repetitions on the box plot in Fig. 5. The homogeneity test between the internal validation methods were statistically significant for Lasso (p < 0.001), IPF-Lasso (p < 0.001) and PL (p < 0.001) in favor of a reduction of the IBS assessment using NCV or SNCV. The ANOVA comparison of the 3 years IBS between NCV, SNCV, NCV-combat, HO and HO-combat were not statistically different according to Lasso (p = 0.99), IPF-Lasso (p = 0.99) and PL (p = 0.10). Violin plot representing the 3 years IBS according to the statistical model and the internal validation method For all the tested model selection, the mean discriminative and calibration performance were not statistically different using NCV, SNCV and HO in favor to absence of introduction of optimism bias in ANOVA comparison. Due to absence of statistical difference in ANOVA testing for both discriminative and calibration performance, the pairwise comparison was not conducted. The Bartlett testing of the variance rejected the null hypothesis of equality of the variance according to the NCV, SNCV and HO in favor of an improvement of the stability of the NCV and SNCV. Application As determined by the first step of analysis, we described an IPF-lasso cox model using TCGA and SCANDARE data determined by using the stratified NCV. Using this method we integrated the clinical variables: Age (numeric code), Sex (binary code), TNM (Stage 1 to 4 as numeric code), Tobacco consumption (binary code) and the transcriptomic matrix using the logTPM normalization detailed in the methods section. The developed IPF-Lasso model using SNCV with their non-coefficients after shrinkage is represented on Table 4 including 35 selected variables. The 3 years AUC of the best model were respectively 0.711 and the 3 years IBS was 0.08. Table 4Selected variables with their non-zero coefficients and tumoral gene expressionVariablesCoefficientsHRGenesTNM0.0102061.010258AGE0.0148641.014975ENSG00000008300−0.093210.910999CELSR3ENSG000000133060.0445461.045553SLC25A39ENSG000000670570.0202351.020441PFKFENSG00000105369−0.005840.994172CD79AENSG00000105523−0.000340.999663FAM83EENSG000001226960.0727861.075501SLC25A51ENSG00000124678−0.030790.969684TCP11ENSG000001250370.0208971.021117EMC3ENSG000001252660.0041191.004127EFNB2ENSG00000127241−0.060670.941133MASP1ENSG00000131055−0.005420.994599COX4I2ENSG00000136059−0.073670.928974VILLENSG000001363270.0144961.014602NKX2-8ENSG000001382070.0044251.004435RBP4ENSG000001415430.0136231.013717EIF4A3ENSG00000143248−0.024460.975839RGS5ENSG000001597200.0040031.004011ATP6V0D1ENSG000001601940.0457031.046764NDUFV3ENSG00000163599−0.007870.992161CTLA4ENSG000001684390.0348641.035479STIP1ENSG000001687850.0297371.030183TSPAN5ENSG000001717630.019171.019355SPATA5L1ENSG00000174652−0.05610.945446ZNF266ENSG00000187824−0.004930.995084TMEM220ENSG00000196503−0.029720.97072ARL9ENSG000001977470.0519891.053364S100A10ENSG00000211897−0.045950.95509IGHG3ENSG000002137620.0263111.02666ZNF134ENSG000002139830.0701891.07271AP1G2ENSG00000214401−0.000860.999137KANSL1ENSG00000223350−0.003050.99695IGLV9-49ENSG000002415530.0019051.001907ARPC4ENSG00000249082−0.004970.995037C5ORF66-AS1 Selected variables with their non-zero coefficients and tumoral gene expression Discussion In this study, we aimed to enhance different implementations of nested cross-validation approach for penalized regression applied to correlated data from multiple sources, with a time-to-event endpoint in a high-dimensional setting. We demonstrated that incorporating clustering-based stratification within the nested cross-validation framework is a feasible and effective strategy. The stability of the tested nested cross-validation implementations appeared similar in this study. These implementations reduce the instability of the validation process compared to hold-out methods and limit optimism bias in both discrimination and calibration metrics. In high-dimensional settings, a compromise must be made between partitioning the data for model development and evaluating predictive performance. Cross-validation methods have been implemented to allow the use of the data both for building the model and for assessing its predictive accuracy. When developing a model using multiple data sources, especially under conditions of limited data availability, an inadequate modeling strategy and performance evaluation approach can lead to unstable and non-reproducible results. The regularization pathway could be a source of instability in model development [24]. Internal validation of high-dimensional models remains a significant challenge and cross-validation for model prediction error assessment reduce bias error estimation [25] and particularly in high dimensional setting using time to event outcome [26, 27]. Assessing the true performance of a developed model in settings with limited sample sizes often leads to overly optimistic estimates and complicates the application of the model to external datasets. When multiple sources of correlated high-dimensional data are available, a critical trade-off arises between using a subset of the data for model development where predictor selection and coefficient estimation are inherently difficult and utilizing the entire dataset for both training and performance evaluation. This compromise requires careful consideration to ensure robust model development and reliable assessment of predictive performance. Various internal and combined internal/external validation approaches have been developed to assess model performance, including nested cross-validation [28] and hold-out validation. In the context of multi-platform integration for time-to-event outcomes, penalized Cox regression methods are particularly appropriate, as they maintain interpretability. Examples include Lasso Cox regression, IPF-Lasso, and Priority Lasso [4, 5, 29]. Additionally, other methods such as elastic net, adaptive regularization, and Bayesian approaches can be integrated to enhance performance and account for the complex correlations among features, which is especially relevant for high-dimensional data types like genomic, transcriptomic, proteomic, and DNA methylation datasets [30]. Bayesian methods offer a principled framework for quantifying uncertainty in parameter estimates through posterior distributions [31]. These features are particularly valuable for omics data, where complex correlations among features are prevalent and hierarchical or network-based Bayesian models can be leveraged to capture these dependencies. However, these methods often entail substantial computational complexity and require careful prior specification and tuning of hyperparameters. Despite these challenges, their ability to simultaneously perform feature selection, model prediction, and missing data handling presents a promising direction for future research in integrative multi-omics analysis. In our current study, we focused on benchmarking established regularized frequentist methods for survival prediction to provide a clear comparative framework, reserving the exploration of Bayesian hierarchical models for subsequent investigations where computational resources and data completeness allow for their effective application.The inner loop optimization of the nested-cross validation could be implemented by different model selection approach using the both of nested-cross validation. The stratification of the cross-validation folds is described to address unbalanced class covariable linked to outcome and optimistic bias in model development and assessment from electronic health records due to data leakage and duplication records [32]. When using genomic or transcriptomic data, batch effects must be taken into account, as they influence the correlation structure inherent to the original data and the variation of the experimental conditions [33, 34]. It could involve transforming the transcriptomic data using appropriate algorithms such as ComBat, or clustering the transcriptomic data matrix to use as a stratification factor for development and validation. Developing a model based on data from a single source of data with a limited sample size, without incorporating information from other sources, poses significant challenges for model validation. Our proposed approach combines nested cross-validation with penalized regression for survival prediction, enabling both model development and performance assessment. This strategy improves the quality of predictor selection and coefficient estimation while reducing instability and mitigating optimism in discrimination and calibration metrics. When disposing different replicates sample of omics platform analysis other methods as the Ratio-based scaling [35] could be performed to limit this batch effect. Additionally, clustering genomic or transcriptomic data or more straightforwardly, labelling samples by their sources/clusters provides a means to incorporate proportional contributions from all data sources during model development and to limit the inconsistency for estimation of coefficients associated to each variables. Using this approach, we selected 35 variables in the prognostic model that demonstrated promising predictive performance in both discrimination and calibration. A recent narrative review identified 34 studies reporting prognostic gene signatures for head and neck cancer survival, primarily derived using Lasso-Cox regression [36]. These studies classified the gene signatures into key cancer-related pathways such as programmed cell death, epigenetic regulation, immune response modulation, metabolism, and resistance to cell death. Consistent with these classifications, our model included genes associated with tumor migration (ARPC4 [37]), epigenetic regulation (RBP4 [38], EIF4A3 [39], ZNF134 [40]), tumor suppressor (C5ORF66-AS1 [41]), tumor immune invasion and immune response modulation (CELSR3 [42], SLC25A39 [43], CD79A [44], MASP1 [45], CTLA4 [46], IGLV9-49 [47]), metabolism regulation (xxx), angiogenesis (EFNB2 [48]), and resistance to cell death and drug resistance (FAM83 [49], ATP6V0D1 [50], STIP1 [51], S100A10 [52]). Some genes as SLC25A51, EMC3 are described as important mitochondrial metabolic regulator with possible links to tumor biology, but their use as a prognostic biomarker in head and neck cancer remains to be confirmed. The COX4I2 (colorectal cancer), TCP11 (cervical cancer) [53], VILL, NKX2-8, RGS5, NDUFV3, TSPAN5, IGHG3 genes have been linked to anti-tumor effects in various cancers, but have not been described in head and neck cancers. Additionally, our statistical approach identified genes (PFKF, SPATA5L1, ZNF266, TMEM22, ARL9, AP1G2, KANSL1) not previously associated with head and neck cancer prognosis, which may represent either correlated pathways or potential false-positive findings. Limits We conducted a benchmark study utilizing three implementations of nested cross-validation, integrated with three penalized regression methods. The study design, however, does not permit the quantification of optimism bias through external validation or simulation-based conditions. Additionally, data clustering in this study was performed using UMAP and K-means approaches. While effective, alternative methods such as principal component analysis or simpler strategies such as incorporating a center variable directly into the modeling could also be considered for further exploration. Conclusion In high-dimensional settings with time-to-event endpoints, incorporating clustering-based stratification within nested cross-validation provides an effective compromise between robust model development and reliable assessment of predictive performance reducing the instability of the process. This approach uses clustering-informed stratified cross-validation as a validation strategy for predictive models and requires testing across diverse patient populations and data settings. Supplementary Information Supplementary Material 1.eFigure 1: Principal Component analysis of the transcriptomic data after ComBat transformation by TCGA (dark blue) and SCANDARE (light blue) cohorts. Supplementary Material 2.eFigure 2: Optimization of tke K-clustering using Elbow method (Panel A) and Gap statistic (Panel B). Acknowledgements We gratefully acknowledge the support of the U1331 laboratory, the STAMP team, and the Institut Curie of Paris and Saint-Cloud, France, for their contributions to this work.We would like to acknowledge the patients, their families, and the multidisciplinary and continuously growing SCANDARE Team: the Department of Drug development and Innovation (Maud Kamal, Maral Halladjian, Linda Larbi Chérif, Charlotte Lecerf), the department of head and neck surgery (Maria Lesnik, Nathalie Badois, Pauline Nicol, Wahib Ghanem, Manuel Erminy, Christian Chappey, Baptiste Sabran, Said Zaer), Department of Theranostic and Diagnostic Medicine (Rémi Goudefroye, Anne-Vincent Salomon, Thierry Bronzini, Sabrina Ibadioune, Laura Courtois, Emmanuelle Jeannot, Philémon Sirven, Yves Allory, Emmanuelle Menet, Magalie Toutain), The CurieCoreTech – ICGex (Sylvain Baulande, Patricia Legoix, Virginie Raynal), U1331 INSERM team (Philippe Hupé, Emmanuel Barrillot, Choumouss Kamoun) and the clinical research direction (Xavier Paoletti, Nesrine Ben Nasr, Anne-Sophie Plissonnier, Katia Ouar, Mehdi Ziane, Atikatou Mama, Tiphaine Batteux, Gwenaëlle Jarry) of the Institut Curie. Authors’ contributions A.D.V. and J.M. prepared the experimental design; A.D.V. and J.M. conducted the data analysis; C.L., C.D., A.D.V. processed the data curation; A.D.V., C.L.T and J.M. wrote the original draft preparation; A.D.V., C.L.T, C.L., J.M., O.C., I.B., J.M.; C.L., G.M., J.K., L.A., C.D., S.V. wrote, reviewed and edited the final manuscript; C.L., C.D., G.M. make ressources. All authors have read and agreed to the published version of the manuscript. All authors are read and approved by the final manuscript. Funding SCANDARE is supported by the grant ANR-10-EQPX-03 and ANR10-INBS-09-08 from the Agence Nationale de la Recherche by the Site de recherche intégré contre le cancer (SiRIC) and by the Cancéropôle Ile-de-France. Data availability Data are available on reasonable request. Declarations Ethics approval and consent to participate The study protocol of SCANDARE was reviewed and approved by the “Comité de protection des personnes Ile de France III” prior to initiation of the study (Approval Number: 2016-A01095-46). Written informed consent was obtained voluntarily from all participants before enrolment in accordance to the Declaration of Helsinki regarding ethics committee approval, informed consent, participant rights, and transparency. Consent for publication NA. Competing interests The authors declare no competing interests. Footnotes Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Citations Supplementary Materials Supplementary Material 1.eFigure 1: Principal Component analysis of the transcriptomic data after ComBat transformation by TCGA (dark blue) and SCANDARE (light blue) cohorts. Supplementary Material 2.eFigure 2: Optimization of tke K-clustering using Elbow method (Panel A) and Gap statistic (Panel B). Data Availability Statement Data are available on reasonable request.