When complexity does not pay: benchmarking deep learning and ensemble methods for biomarker discovery Authors: Cyrille Mesue Njume, Irene Petracci, Sonia Bellini, Katarzyna Goljanek-Whysall, Leo R Quinlan, Agnieszka Fiszer, Barbara Borroni, Roberta Ghidoni, Asli Kumbasar, Ali Cakmak Published: 2026 ABSTRACT AbstractThe integration of multi-omics data holds great promise for identifying robust and clinically relevant biomarkers, yet the increasing complexity of computational methods raises questions about their practical utility. In this study, we present a comprehensive benchmarking framework that evaluates 27 feature selection strategies and 11 predictive models across three real-world disease cohorts: Alzheimer’s disease, progressive supranuclear palsy, and breast cancer. We compare traditional machine learning, ensemble-based methods, and state-of-the-art deep learning models in terms of predictive performance, stability, and biological interpretability. Our results reveal that ensemble feature selection consistently improves robustness and accuracy, particularly for compact biomarker panels. Surprisingly, deep learning models did not outperform simpler classifiers such as logistic regression (L.Regression), support vector machines, or multilayer perceptrons, which often achieved comparable or superior results with lower computational cost and greater interpretability. Triple-omics yielded the highest validation, followed by dual-omics and then single-omics (Triple > Dual > Single). Biological validation against five independent databases confirmed the clinical relevance of the identified biomarkers, including both well-established and novel candidates. To support reproducibility and community adoption, we provide a web-based tool for applying our benchmarking pipeline. Our findings advocate for a pragmatic approach to biomarker discovery—prioritizing methodological transparency, reproducibility, and biological insight over algorithmic complexity.Keywords: biomarker discovery, multi-omics integration, feature selection benchmarking, ensemble rank aggregation, integrative bioinformatics Introduction Biomarker discovery has become a cornerstone of precision medicine, enabling earlier disease detection, a more accurate prognosis, and personalized therapeutic strategies. Early studies primarily relied on single-omics approaches, such as transcriptomics or proteomics, which provided valuable insights but often captured only one layer of biological regulation and yielded limited predictive power. With the rapid expansion of high-throughput technologies, researchers can now generate multi-omics datasets that integrate diverse modalities—including, but not limited to, transcriptomics, epigenomics, proteomics, and metabolomics—thereby capturing complementary regulatory layers and offering a more holistic view of disease processes [1]. Integrating these heterogeneous data modalities holds the promise of identifying more robust biomarkers that not only improve clinical decision-making but also provide mechanistic insight into disease biology [2–5]. However, despite the increasing availability of multi-omics data, extracting clinically meaningful and reproducible biomarker panels remains a major challenge. The core difficulty arises from the intrinsic characteristics of multi-omics data. First, most datasets are high-dimensional yet low-sample , where thousands of molecular features are profiled across only a few hundred patients. This imbalance makes analyses highly susceptible to overfitting and spurious associations—false correlations that arise from random noise, confounding, or chance [6–10]. Second, the heterogeneity of omics layers poses additional challenges: gene expression values are continuous, DNA methylation profiles are sparse, and mutation calls are discrete, each reflecting distinct biological processes. Integrating such modalities requires harmonization across measurement scales and distributions, which is nontrivial and risks introducing bias [11, 12]. Third, the limited availability of clinically annotated cohorts exacerbates instability and reduces reproducibility across independent datasets. These challenges make biomarker discovery highly sensitive to small perturbations in the data, often yielding panels that fail to generalize. Single-method feature selection approaches, such as univariate statistical tests or machine learning-based rankers, are particularly vulnerable in this setting, frequently producing unstable biomarker lists with inconsistent biological interpretation [12–14]. Ensemble ranking has been proposed to mitigate such variability and improve feature stability and reproducibility across high-dimensional omics datasets [15–18]. However, comprehensive benchmarking across multiple diseases, omics modalities, and model types remains unexplored. Most existing applications are largely limited to single-omics analyses such as transcriptomics [19, 20], proteomics [21], or general omics feature spaces [22–24], while only a few have benchmarked individual feature-selection methods on multi-omics datasets [25, 26]. More broadly, there remains a lack of comprehensive benchmarking pipelines that jointly evaluate feature selection, predictive modeling, and multi-omics integration strategies. Moreover, very few studies systematically perform external bioinformatic validation of discovered biomarker panels by comparing predicted genes or pathways against independent biological databases—the Comparative Toxicogenomics Database (CTD), which curates chemical–gene–disease relationships and pathway annotations [27]; the Human microRNA Disease Database (HMDD), a manually curated resource of microRNA (miRNA)–disease associations [28]; GeneCards, an integrative compendium of human genes and their disease/phenotype links [29]; and the EWAS-ATLAS, a catalog of Cytosine–Phosphate–Guanine site (CpG)–trait associations from epigenome-wide association studies [30]. This step is critical for assessing the biological interpretability and clinical relevance of computational findings. While some studies include post hoc validation through overlap or enrichment analyses against known databases or literature [31–33], these are typically limited in scope and do not constitute broad benchmarking across diseases, modalities, and for each method. Moreover, more recently, several deep learning-based multi-omics integration frameworks, such as Multi-Omics hypeRgraph integration nEtwork (MORE) [34] and Multi-Omics Graph cOnvolutional NETworks (MOGONET) [35] have been proposed to capture cross-modality relationships and exploit complex nonlinear patterns. However, while such frameworks have demonstrated strong performance across several datasets and integration settings, their comparative stability, reproducibility, and consistency against simpler or ensemble-based approaches remain insufficiently explored. In addition, external validation of biomarker panels—essential for establishing biological and clinical relevance—is still rarely performed in a structured and multi-level manner. As a result, it remains unclear whether increasing methodological complexity consistently yields more reliable and interpretable biomarker panels, or whether simpler and ensemble-based approaches may offer equal or greater robustness. In this study, we present a comprehensive benchmarking framework for multi-omics biomarker discovery across three real-world disease cohorts. We analyze Alzheimer’s disease (AD) using the Religious Orders Study and Memory and Aging Project (ROSMAP) cohort [36], PSP using the Mayo RNAseq Study (MayoRNASeq) data [37], and breast cancer (BC) using The Cancer Genome Atlas Breast Cancer (TCGA-BRCA) cohort [38]. We systematically evaluate 27 biomarker selection strategies—spanning single rankers and ensemble ranking methods—and 11 predictive models, ranging from traditional machine learning algorithms to advanced integration approaches such as MORE and MOGONET. Beyond computational benchmarking, we validate biomarker panels against four independent biological reference databases (CTD [27], HMDD [28], GeneCards [29], or EWAS-ATLAS [30], and Gene Functional Enrichment Profiling Tool (g:Profiler) [39] for pathway enrichment cross-checked against CTD-pathways), ensuring both interpretability and clinical relevance. Together, this work establishes a robust, reproducible, and generalizable pipeline for multi-omics biomarker discovery and provides new insights into when methodological complexity offers genuine advantages over simpler, more interpretable strategies. Materials and methods This section outlines the experimental design, datasets, and computational framework used in this study. We describe the selection and preprocessing of multi-omics datasets, followed by the feature selection, ensemble ranking, and benchmarking procedures applied to evaluate model performance across integration levels. Study cohorts Experiments were conducted using three real-world, publicly available multi-omics datasets: the ROSMAP (AD) [36], the MayoRNASeq cohort (PSP) [37], and the TCGA-BRCA cohort (BC) accessed via the UCSC Xena Browser [38]. Key study characteristics and data types are summarized in Table 1. Table 1Overview of study characteristics Cohort Disease/Condition Data types Sample size Data source ROSMAPADTranscriptomics, microRNA, methylation, clinical378SynapseTCGA-BRCABCTranscriptomics, methylation, microRNA, clinical1229UCSC XenaMayoRNASeqPSPTranscriptomics, metabolomics, proteomics, clinical437Synapse ROSMAP Dataset. UCSC Xena TCGA-BRCA Dataset. MayoRNASeq Dataset. Note: The sample sizes reported here are based on the available clinical datasets for the specific disease and corresponding controls. Different sample counts may apply to specific omics subsets, as detailed in Supplementary Methods S1. Overview of study characteristics ROSMAP Dataset. UCSC Xena TCGA-BRCA Dataset. MayoRNASeq Dataset. Note: The sample sizes reported here are based on the available clinical datasets for the specific disease and corresponding controls. Different sample counts may apply to specific omics subsets, as detailed in Supplementary Methods S1. Religious Orders Study and Memory and Aging Project ROSMAP is a longitudinal cohort study combining the ROSEMAP Project to investigate aging and AD [40]. Participants undergo annual cognitive assessments and consent to brain donation, allowing linkage of clinical trajectories with postmortem molecular data. For this study, we used dorsolateral prefrontal cortex profiles of microRNA expression, messenger RNA (mRNA) expression, and DNA methylation, together with diagnostic information on AD. Only individuals with complete molecular and clinical data after quality control were included [41]. The Mayo RNAseq Study The MayoRNASeq study profiles neurodegenerative diseases using postmortem brain tissue from the Mayo Clinic Brain Bank [37]. We focused on temporal cortex samples from participants diagnosed with PSP and matched controls, with available RNA-Seq gene expression, proteomics, metabolomics, and clinical annotations. Samples with incomplete molecular or phenotypic information were excluded after standard quality control. Breast Cancer Cohort (TCGA-BRCA) TCGA-BRCA is a large-scale, multi-omics resource characterizing the molecular and clinical features of BC [42]. We used bulk RNA-seq transcriptomics, microRNA expression, and DNA methylation (Illumina HumanMethylation450) together with clinical annotations obtained via UCSC Xena. Only samples with complete profiles for these modalities and clinical data were retained. Data preprocessing and dataset construction All data cleaning, preprocessing, and preparation steps were performed within a unified Python 3.11.9 pipeline to ensure consistency across cohorts and omics modalities. Clinical metadata were harmonized with omics expression matrices using biospecimen mapping files, duplicate samples and features were removed, and samples with incomplete multi-omics profiles or missing clinical annotations were excluded. Features with excessive missingness were removed, and the remaining missing entries were imputed using a K-nearest neighbors imputer with distance-weighted averaging. Molecular identifiers (microRNA names and Ensembl gene IDs) were standardized against current reference databases to improve cross-study comparability. Full details of imputation thresholds, identifier mapping, and software versions are provided in Supplementary Methods S1. To reduce noise and improve interpretability, we applied modality-specific filtering and normalization steps inspired by the MOGONET framework [35]. Low-variance features were removed using modality-specific thresholds, and preselection was performed by Analysis of Variance (ANOVA) F-tests on the training data with false discovery rate (FDR) control. To limit redundancy, we enforced a correlation constraint by pruning features until the first principal component explained <50% of the variance. Each omics matrix was then scaled to the range [0, 1]; in PSP proteomics and metabolomics, a transformation was applied prior to normalization. A detailed description of these procedures, including variance thresholds by modality and alternative preprocessing choices, is provided in Supplementary Methods S1. For each cohort, we constructed datasets for three integration settings: (i) single-omics, (ii) all available paired dual-omics combinations, and (iii) triple-omics integration (mRNA + miRNA + methylation in AD and BC; mRNA + proteomics + metabolomics in PSP). This design resulted in different sample sizes and class distributions across disease cohorts. In BC, where substantial class imbalance was observed between tumor and control samples, the majority class was downsampled to 1.2 times the minority class in all experimental settings. A summary of the resulting datasets is shown in Table 2. Table 2Summary of prepared datasets for single-, dual-, and multi-omics analyses Cohort Omics combination Samples Features Classes Class ratio ADmiRNA only378200AD/Control209/169mRNA only378200AD/Control209/169Meth only375200AD/Control207/168mRNA + miRNA375200 + 200AD/Control209/169mRNA + Meth375200 + 200AD/Control207/168miRNA + Meth375200 + 200AD/Control207/168mRNA + miRNA + Meth375200 + 200 + 200AD/Control207/168PSPmRNA only162200PSP/Control84/78Prot only111200PSP/Control83/29Metab only98200PSP/Control78/20mRNA + Prot111200 + 200PSP/Control83/29mRNA + Metab98200 + 200PSP/Control78/20Metab + Prot97200 + 200PSP/Control77/20mRNA + Prot + Metab97200 + 200 + 200PSP/Control77/20BCmiRNA only158200BC/Control83/75mRNA only236200BC/Control124/112Meth only375200BC/Control106/96mRNA + miRNA158200 + 200BC/Control83/75mRNA + Meth108200 + 200BC/Control57/51miRNA + Meth175200 + 200BC/Control92/83mRNA + miRNA + Meth108200 + 200 + 200BC/Control57/51Abbreviations: miRNA, microRNA expression data; mRNA, gene expression data; Meth, DNA methylation data; Prot, proteomics data; Metab, metabolomics data. Summary of prepared datasets for single-, dual-, and multi-omics analyses Abbreviations: miRNA, microRNA expression data; mRNA, gene expression data; Meth, DNA methylation data; Prot, proteomics data; Metab, metabolomics data. Feature ranking and ensemble aggregation To identify predictive and biologically meaningful features across heterogeneous, high-dimensional multi-omics datasets, we applied 15 single-feature ranking approaches spanning statistical tests, regularized regression, tree-based importance measures, explainable AI methods, and deep learning-based multi-omics rankers. These rankers capture complementary linear and nonlinear dependencies and are summarized in Supplementary Table S2. To improve robustness and reduce method-specific biases, we further applied 13 ensemble rank aggregation strategies that combine the outputs of single rankers into consensus feature lists. We considered both rank-based aggregation (which integrates the relative ordering of features across methods) and weight-based aggregation (which combines normalized importance scores). The ensemble methods are summarized in Supplementary Table S3. We excluded MORE-Ranker and MOGONET-Ranker from the ensembles to preserve a clear comparison between (i) conventional single rankers, (ii) ensemble-based aggregations, and (iii) deep learning-based integration models. Benchmark setup Starting from the cleaned and prepared datasets, we designed a benchmarking framework to evaluate feature selection strategies, predictive models, and multi-omics integration levels. Experiments were performed independently for AD, PSP, and BC across three integration settings: single-omics, dual-omics, and triple-omics. In the single-omics setting, each modality (mRNA, miRNA, methylation, proteomics, or metabolomics) was analyzed separately. Dual-omics settings included all pairwise modality combinations available within each cohort, and triple-omics settings integrated three modalities (mRNA + miRNA + methylation for AD and BC; mRNA + proteomics + metabolomics for PSP). To ensure fair comparison with MORE and MOGONET, which are primarily designed for three-layer integration, we restricted analyses to a common three-modality subset in each cohort. For every omics subset, we derived biomarker panels of increasing size: {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}, plus a full-feature panel (200 features for single-omics, 400 for dual-omics, 600 for triple-omics). This allowed us to study performance as a function of panel sparsity and interpretability. In total, we evaluated 27 feature selection strategies (15 single rankers and 12 ensemble rankers) and 11 predictive models: logistic regression (L.Regression), Random Forest, Extreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), Adaptive Boosting (AdaBoost), Gradient Boosting (Gradient Boost), Support Vector Machine (SVM), multilayer perceptron (MLP), Decision Tree (D.Tree), and the multi-omics deep learning models MORE and MOGONET. Model training and evaluation were performed using five-fold stratified cross-validation to maintain balanced class distributions across folds. All analyses were implemented in Python 3.11.9 using scikit-learn for traditional machine learning, xgboost, catboost, and lightgbm for gradient boosting methods, and author-provided implementations for MORE and MOGONET. A schematic overview of the workflow is presented in Figs 1 and 2 . Experimental design and data modalities across cohorts. Overview of datasets and preprocessing steps used in the benchmarking study. Data from AD, PSP, and BC were harmonized through cleaning procedures including duplicate removal, filtering, imputation, and feature standardization, along with modality-specific preprocessing such as variance filtering, ANOVA F-tests, FDR correction, correlation constraints, min–max scaling, and log transformation. The experimental design integrates multiple single-omics, dual-omics, and triple-omics configurations. The lower panel summarizes the number of samples and features available for each omics modality and their combinations per cohort, highlighting the balanced yet heterogeneous structure of the benchmarking framework. Analysis workflow per dataset. Schematic overview of the benchmarking pipeline applied to each cohort. (1) Features were ranked using a diverse set of methods, including statistical tests (t-test, Mann–Whitney U), regularized models (LASSO, Ridge, Elastic Net), model-based importance methods (Random Forest, XGBoost, Boruta, SVM Recursive Feature Elimination (SVM-RFE)), explainability-based approaches (SHapley Additive exPlanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME)), and deep learning-embedded rankers (MORE-Ranker, MOGONET-Ranker). (2) Ranked feature lists were aggregated using ensemble strategies such as mean, median, geometric mean, weighted schemes, robust rank aggregation (RRA), and Stuart rank. (3) From each ranking, biomarker panels of varying sizes (10–100 features and full lists) were selected. (4) These panels were evaluated using five-fold cross-validation across multiple classifiers, including D.Tree, L.Regression, Random Forest, Gradient Boosting, XGBoost, CatBoost, AdaBoost, SVM, and MLP, as well as deep learning frameworks (MORE, MOGONET). (5) Finally, selected biomarker panels were validated against external databases, including HMDD, CTD, GeneCards, and EWAS-ATLAS, to assess biological relevance. Biomarker panel validation To assess biological and clinical relevance, each biomarker panel generated during benchmarking was validated against external reference databases at multiple molecular levels. For integrated panels containing features from different omics types, we first split features into modality-specific subsets (genes, microRNAs, and CpG sites) and then queried the most appropriate resource for each type. MicroRNA–disease associations were evaluated using HMDD (v4.0, 2025) [28]. Gene-level associations were retrieved from the CTD (Release 27 August 2025) [27] and GeneCards (Version 5.25) [29]. DNA methylation sites were cross-referenced against the EWAS-ATLAS (v2025) [30]. For pathway-level validation, we performed functional enrichment using g:Profiler (v0.2, 2025) [39] and compared enriched pathways to curated pathway–disease associations in CTD. Construction of validation datasets and scoring procedures are described in detail in Supplementary Methods S3. Evaluation metrics Predictive performance was assessed within the five-fold stratified cross-validation framework using standard classification metrics. We report accuracy, precision (positive predictive value), recall (sensitivity), specificity (true negative rate), negative predictive value (NPV), F-score, and the area under the receiver operating characteristic curve (AUC-ROC or AUC). F was used as the primary summary metric under class imbalance, and AUC-ROC was used to compare global discriminative performance independently of decision threshold. To quantify robustness, we defined performance stability as the width of the observed performance window for each model–selector combination, computed as the difference between the maximum and minimum F (or AUC) across experimental replicates. Narrower windows indicate more stable and reproducible performance. Formal definitions of all metrics are provided in Supplementary Methods S4. All benchmarking experiments were implemented in a Python 3.11.9 environment on the National Center for High Performance Computing (UHeM) clusters (hostname: sariyer) running CentOS Linux 7 (Core) with Linux kernel version 3.10.0. The server is equipped with dual Intel Xeon E5-2680 v4 processors (2.40 GHz), providing 28 CPU cores and approximately 128 GB of system memory. All models were trained using central processing unit (CPU) resources only, and graphics processing unit (GPU) acceleration was not used. Experiments were executed using batch jobs through the SLURM workload manager, typically allocating up to 40 parallel tasks on a single compute node to accelerate cross-validation and benchmarking runs. To ensure reproducibility, the complete benchmarking framework, including data preprocessing pipelines, feature selection modules, model implementations, and instructions for reproducing the computational environment, is publicly available at the following GitHub repository: https://github.com/itu-bioinformatics-database-lab/biomarker-benchmark Results Baseline model performance without feature selection Across the three cohorts (AD, PSP, and BC), we first evaluated predictive performance using all available features without applying any feature selection (Table 3). In AD, MLP achieved the highest AUC (0.85 0.05) and a correspondingly strong score (0.73 0.05), closely followed by L.Regression (AUC = 0.84 0.05, = 0.71 0.05). Although MOGONET and MORE reached moderate AUCs (0.81 0.03–0.04), they maintained competitive scores around 0.74 0.03–0.05 due to balanced precision–recall behavior. In contrast, D.Tree performed poorest overall (AUC = 0.61 0.08, = 0.56 0.13). Table 3Baseline classification results across cohorts without feature selection, reporting performance for 11 models using triple-omics data. Cohort Model AUC Accuracy Precision Recall Specificity NPV ADL.Regression0.84 0.050.73 0.060.71 0.080.71 0.060.71 0.050.75 0.100.76 0.05Random Forest0.78 0.040.70 0.040.67 0.040.65 0.090.66 0.060.74 0.050.73 0.05XGBoost0.80 0.040.71 0.040.68 0.050.68 0.080.68 0.050.73 0.060.74 0.05D.Tree0.61 0.080.61 0.070.55 0.090.59 0.160.56 0.130.63 0.030.66 0.07Gradient Boost0.79 0.050.71 0.050.67 0.070.69 0.060.68 0.040.72 0.090.74 0.04CatBoost0.81 0.050.73 0.050.71 0.060.68 0.080.69 0.050.76 0.090.75 0.05AdaBoost0.79 0.060.73 0.060.71 0.060.67 0.080.69 0.07 0.77 0.05 0.75 0.06MLP 0.85 0.05 0.74 0.060.70 0.070.74 0.050.72 0.060.73 0.080.78 0.05SVM0.81 0.040.72 0.040.68 0.020.71 0.130.69 0.060.72 0.060.77 0.07MORE0.81 0.030.73 0.050.70 0.08 0.75 0.11 0.72 0.050.72 0.13 0.79 0.06 MOGONET0.81 0.04 0.76 0.03 0.73 0.05 0.75 0.06 0.74 0.03 0.77 0.06 0.79 0.03 PSPL.Regression0.98 0.030.94 0.070.94 0.060.99 0.020.96 0.040.75 0.270.90 0.20Random Forest0.92 0.050.90 0.070.91 0.060.97 0.030.94 0.040.60 0.300.85 0.20XGBoost0.96 0.030.91 0.060.92 0.060.97 0.030.94 0.040.65 0.250.88 0.15D.Tree0.64 0.120.82 0.070.85 0.050.94 0.080.89 0.040.35 0.250.57 0.40Gradient Boost0.83 0.150.88 0.120.90 0.090.96 0.050.93 0.070.55 0.400.70 0.40CatBoost0.95 0.050.90 0.050.89 0.05 1.00 0.00 0.94 0.030.50 0.22 1.00 0.00 AdaBoost0.95 0.040.93 0.080.95 0.050.96 0.050.96 0.050.80 0.190.85 0.20MLP0.97 0.03 0.97 0.03 0.97 0.03 0.99 0.02 0.98 0.02 0.90 0.12 0.96 0.08SVM0.98 0.020.93 0.050.93 0.040.99 0.020.96 0.030.70 0.190.93 0.13MORE 0.99 0.01 0.76 0.190.57 0.23 1.00 0.00 0.69 0.190.70 0.24 1.00 0.00 MOGONET0.89 0.000.75 0.000.33 0.000.25 0.000.29 0.000.88 0.000.82 0.00BCL.Regression 1.00 0.01 0.98 0.020.98 0.030.98 0.040.98 0.020.98 0.040.98 0.03Random Forest 1.00 0.01 0.99 0.02 1.00 0.00 0.98 0.04 0.99 0.02 1.00 0.00 0.98 0.03XGBoost0.97 0.040.97 0.040.98 0.040.97 0.040.97 0.040.98 0.040.96 0.04D.Tree0.93 0.080.93 0.080.94 0.050.91 0.140.92 0.090.94 0.050.92 0.11Gradient Boost0.98 0.030.96 0.030.98 0.030.95 0.070.96 0.040.98 0.040.95 0.06CatBoost 1.00 0.00 0.99 0.02 1.00 0.00 0.98 0.04 0.99 0.02 1.00 0.00 0.98 0.03AdaBoost0.97 0.040.96 0.030.98 0.030.95 0.070.96 0.040.98 0.040.95 0.06MLP 1.00 0.01 0.98 0.020.98 0.030.98 0.040.98 0.020.98 0.040.98 0.03SVM 1.00 0.01 0.99 0.02 1.00 0.00 0.98 0.04 0.99 0.02 1.00 0.00 0.98 0.03MORE 1.00 0.00 0.95 0.000.91 0.00 1.00 0.00 0.95 0.000.91 0.00 1.00 0.00 MOGONET0.92 0.000.95 0.000.92 0.00 1.00 0.00 0.96 0.000.91 0.00 1.00 0.00 Bold values indicate the best-performing result for each evaluation metric (column). Baseline classification results across cohorts without feature selection, reporting performance for 11 models using triple-omics data. Bold values indicate the best-performing result for each evaluation metric (column). In PSP, nearly all models achieved excellent discrimination (AUC > 0.95), with MLP and SVM delivering consistently high scores (0.98 0.02) and balanced accuracy. MORE achieved the top AUC (0.99 0.01) and perfect recall (1.00 0.00) but exhibited low precision (0.57 0.23), which reduced its to 0.69 0.19. MOGONET, by contrast, showed a sharp drop in both precision and recall, yielding an anomalously low (0.29 0.00) despite a relatively high AUC (0.89 0.00), suggesting overfitting or miscalibration in class probability outputs. In BC, ceiling effects were observed across nearly all classifiers. L.Regression, SVM, and MORE achieved near-perfect performance with AUCs between 0.98 and 1.00 and scores ranging from 0.96 to 1.00. Gradient-based ensemble models (e.g. XGBoost, CatBoost) also performed at parity, with both AUC and > 0.97, indicating minimal performance differentiation. MOGONET slightly trailed with AUC 0.92 0.00 and 0.96 0.00 despite perfect recall, again hinting at potential imbalance handling issues. Taken together, these results demonstrate that while deep integration models such as MORE and MOGONET can achieve high AUCs, they do not always translate these into equally strong scores—particularly in smaller or less balanced cohorts. Conversely, traditional single-model approaches such as L.Regression, SVM, and MLP often deliver more stable and interpretable performance, matching or surpassing complex models when no feature selection is applied. Impact of feature selection Applying feature selection led to consistent performance gains across all cohorts (Fig. 3a). Mean F1-scores increased notably when models were trained on subsets of top-ranked biomarkers rather than the full feature sets, reflecting the benefit of reducing noise and redundancy. The most pronounced improvements were observed in AD and BC, where cross-validation stability was enhanced and several traditional methods matched or outperformed more complex architectures once dimensionality was reduced. PSP models, already near ceiling without selection, showed modest but still measurable gains in precision and F1, suggesting that targeted feature reduction can fine-tune performance even in highly separable cohorts. Overall, feature selection improved predictive accuracy while also keeping simpler models competitive, making the resulting biomarker panels easier to interpret and more suitable for clinical use. Benchmarking model performance across datasets, omics levels, models, and feature selection strategies. (a) Comparison of mean F1-scores with and without feature selection across AD, PSP, and BC, showing consistent performance improvements with feature selection. (b) Performance by data modality and integration strategy, where triple-omics combinations frequently outperform single- and dual-omics configurations. (c) Comparison of single-omics, dual-omics, and triple-omics integration across cohorts, highlighting the trade-off between information gain and potential noise. (d) Performance of individual machine learning models, showing that traditional linear and tree-based methods remain competitive with deep learning approaches such as MORE and MOGONET. (e) Impact of different feature selection rankers, indicating that ensemble aggregation strategies provide more stable and accurate results compared with individual rankers. Collectively, these panels illustrate how feature selection and multi-omics integration influence predictive performance across benchmarking settings. Effect of omics integration strategies We next assessed the effect of single-, dual-, and triple-omics integration strategies on predictive performance (Figs 3b and c; and 4). Figure 3b and c summarizes the median of average cross-validation -scores across selector–classifier–panel size combinations, while Fig. 4 shows the peak performance achieved across omics configurations. Together, these complementary perspectives capture both central trends and best-case outcomes. Across all cohorts, predictive performance improved progressively from single-omics to dual-omics and reached its highest levels with triple-omics integration. In BC, all integration levels achieved near-ceiling medians and maxima (0.95–1.0 across metrics), confirming that the dataset is highly separable regardless of modality configuration, though triple-omics still achieved the most consistent top scores. In PSP, single-omics models already performed strongly, but integration—particularly triple-omics—yielded small yet measurable gains in precision and scores. AD showed the most pronounced benefit from integration: performance increased substantially when moving from single- to dual-omics, and further improved under triple-omics configurations, with the mRNA + methylation + miRNA combination consistently outperforming all alternatives. Overall, these results demonstrate that predictive accuracy and stability improve as the level of omics integration increases, with triple-omics providing the most comprehensive and robust representation of the underlying biological signal, followed by dual-omics and single-omics analyses. Model performance across omics types and integration strategies in AD, PSP, and BC, where AUC, accuracy, F1-score, NPV, precision, recall, and specificity are reported for the best-performing combinations of feature selector, classifier, and panel size within each omics configuration, with bars representing individual omics inputs (mRNA, miRNA, methylation, proteomics, metabolomics) and their pairwise or three-way integrations, showing that triple-omics configurations achieve the highest performance across most metrics while dual-omics provide intermediate performance between single- and triple-omics settings, illustrating how predictive performance varies across integration strategies. Benchmarking across classifiers Classifier performance varied across cohorts and integration strategies, but a consistent trend emerged: traditional machine learning approaches often matched or surpassed deep learning frameworks (Figs 3d, 5b and 6). In the median F1 comparison (Fig. 3d), L.Regression, SVM, ensemble tree models (AdaBoost and CatBoost), and MLP consistently occupied the top tier across cohorts, whereas D.Tree, Gradient Boost, MORE, and MOGONET generally ranked lower with reduced medians. This indicates that well-regularized linear/nonlinear baselines and MLP yield the most reliable central performance, while the more specialized multiview methods do not confer a universal advantage. Comparative performance of feature selection strategies and classifiers across cohorts. (a) Precision–recall plots summarizing the trade-off between recall and precision for different feature selection rankers across AD, PSP, and BC, where ensemble rankers demonstrate improved balance compared with individual methods. (b) Precision–recall analysis across classifiers, showing that tree-based ensemble methods (AdaBoost, XGBoost, and CatBoost) and linear models (L.Regression, SVM, and MLP) achieve performance comparable to deep learning models (MORE and MOGONET). Model performance across biomarker panel sizes in AD, PSP, and BC cohorts, where AUC, accuracy, F1-score, NPV, precision, recall, and specificity are plotted against the number of selected features for multiple classifiers—including traditional linear models (L.Regression and support vector classifier, SVC), tree-based ensemble methods (Random Forest, XGBoost, CatBoost, AdaBoost, and Gradient Boosting), MLP, and deep learning approaches (MORE and MOGONET)—showing rapid performance gains at small panel sizes (10–30 features), stabilization at intermediate sizes (50–75 features), and minimal changes beyond 100 features, illustrating how predictive performance varies with biomarker panel size across modeling approaches. Further insights come from the precision–recall analysis in Fig. 5b. Points closest to the top-right corner correspond to classifiers achieving the best balance between precision and recall. MLP and L.Regression clustered near this frontier across all three cohorts, with CatBoost and Gradient Boost also performing strongly. AdaBoost trailed slightly beneath this group, while D.Tree formed a mid-tier cluster with moderate precision and recall. Random Forest, SVM, and XGBoost generally occupied the upper-mid to upper-right regions, reflecting high recall with somewhat lower precision depending on the dataset. By contrast, MORE and MOGONET consistently populated the lower parts of the panels, indicating simultaneously weaker precision and recall across cohorts. A more granular perspective is provided by Fig. 6, which tracks classifier performance across feature set sizes and multiple evaluation metrics (AUC, accuracy, F1, NPV, precision, recall, and specificity) in BC, PSP, and AD. In BC, nearly all classifiers achieved ceiling-level performance across all metrics regardless of feature number. Linear (L.Regression) and nonlinear models (CatBoost, SVM, XGBoost, and MLP) maintained mean values near 1.0 with minimal sensitivity to panel size, underscoring the strength of the dataset signal and suggesting that even simple models with modest panels can achieve near-perfect discrimination. By contrast, PSP revealed greater variability. While AUC, accuracy, and F1 scores generally stabilized >0.9 across classifiers, specificity declined with larger feature sets, especially in Gradient Boost and XGBoost. L.Regression, SVM, and MLP displayed the most consistent profiles, maintaining balanced sensitivity and precision with narrow fluctuations. D.Tree, Gradient Boost, Random Forest, MORE, and MOGONET lagged, with reduced precision and specificity despite achieving moderate recall. In AD, the most challenging cohort, feature set size had a pronounced effect on model performance. L.Regression, MOGONET, and MLP exhibited steady improvements across all evaluation metrics as biomarker panels expanded, with performance plateauing 50–60 features across nearly all metrics. Tree-based ensembles delivered competitive recall but showed greater metric-specific variability: Random Forest and XGBoost achieved strong recall yet more modest precision. These results highlight the influence of panel size on predictive stability and emphasize the need for balanced feature selection in noisy, multi-omics Alzheimer’s data. Collectively, these results demonstrate that metric-specific behaviors are as important as overall accuracy. Precision and specificity tended to degrade more rapidly with larger feature sets, particularly in PSP and AD, while recall remained relatively stable. Traditional methods and MLP consistently balanced these tradeoffs, whereas MORE and MOGONET struggled to maintain parity across metrics. These findings highlight the dual importance of parsimonious feature selection and classifier choice in multi-omics biomarker discovery, and caution against assuming that increased model complexity alone yields superior results. Performance comparison of feature selection strategies We benchmarked 27 feature ranking methods, including MORE and MOGONET, single rankers, and ensemble aggregation approaches. Figure 3e summarizes their mean F1 score distributions across cohorts. Figure 7 provides a direct comparison between single feature selectors, ensemble aggregation methods, and deep learning-embedded rankers across cohorts . Ensemble-based selectors such as geometric mean rank, mean rank, median rank, median rank algorithm, Stuart rank, max weight, median weight, and RRA rank achieved consistently strong performance with high medians and relatively narrow spreads. Several traditional selectors, including t-Test, Least Absolute Shrinkage and Selection Operator (LASSO), Mann–Whitney U, random forest feature importance (RF-FI), and XGBoost feature importance (XGB-FI), performed comparably well, highlighting their continued competitiveness. By contrast, LIME and the more recent multiview-specific deep learning rankers (MOGONET-Ranker and MORE-Ranker) exhibited lower medians and broader variability, indicating weaker and less stable performance. SHAP fell into an intermediate category, outperforming the weakest selectors but not matching the stability of ensembles or top traditional methods. Comparison of single and ensemble feature selection methods across cohorts. Mean F1-scores for individual (single) rankers, ensemble aggregation methods, and embedded rankers from deep learning frameworks (MORE and MOGONET) across BC, PSP, and AD datasets. Bars represent average performance across classifiers and biomarker panel sizes. Ensemble methods such as Mean Weight, Geomean Weight, TA, RRA, and Stuart show higher and more stable performance across cohorts, while regularized single methods (Elastic Net, LASSO, and Ridge) achieve comparable results in some settings, particularly in BC. Deep learning-embedded rankers (MORE and MOGONET) exhibit variable performance relative to these approaches. These results illustrate differences in performance across feature selection strategies. Figure 5a presents precision–recall scatter plots per selector, with points colored by cohort. The apparent groupings are primarily cohort effects: BC concentrates near the upper-right (high precision and recall), PSP forms an intermediate band, and AD lies lower—especially on recall. Against this backdrop, selectors differ mainly in (i) the overall location of this three-cohort triad and (ii) the spread between cohorts. Ensemble aggregators (e.g. mean/median/geometric-mean rank/weight, RRA, Stuart) and strong traditional methods (t-Test, LASSO, Elastic Net, RF-FI, XGB-FI, and Boruta) place the triad higher with tighter dispersion, whereas SHAP, LIME, SVM-RFE, Ridge, Min Rank, Max Weight, and Median Rank Aggregation show mid-level placement and wider spread. The multiview rankers (MORE-Ranker and MOGONET-Ranker) and Threshold Algorithm (TA) tend to sit lower overall and exhibit larger cohort gaps. Thus, the separation visible in the figure reflects differences in cohort separability rather than intrinsic clustering of selectors; methods that both elevate the triad and compress the BC–PSP–AD gap are the most robust. Figure 8 further summarizes how classifier performance varies across feature rankers and how biomarker panel size influences -scores across AD, PSP, and BC. Model performance across rankers, classifiers, and biomarker panel sizes in three cohorts. (a) Mean F1-scores of different classifiers (D.Tree, L.Regression, SVM, Random Forest, Gradient Boosting, AdaBoost, CatBoost, XGBoost, MLP, MORE, MOGONET) when paired with feature subsets generated by multiple rankers across AD, PSP, and BC. Ensemble rankers are associated with higher classifier performance compared to individual rankers, while tree-based ensemble methods and linear models show performance comparable to deep learning frameworks. (b) Effect of biomarker panel size on F1-score across rankers and cohorts. Performance increases with small panel sizes (10–30 features) and stabilizes at larger panel sizes. Together, these panels illustrate how feature selection strategy, classifier choice, and panel size influence predictive performance across cohorts. Biological validation of discovered biomarkers We quantified biological plausibility as true positives (TPs)—the overlap between the top- selected biomarkers and the top-1000 entries from external databases (HMDD, CTD-pathways, CTD, GeneCards, and EWAS-ATLAS)—for (Figs 9– 12). Across feature selectors, ensemble methods (Geom. Mean (rank/weight), Mean/Median (rank), Stuart, RRA) consistently achieved the highest TP counts, with strong traditional rankers [t-test, LASSO, RF-FI, XGBoost Feature Importance (XGB-FI), SVM-RFE] often close behind; methods such as LIME, TA, MOGONET-Ranker, and MORE-Ranker generally validated less. Across omics strategies, single-omics (top panels) showed the lowest recoveries, while dual-omics (middle) and especially triple-omics (bottom) yielded higher overlap across databases. Cohort differences were evident but stable across cutoffs: BC typically attained the highest absolute TPs (including CpG validations), AD benefited most in miRNA-based validation (HMDD), and PSP showed moderate gains . Increasing raised absolute TP counts and slightly compressed between-method gaps, but the qualitative ordering—and the advantage of multi-omics integration—remained consistent across all four figures. External validation results at the Top-10 cutoff. Bars represent overlaps with curated disease-associated features across multiple databases, including HMDD (miRNAs), CTD-pathways, CTD, GeneCards, and EWAS-ATLAS, for single-, dual-, and triple-omics configurations across AD, PSP, and BC cohorts. Multi-omics integration is associated with higher overlap counts, particularly in CTD-pathways, while ensemble-based methods show consistently strong performance across cohorts. External validation results at the Top-100 cutoff. Bars represent overlaps with curated disease-associated features across validation databases for AD, PSP, and BC cohorts. Overlap counts are highest at this panel size, with signs of saturation in some datasets. Multi-omics configurations continue to show higher overlap values, and the relative performance differences between methods are largely maintained. External validation results at the Top-30 cutoff. Bars represent overlaps with curated disease-associated features across HMDD, CTD-pathways, CTD, GeneCards, and EWAS-ATLAS across AD, PSP, and BC cohorts. A wider dynamic range of overlap counts is observed compared with smaller panel sizes, with CTD-pathways and EWAS-ATLAS showing increased validation. The relative ordering of methods remains similar, with ensemble-based approaches generally achieving higher overlap counts. In this study, we adopt a data-driven definition of biomarker discovery, referring to the identification of molecular features that robustly discriminate between disease and control samples within each omics layer. This definition aligns with the analytical phase of biomarker research, where computational models prioritize disease-associated molecules based on reproducibility and predictive value, rather than immediate clinical applicability. Following the classical framework of biomarker classification [43], our findings should be interpreted as molecular signatures of disease derived from postmortem tissue (AD and PSP) and tumor tissue (BC). While these signatures provide valuable insight into disease mechanisms and may guide future development of accessible biomarkers in biofluids (e.g. blood or cerebrospinal fluid), the current analyses focus on computational prioritization within affected tissue contexts. Recent large-scale efforts, such as the blood-based molecular atlas of AD [44], further emphasize the importance of translating tissue-derived molecular signatures into accessible biofluid biomarkers. Such complementary studies support the view that multi-omics profiling in postmortem tissue can inform the search for peripheral correlates of neurodegenerative processes. External validation results at the Top-50 cutoff. Bars represent overlaps with curated disease-associated features across multiple validation databases for AD, PSP, and BC cohorts. Overlap counts increase across all cohorts, with dual- and triple-omics configurations generally showing higher values. Differences between feature selection methods become less pronounced at this panel size. Cross-omics biomarkers identified per cohort Across cohorts, our framework revealed both validated and novel biomarker candidates reproducibly prioritized across multiple omics levels and feature selection strategies. Selection frequencies and rank heatmaps (Tables 4– 6; Figs 13– 15) summarize the consistency and stability of identified features. To focus on reproducible signals, we restricted the lists to markers that appeared among the top 10 ranked features in at least 20 distinct rankers. For each candidate, the tables report the omics levels in which it was selected, the number of rankers supporting its inclusion (), and the total frequency of top-10 appearances across omics settings. The corresponding heatmaps display the same set of features, showing their minimum rank (achieved at each omics level) across all rankers, thereby complementing the frequency-based summaries with a visual representation of ranking stability. Table 4AD summary table of reproducible cross-omics biomarkersFeatureFeatureTypeOmicsLevelSelection frequencyN (Selectors)cg06690548MethSingle+Dual+Triple3720cg12981137MethSingle+Dual+Triple3517cg12845808MethSingle+Dual3421cg19832721MethSingle+Dual+Triple3318cg16003238MethSingle+Dual+Triple3218cg22442730MethSingle+Dual+Triple2917cg12864235MethSingle+Dual2416cg24765079MethSingle+Dual+Triple2016cg02489552MethSingle+Dual+Triple2013APLNmRNASingle+Dual+Triple5118CCDC69mRNASingle+Dual+Triple4717PPDPFmRNASingle+Dual+Triple4317SLC6A12mRNASingle+Dual+Triple4216PTPRFmRNASingle+Dual+Triple3618MEIS3mRNASingle+Dual+Triple3414CNN3-DTmRNASingle+Dual3318PLEKHM2mRNASingle+Dual+Triple3217RPL29mRNASingle+Dual+Triple3214QDPRmRNASingle+Dual+Triple2415SPACA6mRNASingle+Dual+Triple2114hsa-miR-129-5pmiRNASingle+Dual+Triple8322hsa-miR-132miRNASingle+Dual+Triple8022hsa-miR-885-5pmiRNASingle+Dual+Triple5520hsa-miR-29amiRNASingle+Dual+Triple3722hsa-miR-146b-5pmiRNASingle+Dual+Triple2715hsa-miR-99amiRNASingle+Dual+Triple2312hsa-miR-129-3pmiRNASingle+Dual+Triple2219 AD summary table of reproducible cross-omics biomarkers Table 6BC summary table of reproducible cross-omics biomarkersFeatureFeatureTypeOmicsLevelSelection frequencyN (Selectors)cg15700197MethSingle+Dual+Triple5419cg23290344MethDual+Triple3319cg21504624MethSingle+Dual2320cg11052143MethSingle+Dual2317cg25691167MethSingle+Dual+Triple2219cg26353877MethDual+Triple2015COL10A1mRNASingle+Dual+Triple7224MMP11mRNASingle+Dual+Triple5724LRRC3BmRNASingle+Dual+Triple5322FIGFmRNASingle+Dual3717CD300LGmRNASingle+Dual+Triple3619SPRY2mRNASingle+Dual3319PPAPDC1AmRNASingle+Dual+Triple2814HSD17B6mRNASingle+Dual+Triple2722ADAMTS5mRNASingle+Dual+Triple2619PPP1R12BmRNADual+Triple2616TMEM220mRNASingle+Dual+Triple2312SDPRmRNASingle+Dual2115CPA1mRNASingle+Dual+Triple2016CA4mRNASingle+Dual+Triple2014hsa-miR-21-5pmiRNASingle+Dual+Triple6924hsa-miR-139-5pmiRNASingle+Dual+Triple3520hsa-miR-96-5pmiRNASingle+Dual+Triple3221hsa-miR-10b-5pmiRNASingle+Dual+Triple3119hsa-miR-183-5pmiRNASingle+Dual+Triple2820hsa-miR-139-3pmiRNASingle+Dual+Triple2818hsa-miR-1307-5pmiRNASingle+Dual+Triple2419 BC summary table of reproducible cross-omics biomarkers Rank heatmap for the AD cohort showing feature ranking stability across multiple selection methods, where rows represent candidate microRNAs, genes, and CpG methylation sites and columns correspond to single and ensemble feature selection approaches, with each cell indicating the assigned rank (lower values denote higher importance), highlighting consistently top-ranked features such as hsa-miR-129-5p, hsa-miR-132, hsa-miR-146b-5p, ARRDC2, PLEKHM2, and PPDPF, along with additional candidates including hsa-miR-885-5p, APLN, CCDC69, SPACA6, MEIS3, and CNN3-DT, and several CpG sites (e.g., cg12981137, cg22442730, cg16003238, cg19832721, and cg12864235) that also show consistently high rankings across methods. Rank heatmap for the PSP cohort showing feature ranking stability across multiple selection methods, where rows represent candidate mRNA, proteins, and metabolites and columns correspond to single and ensemble feature selection approaches, with each cell indicating the assigned rank (lower values denote higher importance), highlighting consistently top-ranked features such as TMAO, ANKRD11, and PPT1, along with additional candidates including TBCK, ARHGAP19-SLIT1, ARHGAP35, ASAH1, and metabolites such as stachydrine, trigonelline, salicylate, 1-methyl-5-imidazoleacetate, 2-methylserine, 12-HHTrE, and 1-linoleoyl–GPC (18:2). Rank heatmap for BC. Feature ranking stability across multiple selection methods in the BC cohort. Rows represent candidate features, including microRNAs, genes, and CpG methylation sites, while columns correspond to feature selection methods grouped into single and ensemble approaches. Each cell displays the rank assigned to a feature by a given method, with smaller values indicating higher importance. Several features, including hsa-miR-21-5p, hsa-miR-139-5p/3p, hsa-miR-96-5p, COL10A1, MMP11, and FIGF, are consistently ranked highly across methods, along with additional candidates such as PPP1R12B, LRRC3B, TMEM220, PPAPDC1A, HSD17B6, ADAMTS5, SDPR, and SPRY2. Multiple CpG methylation sites (e.g. cg23290344, cg26353877, cg25691167, cg11052143, and cg15700197) also show consistently high rankings across methods. In AD, miRNAs emerged as particularly stable cross-omics markers. hsa-miR-129-5p and hsa-miR-132 were selected by 22 rankers with the highest frequencies (83 and 80, respectively), consistent with their strong prior evidence in AD-related synaptic regulation and neuroinflammation. Other validated miRNAs such as hsa-miR-29a and hsa-miR-146b-5p also ranked highly, alongside novel signals like hsa-miR-885-5p, which showed robust reproducibility across rankers. Several mRNAs (APLN, PPDPF, PLEKHM2, MEIS3, CCDC69, and SPACA6) were consistently identified, some with previous links to AD pathology, while others remain poorly characterized. Importantly, CpG methylation sites (cg06690548, cg12981137, cg19832721, cg16003238, and cg22442730) formed a reproducible block of epigenetic candidates, underscoring the contribution of DNA methylation to AD-associated regulatory disruption. Together, these findings demonstrate the ability of ensemble feature selection to recover both known AD biomarkers and underexplored regulatory signals across omics layers. In PSP, metabolite signals were dominant, reflecting strong metabolic dysregulation. Highly ranked metabolites included trimethylamine N-oxide (TMAO), a known neuroinflammatory marker, along with 1-linoleoyl–GPC (18:2), stachydrine, trigonelline, salicylate, and 1-methyl-5-imidazoleacetate, which were selected with high frequency across rankers and omics levels. Protein markers such as TBCK, ANKRD11, and PPT1 were also reproducibly prioritized, with PPT1 aligning with known lysosomal dysfunction in neurodegeneration. Several novel candidates, including ARHGAP19-SLIT1, ARHGAP35, and ASAH1, were consistently ranked but have limited prior evidence in PSP, suggesting new directions for functional validation. The integration of metabolomics and proteomics proved especially effective in uncovering reproducible signals, highlighting the relevance of metabolic-lysosomal interactions in PSP disease pathology. In BC, both miRNAs and mRNAs were robustly recovered, with strong overlap with established oncogenic drivers. Validated candidates such as hsa-miR-21-5p, hsa-miR-139-5p/3p, hsa-miR-141-3p, COL10A1, MMP11, and FIGF were consistently prioritized, confirming the framework’s ability to detect well-characterized cancer biomarkers. At the same time, several mRNAs lacking strong prior association with BC (PPP1R12B, LRRC3B, TMEM220, PPAPD1C1A, HSD17B6, ADAMTS5, SDPR, and SPRY2) were reproducibly selected across rankers, highlighting potential novel players in BC biology. Similarly, CpG sites such as cg26353877, cg23290344, and cg11052143 repeatedly achieved top rankings, despite limited prior characterization, suggesting unexplored epigenetic signals. These results emphasize how multi-omics integration not only validates known oncogenic pathways but also systematically identifies reproducible novel features across data modalities. Together, these analyses show that ensemble ranking consistently recovers disease-relevant biomarkers across AD, PSP, and BC, while also revealing reproducible novel features that warrant further experimental investigation. Table 5PSP summary table of reproducible cross-omics biomarkersFeatureFeatureTypeOmicsLevelSelection frequencyN (Selectors)N-acetyl-3-methylhistidine*MetabSingle+Dual+Triple9023stachydrineMetabSingle+Dual+Triple88231-linoleoyl-GPC (18:2)MetabSingle+Dual+Triple84241-methyl-5-imidazolelactateMetabSingle+Dual+Triple8024trigonelline (N’-methylnicotinate)MetabSingle+Dual+Triple7320trimethylamine N-oxideMetabSingle+Dual+Triple63221-methyl-5-imidazoleacetateMetabSingle+Dual+Triple60181,2-dilinoleoyl-GPC (18:2/18:2)MetabSingle+Dual+Triple361812-HHTrEMetabSingle+Dual+Triple2617salicylateMetabSingle+Dual+Triple23152-methylserineMetabSingle+Dual+Triple2015TBCKProtSingle+Dual+Triple5822NTN5ProtSingle+Dual+Triple3718ANKRD11ProtSingle+Dual+Triple3517ARHGAP19-SLIT1ProtSingle+Dual+Triple3017ARHGAP35ProtSingle+Dual+Triple2722E9PIM6mRNASingle+Dual4123PPT1mRNASingle+Dual3421LIPA4mRNASingle+Dual+Triple3015PCPmRNASingle+Dual2420ASAH1mRNASingle+Dual2120 PSP summary table of reproducible cross-omics biomarkers Discussion In this study, we benchmarked 27 feature ranking methods, multiple classifiers, and three levels of omics integration across heterogeneous cohorts (AD, PSP, and BC). By jointly evaluating predictive performance and biological validity, we provide a comprehensive assessment of how methodological choices influence reproducibility and biomarker reliability in high-dimensional, low-sample omics settings. Our results show that model complexity does not necessarily translate into higher accuracy. Traditional classifiers—including L.Regression, SVMs, Random Forests, XGBoost, CatBoost, and MLPs—consistently matched or outperformed multiview deep learning models such as MORE and MOGONET. This trend was most pronounced in BC, where nearly all classifiers reached ceiling-level performance, demonstrating that well-regularized shallow models can fully capture strong molecular signals. Deep models showed lower median accuracy and greater variability, reflecting the difficulty of training high-parameter architectures with limited omics samples. Feature selection emerged as a key determinant of stability and performance. Ensemble aggregation methods—including Mean/Median Rank, Mean/Median Weight, Geometric Means, RRA, Stuart, and TA—consistently provided the most robust panels across cohorts, integration levels, and panel sizes. Several classical selectors (LASSO, Elastic Net, Ridge, and SHAP) also performed strongly, reinforcing their utility in omics pipelines. In contrast, Boruta, MORE-Ranker, and MOGONET-Ranker tended to produce less stable or less predictive panels. These findings highlight the advantages of consensus-based feature aggregation. Omics integration had a strong and systematic effect on performance. Dual-omics often provided the best trade-off between accuracy and generalization. Triple-omics achieved the highest maximum accuracy across selectors and classifiers, although gains depended on modality balance and cohort characteristics. AD benefitted strongly from miRNA–methylation combinations, PSP from mRNA–proteomics, and BC showed uniformly high accuracy regardless of integration level. These results highlight that multi-omics integration yields synergistic gains when modality quality is balanced. Biological validation revealed that integration concentrates true disease signal. Across HMDD, CTD, GeneCards, and EWAS-ATLAS, dual- and triple-omics consistently increased true-positive overlaps even though each modality contributed fewer features in fixed-size panels. This pattern suggests that cross-omics prioritization filters modality-specific noise and enriches shared biological mechanisms—an effect observed consistently across cutoffs . Cohort differences reflected biological and technical factors. BC achieved the highest TP counts, likely due to sample size and strong transcriptomic signals. AD showed the strongest miRNA recovery (HMDD), consistent with modality strengths. PSP showed moderate but consistent gains across integration levels. Nevertheless, the selector hierarchy and integration benefit remained stable across cohorts. Taken together, our results recommend multi-omics pipelines that integrate across molecular layers and employ ensemble-based feature aggregation. Mid-sized panels (30–60 features) paired with linear models, SVMs, or MLPs provided the best balance of accuracy, stability, and interpretability. Large panels () offered only modest improvements, and deep learning models rarely outperformed simpler baselines, suggesting that their use should be reserved for substantially larger datasets with independent validation. This study has limitations. Cohort sizes—particularly PSP—limit statistical power. Cohort heterogeneity complicates generalization, and technical differences (e.g. CpG mapping, platform variability) may influence rankings. Deep learning models may perform better in larger cohorts or with transfer learning. Beyond methodological benchmarking, this study also has implications for translational biomarker discovery. By systematically evaluating feature selection strategies across multiple omics modalities and disease cohorts, the proposed framework provides a practical guide for identifying robust candidate biomarkers that warrant further biological and clinical investigation. For example, several top-ranked features identified across cohorts correspond to molecules previously implicated in disease-related pathways, supporting the biological relevance of the selected signatures. While the biomarkers identified in this study require experimental validation and prospective clinical evaluation, the benchmarking framework helps prioritize candidates that demonstrate consistent predictive performance and biological support across datasets and external validation resources. In future work, integrating this framework with additional data modalities such as genomic variants—including gene mutation profiles widely available in resources such as TCGA—may further improve biomarker robustness and facilitate translation into clinically actionable diagnostic or prognostic panels. Future work should incorporate larger AD cohorts [Alzheimer’s Disease Neuroimaging Initiative (ADNI), Accelerating Medicines Partnership-Alzheimer’s Disease (AMP-AD)], broader multi-omics integration—including richer proteomic, metabolomic, and genomic variant data—and mechanistic validation of novel candidates identified here. Longitudinal prediction and treatment–response modeling represent promising next steps. To promote transparency, we provide two complementary resources. BioMark is an online platform for user-defined multi-omics analysis that supports dimensionality reduction, feature selection using single- and ensemble-rankers, and predictive modeling [45]. BioMark is available at https://bioinf.itu.edu.tr/biomark/. Furthermore, an interactive benchmark explorer is also provided to inspect results generated using the benchmarking framework presented in this study. The benchmark framework and explorer application are available at https://github.com/itu-bioinformatics-database-lab/biomarker-benchmark. Further details are provided in the Supplementary Information. Conclusion This study presents a comprehensive benchmarking framework for multi-omics biomarker discovery, addressing a critical question in bioinformatics and translational research: Does increasing methodological complexity yield better biomarkers? By systematically evaluating 27 feature selection strategies and 11 predictive models across three diverse disease cohorts—AD, PSP, and BC—we provide robust evidence that challenges the assumption that deep learning and complex integration methods inherently outperform traditional approaches. Our findings demonstrate that ensemble-based feature selection methods, particularly those leveraging rank and weight aggregation, consistently enhance the stability, reproducibility, and biological relevance of biomarker panels. Notably, traditional machine learning models, such as L.Regression, SVMs, and MLPs often matched or exceeded the performance of advanced deep learning frameworks like MORE and MOGONET, especially in settings with limited sample sizes and high-dimensional data. We show a monotonic trend: dual-omics outperform single-omics, and triple-omics outperform dual-omics—triple-omics generally yields the best overall performance, with gains most evident when data quality and sample size are adequate. Compact biomarker panels (30–60 features) derived from ensemble selectors and interpretable models achieve high predictive accuracy and facilitate biological interpretation. Biological validation against curated databases reinforces the clinical relevance of our approach. The reproducible identification of known and novel candidates underscores the utility of ensemble strategies. We provide a web-based interactive explorer for visualizing benchmarking results. In conclusion, our work advocates for a pragmatic and evidence-driven approach to multi-omics biomarker discovery—one that prioritizes robustness, interpretability, and biological validation over algorithmic complexity. We encourage researchers to critically assess the tradeoffs between model sophistication and practical utility, especially in the context of translational applications. Key Points Ensemble feature selection consistently yields more stable and accurate biomarker panels than individual rankers. Ensemble feature selection consistently yields more stable and accurate biomarker panels than individual rankers. Deep learning models (Multi-Omics hypergraph integration network, Multi-Omics Graph convolutional networks) do not outperform simpler classifiers such as logistic regression, Support Vector Machine, or Multilayer Perceptron. Deep learning models (Multi-Omics hypergraph integration network, Multi-Omics Graph convolutional networks) do not outperform simpler classifiers such as logistic regression, Support Vector Machine, or Multilayer Perceptron. Predictive performance improves with integration level (Triple-omics > Dual-omics > Single-omics). Predictive performance improves with integration level (Triple-omics > Dual-omics > Single-omics). External validation using Comparative Toxicogenomics Database, Human Gene Database, Human microRNA Disease Database, and the Epigenome-Wide Association Study Atlas confirms the biological and clinical relevance of the discovered biomarkers. External validation using Comparative Toxicogenomics Database, Human Gene Database, Human microRNA Disease Database, and the Epigenome-Wide Association Study Atlas confirms the biological and clinical relevance of the discovered biomarkers. The benchmarking pipeline promotes transparency, reproducibility, and practical applicability in multi-omics biomarker discovery. The benchmarking pipeline promotes transparency, reproducibility, and practical applicability in multi-omics biomarker discovery. Supplementary Material Contributor Information Cyrille Mesue Njume, Department of Molecular Biology and Genetics, Ayazaga Campus, Istanbul Technical University, Reşitpaşa, Sarıyer, 34467 Istanbul, Turkey. Irene Petracci, Molecular Markers Laboratory, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, 25125 Brescia, Italy. Sonia Bellini, Molecular Markers Laboratory, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, 25125 Brescia, Italy. Katarzyna Goljanek-Whysall, Discipline of Physiology, School of Medicine, University of Galway, H91 TH33 Galway, Ireland. Leo R Quinlan, Discipline of Physiology, School of Medicine, University of Galway, H91 TH33 Galway, Ireland. Agnieszka Fiszer, Department of Medical Biotechnology, Institute of Bioorganic Chemistry, Polish Academy of Sciences, Noskowskiego 12/14, 61-704 Poznan, Poland. Barbara Borroni, Molecular Markers Laboratory, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, 25125 Brescia, Italy. Roberta Ghidoni, Molecular Markers Laboratory, IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli, 25125 Brescia, Italy. Asli Kumbasar, Department of Molecular Biology and Genetics, Ayazaga Campus, Istanbul Technical University, Reşitpaşa, Sarıyer, 34467 Istanbul, Turkey. Ali Cakmak, Department of Computer Engineering, Ayazaga Campus, Istanbul Technical University, Reşitpaşa, Sarıyer, 34467 Istanbul, Turkey. Conflicts of interest None declared. Funding A. Çakmak and C. Mesue Njume were supported by the Scientific and Technological Research Council of Türkiye (TÜBıTAK) through the EU Joint Programme—Neurodegenerative Disease Research (JPND) (Grant No. 124N069), by the Scientific Research Projects Unit of Istanbul Technical University (ITU BAP) (Grant No. TGA-2025-46998), and by the National Center for High-Performance Computing (UHEM) (Grant No. 1009742021). This research was also funded by the Italian Ministry of Health—EU Joint Programme—Neurodegenerative Disease Research (JPND), “Large scale analysis of OMICS data for drug-target finding in neurodegenerative diseases”—ERP-2023-23684212—ERP-2023-JPND-MyRIAD; by the Polish National Science Centre within the same project [2023/05/Y/NZ3/00160]; and by the Health Research Board, JPND-2023-1: EU Joint Programme—Neurodegenerative Disease Research (JPND), “Large scale analysis of OMICS data for drug-target finding in neurodegenerative diseases.” Data availability The multi-omics datasets analyzed in this study are subject to data use agreements and cannot be publicly shared within this repository. Access to the raw data can be obtained directly from the respective repositories in accordance with their access policies. AD data, including mRNA, miRNA, and DNA methylation profiles from postmortem brain tissue, are available under controlled access through the Synapse AMP-AD Knowledge Portal (https://www.synapse.org/#!Synapse:syn3219045). PSP data, comprising transcriptomic, proteomic, and metabolomic measurements, are accessible through the AMP-AD Knowledge Portal (https://adknowledgeportal.synapse.org/) under study accession syn5550404. The associated metabolomics dataset, obtained from the same Mayo Clinic PSP samples, is available under the title “The Landscape of Metabolic Brain Alterations” (syn26401311). BC data, including gene expression, miRNA expression, and DNA methylation profiles, were obtained from the UCSC Xena Browser (https://xena.ucsc.edu/). Only processed and derived outputs—such as ranked feature lists, performance summaries, and validation statistics—are included in the GitHub repository to comply with institutional and repository data use policies. Code availability All code developed for this study is openly available at https://github.com/itu-bioinformatics-database-lab/biomarker-benchmark. The repository contains reproducible pipelines for data preprocessing, feature selection, model benchmarking, and validation analyses. Users can replicate all reported results by obtaining the datasets listed in the Data Availability section or apply the framework to their own multi-omics cohorts . Implementation note. The Python implementations of the Stuart rank aggregation and RRA algorithms were adapted from the RobustRankAggreg R package (version 1.2; Kolde & Laur) [46], released under the GPL-2 license. Algorithmic equivalence with the original R implementation was verified using benchmark gene lists. References Associated Data Supplementary Materials Data Availability Statement The multi-omics datasets analyzed in this study are subject to data use agreements and cannot be publicly shared within this repository. Access to the raw data can be obtained directly from the respective repositories in accordance with their access policies. AD data, including mRNA, miRNA, and DNA methylation profiles from postmortem brain tissue, are available under controlled access through the Synapse AMP-AD Knowledge Portal (https://www.synapse.org/#!Synapse:syn3219045). PSP data, comprising transcriptomic, proteomic, and metabolomic measurements, are accessible through the AMP-AD Knowledge Portal (https://adknowledgeportal.synapse.org/) under study accession syn5550404. The associated metabolomics dataset, obtained from the same Mayo Clinic PSP samples, is available under the title “The Landscape of Metabolic Brain Alterations” (syn26401311). BC data, including gene expression, miRNA expression, and DNA methylation profiles, were obtained from the UCSC Xena Browser (https://xena.ucsc.edu/). Only processed and derived outputs—such as ranked feature lists, performance summaries, and validation statistics—are included in the GitHub repository to comply with institutional and repository data use policies.