Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis
Alexander Partin ∗, Priyanka Vasanthakumari ∗, Oleksandr Narykov, Andreas Wilke, Natasha Koussa, Sara E. Jones, Yitan Zhu, Jamie C. Overbeek, Rajeev Jain, Gayara Demini Fernando, Cesar Sanchez-Villalobos, Cristina Garcia-Cardona, Jamaludin Mohd-Yusof, Nicholas Chia, Justin M. Wozniak, Souparno Ghosh · 2025
Read Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis by Alexander Partin ∗, Priyanka Vasanthakumari ∗, Oleksandr Narykov, Andreas Wilke, Natasha Koussa, Sara E. Jones, Yitan Zhu, Jamie C. Overbeek, Rajeev Jain, Gayara Demini Fernando, Cesar Sanchez-Villalobos, Cristina Garcia-Cardona, Jamaludin Mohd-Yusof, Nicholas Chia, Justin M. Wozniak, Souparno Ghosh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14356.txt.
Introduction
Alexander Partin and a large team of co-authors benchmark how well drug response prediction models generalize across datasets, in a two thousand twenty-five biomedical informatics paper. Models that shine on one cancer drug screen often disappoint on another.
Their framework standardizes five public drug screening datasets and six prediction models in a scalable workflow. New metrics capture both absolute accuracy on unseen data and the performance drop versus within-dataset testing. Drops turn out substantial, underlining the need for rigorous generalization checks. No model wins everywhere, though models trained on the CTRPv2 dataset generalized best. The shared framework aims to make future comparisons honest and reproducible.
Catalog record
| CREATOR | Alexander Partin ∗, Priyanka Vasanthakumari ∗, Oleksandr Narykov, Andreas Wilke, Natasha Koussa, Sara E. Jones, Yitan Zhu, Jamie C. Overbeek, Rajeev Jain, Gayara Demini Fernando, Cesar Sanchez-Villalobos, Cristina Garcia-Cardona, Jamaludin Mohd-Yusof, Nicholas Chia, Justin M. Wozniak, Souparno Ghosh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis Alexander Partin ∗, Priyanka Vasanthakumari ∗, Oleksandr Narykov, Andreas Wilke, Natasha Koussa, Sara E. Jones, Yitan Zhu, Jamie C. Overbeek, Rajeev Jain, Gayara Demini Fernando, Cesar Sanchez-Villalobos, Cristina Garcia-Cardona, Jamaludin Mohd-Yusof, Nicholas Chia, Justin M. Wozniak, Souparno Ghosh, Ranadip Pal, Thomas S. Brettin, M. Ryan Weil, Rick L. Stevens Abstract Deep learning (DL) and machine learning (ML) models have shown promise in drug response prediction (DRP), yet their ability to generalize across datasets remains an open question, raising concerns about their real-world applicability. Due to the lack of standardized benchmarking approaches, model evaluations and comparisons often rely on inconsistent datasets and evaluation criteria, making it difficult to assess true predictive capabilities. In this work, we introduce a benchmarking framework for evaluating cross-dataset prediction generalization in DRP models. Our framework incorporates five publicly available drug screening datasets, six standardized DRP models, and a scalable workflow for systematic evaluation. To assess model generalization, we introduce a set of evaluation metrics that quantify both absolute performance (e.g., predictive accuracy across datasets) and relative performance (e.g., performance drop compared to within-dataset results), enabling a more comprehensive assessment of model transferability. Our results reveal substantial performance drops when models are tested on unseen datasets, underscoring the importance of rigorous generalization assessments. While several models demonstrate relatively strong cross-dataset generalization, no single model consistently outperforms across all datasets. Furthermore, we identify CTRPv2 as the most effective source dataset for training, yielding higher generalization scores across target datasets. By sharing this standardized evaluation framework with the community, our study aims to establish a rigorous foundation for model comparison, and accelerate the development of robust DRP models for real-world applications. A Preprint August 24, 2026 * * footnotetext: These authors contributed equally to this work. Please cite as A. Partin and P. Vasanthakumari et al . Correspondence: [email protected] K eywords Cross-dataset generalization \cdot Cross-study analysis \cdot Model benchmarking \cdot Drug response prediction \cdot Precision oncology \cdot Multiomics \cdot Deep learning 1 Introduction Predictive modeling has become an integral tool in cancer research, offering new capabilities for improving clinical outcomes [ 1 , 2 ] . Anti-cancer drug response prediction (DRP) models provide an in-silico approach to evaluating the potential effects of drugs on cancer, leveraging artificial intelligence (AI) techniques such as deep learning (DL) and classical machine learning (ML) [ 3 , 4 , 5 ] . The increasing availability of high-throughput genomic profiling and drug screening data has enabled the rapid development of these models. However, despite the growing number of DRP models, it remains unclear how well their predictive performance translates to unseen datasets and experimental conditions, limiting their potential for broader real-world applicability [ 1 , 3 , 6 ] . While the ultimate goal is to accurately predict drug responses in patients, in-vitro cell line drug screenings remain the most abundant source of data and, therefore, the primary resource for developing and evaluating DRP models. However, while many published models achieve high predictive accuracy within a single cell line dataset, their performance deteriorates when applied to more complex biological systems such as organoids, patient-derived xenografts (PDX), or patients samples. Therefore, an intermediate step in complexity involves assessing prediction generalization across different cell line datasets, a practice that has recently become widely common [ 6 ] . Demonstrating robust cross-dataset generalization goes beyond simple cross-validation within a single dataset and positions a model as a promising candidate for more complex transfer tasks. Despite the growing number of DRP models, the lack of standardized frameworks for scalable benchmarking presents a major challenge in identifying the most effective methods for cancer treatment prediction. Performance evaluation is often conducted with simple baselines, varied dataset compositions, inconsistent data splits, and diverse scoring metrics, making it difficult to assess model strengths, weaknesses, and key factors contributing to model performance. To demonstrate tangible progress in the field and better understand model capabilities, DRP models should undergo systematic cross-comparisons against multiple other models, ideally demonstrating sufficient cross-dataset generalization, in a rigorous and systematic way. Recognizing this need, collaborative community efforts have emerged to establish guidelines and best practices for improving model development and evaluation [ 1 , 6 , 3 ] . To address the pressing need for standardized benchmarking in DRP, this study, conducted as part of IMPROVE [ 7 , 8 ] project, aims to establish a framework for systematic evaluation and comparison of DL models, applicable to various scientific domains. We present simple benchmarking principles centered around four key aspects: 1) benchmark datasets, 2) standardized models, 3) software tools and protocols, and 4) evaluation workflows. These principles, which form the core contributions of this study, are designed to facilitate systematic model benchmarking. Specifically, we constructed a comprehensive benchmark dataset (Section 2.1 ) comprising drug response data from five publicly available drug screening studies (Table 1 ), along with drug and omics feature representations and pre-computed data splits to ensure consistency across evaluations. We also designed a unified code structure promoting modular design and developed improvelib , a lightweight Python package that standardizes preprocessing, training, and evaluation, ensuring consistent model execution and enhancing reproducibility. Furthermore, we selected five DL-based DRP models and one ML model built with LightGBM [ 9 ] (Table 2 ) and adjusted their code to conform to the required modular structure. Finally, we implemented a scalable workflow to conduct large-scale cross-dataset analysis, applying it to the selected models and benchmark dataset for comprehensive training and evaluation. 2 Materials and Methods The development of DRP models, much like any supervised learning model, typically includes three core components: data preparation, model development, and performance analysis [ 3 ] . The specifics of each component are often contingent upon the application, performance evaluation scheme, required rigor, and scale of the analysis. Within this general framework, we outline the details of the analysis conducted in this work (Fig. 1 ). A benchmark dataset, the product of the data preparation step, is essential for fair analysis and a required ingredient for establishing state-of-the-art (SotA) among models. In the context of our analysis, which investigates DRP models, this dataset comprises drug response data compiled from multiple drug screening studies, omics and drug feature data, and pre-computed data splits to ensure consistent train, validation, and test sets. Model development, in this context, refers to an ML learning pipeline comprising data preprocessing, model training, and inference. These steps take the output of data preparation (benchmark data and potentially additional model-specific data), transform it into suitable model input data, train the model, and generate predictions on a hold-out set through inference. Finally, the evaluation scheme can encompass a wide array of approaches, ranging from basic cross-validation or hold-out set analysis on a single dataset to more complex schemes. These may include cross-dataset generalization across studies from the same [ 10 , 11 , 12 ] or different biological media [ 13 , 14 , 15 , 16 ] , data scaling analysis via learning curves [ 17 , 18 , 12 , 19 ] , and interpretability assessments [ 20 , 21 ] . In this work, a workflow for conducting scalable cross-dataset generalization analysis with the benchmark dataset and