Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation
Chenbin Zhang, Zhiqiang Hu, Chuchu Jiang, Wen Chen, Jie Xu, Shaoting Zhang, MoleculeMind, Peking University, Shanghai AI Laboratory, SenseTime Research · 2025
Read Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation by Chenbin Zhang, Zhiqiang Hu, Chuchu Jiang, Wen Chen, Jie Xu, Shaoting Zhang, MoleculeMind, Peking University, Shanghai AI Laboratory, SenseTime Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09481.txt.
Introduction
Zhang and five co-authors question how drug-target strength models are scored in this twenty twenty-five paper. The usual test splits data at random, so test drugs often look a lot like the drugs the model trained on. That likeness can make scores look better than they would be on truly new drugs.
Inside, the team builds a new split method that can match any target spread of likeness the user wants. They test five known methods on four data sets and compare the new splits with older ones. The new splits fit the target spread much more closely. The work gives labs a fairer way to check models before real use.
Catalog record
| CREATOR | Chenbin Zhang, Zhiqiang Hu, Chuchu Jiang, Wen Chen, Jie Xu, Shaoting Zhang, MoleculeMind, Peking University, Shanghai AI Laboratory, SenseTime Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09481
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation Chenbin Zhang, Zhiqiang Hu, Chuchu Jiang, Wen Chen, Jie Xu, Shaoting Zhang, MoleculeMind, Peking University, Shanghai AI Laboratory, SenseTime Research Abstract Drug-target binding affinity prediction is a fundamental task for drug discovery. It has been extensively explored in literature and promising results are reported. However, in this paper, we demonstrate that the results may be misleading and cannot be well generalized to real practice. The core observation is that the canonical randomized split of a test set in conventional evaluation leaves the test set dominated by samples with high similarity to the training set. The performance of models is severely degraded on samples with lower similarity to the training set but the drawback is highly overlooked in current evaluation. As a result, the performance can hardly be trusted when the model meets low-similarity samples in real practice. To address this problem, we propose a framework of similarity aware evaluation in which a novel split methodology is proposed to adapt to any desired distribution. This is achieved by a formulation of optimization problems which are approximately and efficiently solved by gradient descent. We perform extensive experiments across five representative methods in four datasets for two typical target evaluations and compare them with various counterpart methods. Results demonstrate that the proposed split methodology can significantly better fit desired distributions and guide the development of models. Code is released at https://github.com/Amshoreline/SAE/tree/main . 1 Introduction Drug-target binding affinity (DTA) prediction is a fundamental and crucial task for drug discovery. It evaluates the effectiveness of drug candidates, or samples, and sees its application in a large-scale virtual screening where most ineffective candidates are filtered out to save experimental cost and time ( Chatterjee et al., 2023 ) . DTA is quantitatively measured by inhibition constant Ki, half maximal inhibitory concentration IC50, etc., which are all real-valued ( Monteiro et al., 2022 ) . The prediction performance is commonly evaluated by mean absolute error (MAE) and coefficient of determination (R 2 ). The task of DTA prediction has been extensively studied for decades ( Chen et al., 2018 ; Askr et al., 2023 ) . Related works can be categorized as structure-based, sequence-based, and similarity-based ( Wu et al., 2018 ; Chuang et al., 2020 ) . Structure-based methods rely on 3D structures of samples, target proteins, or their complexes. Although theoretically accessible to most comprehensive information following the dogma “structure determines function”, structure-based methods are limited by available 3D structures, especially experimentally verified structures, and also hindered in practice by poor time efficiency. In contrast, sequence-based and similarity-based methods are fast and do not set 3D structures as prerequisite ( Xu et al., 2017 ; Zhang et al., 2022 ) . Instead, they take as input residual sequences, Simplified Molecular-Input Line-Entry System (SMILES) sequences, fingerprint sequences, atom-bond graphs, or the derived pairwise similarities, which are easier to acquire with lower cost. Moreover, these sequences and similarities are readily processed by diversified sophisticated backbones including convolutional neural networks (CNNs) ( Öztürk et al., 2018 ; Li et al., 2019 ; Hu et al., 2023 ) , recurrent neural networks (RNNs) ( Karimi et al., 2019 ; Yuan et al., 2022 ) , graph neural networks (GNNs) ( Nguyen et al., 2021 ; Yang et al., 2022 ; Tang et al., 2022 ; Wang et al., 2022 ) and transformers ( Chithrananda et al., 2020 ; Zhao et al., 2022 ; Song et al., 2023 ; Jiang et al., 2023 ) , and enjoy the benefits of the development of deep learning techniques. As a result, sequence-based and similarity-based methods are shown to reach new high performance and are drawing increasing attention. Although promising results are reported, we find, surprisingly, that these results may be misleading. Take the task of IC50 prediction for target EGFR as an example, as shown in Figure 1 , we evaluate five state-of-the-art and representative methods and the best-performing one, SAM-DTA ( Hu et al., 2023 ) , achieves a MAE of 0.6012 and R 2 of 0.6505 for the whole test set. However, if we dive into the performance and divide the test set according to the similarity of the sample to the training set, we find a clear performance degradation for low-similarity samples: the MAE deteriorates to 1.2970 for samples with similarity less than 1/3 and R 2 to -0.6385. The gap is huge. Nevertheless, poor performance on low-similarity samples does not affect the whole performance since they only occupy a negligible proportion: only 16 samples with similarity less than 1/3 out of a total of 873 samples in the test set (Figure 1 ). In other words, the test set is dominated by high-similarity samples and performance for low-similarity samples is overwhelmed in current evaluations. We will show that the phenomenon exists across different similarity measures, performance metrics, datasets, and methods, and therefore it is general. Consequently, the evaluation will be misleading to practitioners, especially when the trained model meets low-similarity samples when used in real practice. We argue that the core of the problem lies in the canonical randomized split of the test set. Randomized split follows the assumption of independent identical distribution (I.I.D.), which is the foundation of most statistical learning theories. However, in drug discovery samples are not necessarily independent of each other: in practice, mutually similar variants are more likely to be tested together in high-throughput experiments, while at the same time, they have to avoid high similarity to approved drugs for intellectual property issues ( Harren et al., 2024 ) . Empirically, drugs developed at different times show significant distinction in their properties ( Sheridan et al., 2022 ) . As a result, practitioners would not always expect the samples they are going to test to follow the same distribution as historical samples. This in turn raises a request to model development that the test set should satisfy a desired distribution. For example, one may need a test set that is uniform at different similarity bins; others may ask the test set samples to be all limited within predefined similarity bounds, and so on ( Li & Yang, 2017 ; Simm et al., 2021 ; Luo et al., 2024 ; Tricarico et al., 2024 ) . We formulate the test set split with a desired distribution as a combination optimization problem. The problem is infeasible to solve for optimum due to efficiency issues. In this work, we address this challenge by relaxing it to a continuous optimization problem where samples are allowed to coexist in the training set and test set with a “probability” or weight. Further, the objective function contains non-differentiable operations including taking the maximum and counting in similarity bins, and are approximated in this work by differentiable counterparts. We will show that the degree of approximation is adjustable by introduced hyper-parameters. Next, the resulting optimization problem has no closed-form solution, and hence we have resorted to Lagrangian multipliers with a numerical method implemented by PyTorch and Cooper ( Gallego-Posada & Ramirez, 2022 ) . Finally, we analyze the continuously-valued solution and find the non-negligible approximation error induced by the relaxation. To this end, we introduce a regularization term