Toward more realistic drug-target interaction predictions.

Pahikkala T, Airola A, Pietilä S, Shakyawar S, Szwajda A, Tang J, Aittokallio T. · 2015

Read Toward more realistic drug-target interaction predictions. by Pahikkala T, Airola A, Pietilä S, Shakyawar S, Szwajda A, Tang J, Aittokallio T. (2015) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4364066.txt.

Introduction

Tapio Pahikkala and co-authors warned that drug-target forecasts often look too good. Their article ran in Briefings in Bioinformatics in 2015. Computer models can predict which drugs will act on which proteins. Such forecasts could help find new uses for old drugs. Yet many studies test these models in ways that are too simple. Real use is far messier. The team showed how much those choices inflate the scores.

Four choices mattered most. One was treating the task as yes-or-no sorting. Graded prediction is closer to real life. A second was which drug and protein families filled the test set. A third was how the test loop was set up. A fourth was whether test and training sets shared drugs or targets. Each choice could shift results a lot. The authors give fair test rules for future work.

Catalog record

CREATORPahikkala T, Airola A, Pietilä S, Shakyawar S, Szwajda A, Tang J, Aittokallio T.
YEAR2015
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC4364066

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward more realistic drug–target interaction predictions By Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang, Tero Aittokallio Published in: Briefings in Bioinformatics DOI: 10.1093/bib/bbu010 ABSTRACT A number of supervised machine learning models have recently been introduced for the prediction of drug–target interactions based on chemical structure and genomic sequence information. Although these models could offer improved means for many network pharmacology applications, such as repositioning of drugs for new therapeutic uses, the prediction models are often being constructed and evaluated under overly simplified settings that do not reflect the real-life problem in practical applications. Using quantitative drug–target bioactivity assays for kinase inhibitors, as well as a popular benchmarking data set of binary drug–target interactions for enzyme, ion channel, nuclear receptor and G protein-coupled receptor targets, we illustrate here the effects of four factors that may lead to dramatic differences in the prediction results: (i) problem formulation (standard binary classification or more realistic regression formulation), (ii) evaluation data set (drug and target families in the application use case), (iii) evaluation procedure (simple or nested cross-validation) and (iv) experimental setting (whether training and test sets share common drugs and targets, only drugs or targets or neither). Each of these factors should be taken into consideration to avoid reporting overoptimistic drug–target interaction prediction results. We also suggest guidelines on how to make the supervised drug–target interaction prediction studies more realistic in terms of such model formulations and evaluation setups that better address the inherent complexity of the prediction task in the practical applications, as well as novel benchmarking data sets that capture the continuous nature of the drug–target interactions for kinase inhibitors. INTRODUCTION System-level understanding of the relationships between chemical compounds and their potential cellular targets is an important prerequisite for a rational drug development process. Chemical–protein interactions provide insights into the mode of action and potential side effects of the selected lead compounds in phenotype-based drug testing as well as facilitate choosing those compounds that selectively target a particular protein in the target-based drug discovery. Because experimental mapping of the compound–target interaction networks remains limited both in coverage and throughput, a wide spectrum of in silico approaches have been developed for systematic prioritization and speeding up the experimental work by means of computational prediction of the most potent drug–target interactions, using various ligand- and/or structure-based approaches, such as those that relate compounds and proteins through quantitative structure activity relationships (QSARs), pharmacophore modeling, chemogenomic relationships or molecular docking [1–6]. In particular, supervised machine learning methods have the potential to effectively learn and make use of both structural similarities among the compounds as well as genomic similarities among their potential target proteins, when making predictions for novel drug–target interactions (for recent reviews, see [7, 8]). Such computational approaches could provide systematic means, for instance, toward streamlining drug repositioning strategies for predicting new therapeutic targets for existing drugs through network pharmacology approaches [9–12]. Compound–target interaction is not a simple binary on-off relationship, but it depends on several factors, such as the concentrations of the two molecules and their intermolecular interactions. The interaction affinity between a ligand molecule (e.g. drug compound) and a target molecule (e.g. receptor or protein kinase) reflects how tightly the ligand binds to a particular target, quantified using measures such as the dissociation constant (Kd) or inhibition constant (Ki). Such bioactivity assays provide a convenient means to quantify the full spectrum of reactivity of the chemical compounds across their potential target space. However, most supervised machine learning prediction models treat the drug–target interaction prediction as a binary classification problem (i.e. interaction or no interaction). To demonstrate improved prediction performance, most authors have used common evaluation data sets, typically the ‘gold standard’ drug–target links collected for enzymes (E), ion channels (ICs), nuclear receptor (NR) and G protein-coupled receptor (GPCR) targets from public databases, including KEGG, BRITE, BRENDA, SuperTarget and DrugBank, first introduced by Yamanishi et al. [13]. Although convenient for cross-comparing different machine learning models, a limitation of these databases is that they contain only true-positive interactions detected under various experimental settings. Such unary data sets also ignore many important aspects of the drug–target interactions, including their dose-dependence and quantitative affinities. Moreover, the prediction formulations have conventionally been based on the practically unrealistic assumption that one has full information about the space of targets and drugs when constructing the models and evaluating their predictive accuracy. In particular, model evaluation is typically done using leave-one-out cross-validation (LOO-CV), which assumes that the drug–target pairs to be predicted are randomly scattered in the known drug–target interaction matrix. However, in the context of paired input problems, such as prediction of protein–protein or drug–target interactions, one should in practice consider separately the settings where the training and test sets share common drugs or proteins [8, 14–16]. For example, the recent study by van Laarhoven et al. [17] showed that a regularized least-squares (RLS) model was able to predict binary drug–target interactions at almost perfect prediction accuracies when evaluated using a simple LOO-CV. Although RLS has proven to be an effective model in many applications [18, 19], we argue that a part of this superior predictive power can be attributed to the oversimplified formulation of the drug–target prediction problem, as well as unrealistic evaluation of the model performance. Another source of potential bias is that simple cross-validation (CV) cannot evaluate the effect of adjusting the model parameters, and may therefore easily lead to selection bias and overoptimistic prediction results [20–22]. Nested CV has been proposed as a solution to provide more realistic performance estimates in the context of drug–target prediction or other feature selection applications [8, 23]. Here, we illustrate that a more realistic formulation of the drug–target prediction problem may lead to drastically decreased prediction accuracies, better reflecting the true complexity of the drug–target prediction problem in practical applications. Although the van Laarhoven et al. study [17] is used as an example, we note that similar problem formulations and evaluation setups have been used in many recent studies that have introduced new models and showed improved prediction accuracies [14, 24–26]. Although these works have provided important insights into the performance of the supervised machine learning methods, we believe they fall short in demonstrating the realistic performance of the predictive models in practice. A particular contribution of the present work is to formulate the drug–target interaction prediction as a ranking problem, in contrast to the standard binary classification. In comparison to the binary drug–target data sets by Yamanishi et al. [13], we use here two large-scale data sets from biochemical selectivity assays for clinically relevant kinase inhibitors by Davis et al. and Metz et al. [27, 28]. Rather than reporting only true-positive interactions, these systematic mappings of the quantitative Kd and Ki bioactivity spectra in standardized settings provide broader insights into the interaction patterns across wide panels kinase inhibitors and their potential cellular targets for model evaluation purposes. Protein kinases play important roles in a wide range of diseases, such as cardiovascular disorders and cancer; however, members of the same kinase family are relatively similar to each other, which leads to prevalent target promiscuity and polypharmacological effects—and a challenging drug–target prediction problem. MODELS AND METHODS Predictive models We used the same machine learning predictive model

Read the complete text