Variability in docking success rates due to dataset preparation.
Corbeil CR, Williams CI, Labute P. · 2012
Read Variability in docking success rates due to dataset preparation. by Corbeil CR, Williams CI, Labute P. (2012) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3397132.txt.
Introduction
Corbeil, Williams, and Labute warned in 2012 that how a test dataset is prepared can swing the reported success of drug-docking software. Docking programs try to fit candidate molecules into protein pockets. The team ran their MOE software on the standard Astex test set with a new scoring function.
For most targets the top-ranked fit landed close to the known X-ray structure, and nearly all targets had a good fit among the top thirty tries. The failures traced mostly to water molecules missing from the source data. A small energy-scale slip in preparation moved success rates by more than a fifth. Clean, complete datasets, they argue, matter as much as clever software.
Catalog record
| CREATOR | Corbeil CR, Williams CI, Labute P. |
|---|---|
| YEAR | 2012 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Variability in docking success rates due to dataset preparation By Christopher R Corbeil, Christopher I Williams, Paul Labute Published in: Journal of Computer-Aided Molecular Design DOI: 10.1007/s10822-012-9570-1 ABSTRACT Abstract The results of cognate docking with the prepared Astex dataset provided by the organizers of the “Docking and Scoring: A Review of Docking Programs” session at the 241st ACS national meeting are presented. The MOE software with the newly developed GBVI/WSA dG scoring function is used throughout the study. For 80 % of the Astex targets, the MOE docker produces a top-scoring pose within 2 Å of the X-ray structure. For 91 % of the targets a pose within 2 Å of the X-ray structure is produced in the top 30 poses. Docking failures, defined as cases where the top scoring pose is greater than 2 Å from the experimental structure, are shown to be largely due to the absence of bound waters in the source dataset, highlighting the need to include these and other crucial information in future standardized sets. Docking success is shown to depend heavily on data preparation. A “dataset preparation” error of 0.5 kcal/mol is shown to cause fluctuations of over 20 % in docking success rates. Keywords: Docking, Scoring, Errors, MOE, GBVI/WSA INTRODUCTION Docking methods have now been in existence for over 35 years, starting with Levinthal et al.’s use of docking to predict possible conformations of hemoglobin fibers [1]. Since then many docking programs have been developed [2], primarily for protein–ligand docking in the context of small-molecule structure-based drug discovery. While docking programs have become widespread, many issues remain unresolved such as the proper treatment of protein flexibility, solvation and ultimately, the accurate prediction of binding affinities [3–6]. To monitor improvements and the current status of the field, it has become popular to compare various docking methods with studies aimed at assessing the accuracy and limitations of the different programs and protocols. [7–10]. However, despite a large number of comparative studies, it still remains difficult to determine which programs and protocols result in overall performance improvements. Many studies have shown that docking success rates are heavily dependent on many variables, ranging from the scoring function being used [7], the target being investigated [8], the input for docking [9, 10], and even the metrics used to determine success in the study [11]. As a result, it can be a challenge to compare results from different validation studies, which often present contradictory conclusions. One major stumbling block to the advancement of protein–ligand docking validation has been the lack of a standard test set agreed upon and used by the entire community. The absence of such a set is one reason why it can be difficult to compare or even reproduce published docking results, because access to the primary data used in the computational experiments is often limited [12]. The need for validation test sets has been partially addressed in other computational chemistry fields through competitions such as CSAR (Community Structure–Activity Resource) for binding affinity prediction [13, 14], CASP (Critical Assessment of protein Structure Prediction) [15–23] for protein structure prediction and CAPRI (Critical Assessment of PRediction of Interactions) [24–27] for protein–protein docking. In these events the data is curated by the organization and given to the participants (blinded or not) who then asses how their methods perform. The use of standard community tests sets in these competitions makes direct comparison of validation studies straightforward. To overcome the lack of organized competitions and standardized test sets in protein–ligand docking, efforts have been made to publish datasets, such as Astex [28], and DUD [29], for use when conducting docking and/or binding affinity prediction experiments. However, even when docking studies use these published sets, it can be difficult to compare or reproduce results, because the researchers often significantly process and manipulate the data before using them as input for docking programs. The details of these manipulations are often subtle, and can have a profound effect on results; a small change in a hydroxyl rotamer in the binding pocket, or the inclusion or deletion of a bound water, can have huge effects on docking performance [30]. Unfortunately, exclusion of any of these small details from a methods section can make it difficult or impossible to reproduce published results. With no standing organization to produce standardized datasets and to run competitions for protein–ligand docking, it is left up to individuals to organize fair and un-biased events [8, 13, 31–33]. One such event occurred in the “Docking and Scoring: A Review of Docking Programs” session during the 241st ACS national meeting in which we participated. The ultimate goal of this event was to assess the current status of docking programs. The session consisted of using the Astex Diverse Set [28] for pose prediction and the DUD set [29] for virtual screening accuracy. In a standardization effort the organizers of the competition prepared the input data themselves, and asked the participants to use the structures “as-given”. This would hopefully remove biases associated with dataset preparation and evolve some standardized datasets. The organizers also defined how the results should be reported, to minimize difficulties in comparisons that arise from using different success metrics. This paper presents the results of our participation in this session and covers four major points of discussion: - Development of a new scoring function, GBVI/WSA dG. - The results of cognate docking with MOE using the ACS-Astex set input ‘as-given’ and after in-house manipulations. - Detailed analysis of docking failure cases, which point out errors and inconsistencies in source test datasets. - The effect of dataset preparation on docking error, and the effect of error on docking success rates. The MOE docking architecture provides a standardized docking workflow that divides docking into a series of protocols, each of which can be modified and adjusted independently of the others. Thus it provides a good starting platform to compare the effect of each aspect of the docking workflow on the final results. To this effect, we developed a new scoring function, which was easily plugged into the existing MOE architecture for this study. We elected to develop a new scoring function instead of using existing functions because (a) we wanted to test a scoring function developed and trained on data not in the ACS-Astex set, (thus removing the bias of a scoring function trained to recover the Astex crystallographic pose) and (b) we wanted a simple force field-based scoring function with fewer terms, which will have a smaller probability of being over-fitted compared to more complex scoring functions [34]. The results of cognate docking to the ACS-Astex set are presented, both using the data ‘as-given’, as suggested by the competition organizers and after in-house manipulations. The performance improvements along with details and justification of the in-house input manipulations will be discussed. Special attention is given to the docking failures, especially in cases where failure should be expected because of problems with the source data, such as bad contacts and incorrect chirality. We show that even the highly-curated ACS-Astex dataset used in this study has several problems, despite being of modest size and having been examined by experts in the field. This highlights the real technical and scientific difficulties in preparing protein–ligand validation test sets.