Benchmarking and Analysis of Protein Docking Performance in Rosetta v3.2 By Sidhartha Chaudhury, Monica Berrondo, Brian D Weitzner, Pravin Muthu, Hannah Bergman, Jeffrey J Gray, Vladimir N Uversky Published in: PLoS ONE DOI: 10.1371/journal.pone.0022477 ABSTRACT Abstract RosettaDock has been increasingly used in protein docking and design strategies in order to predict the structure of protein-protein interfaces. Here we test capabilities of RosettaDock 3.2, part of the newly developed Rosetta v3.2 modeling suite, against Docking Benchmark 3.0, and compare it with RosettaDock v2.3, the latest version of the previous Rosetta software package. The benchmark contains a diverse set of 116 docking targets including 22 antibody-antigen complexes, 33 enzyme-inhibitor complexes, and 60 ‘other’ complexes. These targets were further classified by expected docking difficulty into 84 rigid-body targets, 17 medium targets, and 14 difficult targets. We carried out local docking perturbations for each target, using the unbound structures when available, in both RosettaDock v2.3 and v3.2. Overall the performances of RosettaDock v2.3 and v3.2 were similar. RosettaDock v3.2 achieved 56 docking funnels, compared to 49 in v2.3. A breakdown of docking performance by protein complex type shows that RosettaDock v3.2 achieved docking funnels for 63% of antibody-antigen targets, 62% of enzyme-inhibitor targets, and 35% of ‘other’ targets. In terms of docking difficulty, RosettaDock v3.2 achieved funnels for 58% of rigid-body targets, 30% of medium targets, and 14% of difficult targets. For targets that failed, we carry out additional analyses to identify the cause of failure, which showed that binding-induced backbone conformation changes account for a majority of failures. We also present a bootstrap statistical analysis that quantifies the reliability of the stochastic docking results. Finally, we demonstrate the additional functionality available in RosettaDock v3.2 by incorporating small-molecules and non-protein co-factors in docking of a smaller target set. This study marks the most extensive benchmarking of the RosettaDock module to date and establishes a baseline for future research in protein interface modeling and structure prediction. INTRODUCTION The formation of highly specific protein complexes is a fundamental process in biology, and the structures of these complexes can yield deep insight into the mechanisms of protein function. Computational protein docking provides a means by which to predict the structure of protein-protein complexes from their unbound structures. Blind structure-prediction efforts, such as the Critical Assessment of Protein Interactions (CAPRI) [1], [2] have showcased a number of successful docking strategies using a range of methods from course-grained fast-Fourier transform approaches which identify surface complementarity between two partners [3], [4] to all-atom stochastic methods that can accommodate intricate protein conformational changes [5], [6]. In a number of CAPRI strategies, [3], [7], [8], [9], [10] as well as other protein docking studies [11], [12], the protein docking component of the Rosetta v2 software package, RosettaDock [13], has proved useful for a range of protein docking applications. RosettaDock was first introduced as a multi-scale Monte Carlo based docking algorithm that utilized a centroid-based coarse grain stage to quickly identify favorable docking poses and an all-atom refinement stage that simultaneously optimized rigid-body position and side-chain conformation. Since then RosettaDock has been modified to address the critical challenge in protein-protein docking: binding-induced backbone conformational changes. Wang et al. introduced explicit loop modeling and backbone minimization [6] while we added ensemble-based docking [14] and conformational move sets specific to antibody docking [15]. In that span, RosettaDock has been used for a wide range of applications from antibody-antigen docking [11], [12], to peptide docking and specificity [16], [17] to multi-body [18] and symmetric docking.[19] The current version of Rosetta, v3.2, has been in development for the past two years. The original Rosetta software package was written primarily for ab initio protein folding [20] but quickly expanded to include an array of molecular modeling applications from protein docking to enzyme design. The new Rosetta software package [21] was written from the ground up with these diverse applications in mind. Essential components such as energy function calculators, protein structure objects, and chemical parameters were assembled into common software layers accessible to all protocols. Protocols such as side-chain packing, or energy minimization, were written with a modular object-oriented architecture that allows users and programmers to easily combine different molecular modeling objects and functions. Control objects were written to give users a generalized scheme from which to precisely specify the sampling strategy for a given protocol. Finally, user interfaces such as RosettaScripts,[22] PyRosetta [23], and a PyMol interface [24] were developed to provide unprecedented accessibility of the code. The protein docking component of Rosetta v3.2, was written with two main goals. The first goal was to include all the core docking capabilities of Rosetta v2.3. The second, take advantage of the modular Rosetta v3.2 architecture to easily include new features such as modeling small-molecules, [25] noncanonical amino acids, and post-translational modifications, adding more customized conformational constraints, or allowing for alternative side-chain packing or design schemes. In order to systematically evaluate docking performance, we ran both RosettaDock v2.3 and RosettaDock v3.2 against the recently expanded Protein Docking Benchmark 3.0 [26]. The results of this benchmark can determine whether RosettaDock v3.2 successfully reproduces or improves upon the results of RosettaDock v2.3. More importantly, benchmarking identifies the strengths and weakness of the core RosettaDock algorithm against a large diverse set of targets to guide future development. Finally, in order to showcase the additional capabilities of the Rosetta v3 software package, we identified a subset of targets in the benchmark that contain small-molecule co-factors in or near the binding site. Although these co-factors are critical to biological protein function and interactions, due to their non-protein nature, they are often excluded from many docking algorithms, including Rosetta v2.3. We utilize the small-molecule modeling components of Rosetta v3.2 to incorporate these co-factors in the docking process to test whether performance would improve. METHODS Overview of the RosettaDock algorithm RosettaDock is a Monte Carlo (MC) based multi-scale docking algorithm that incorporates both a low-resolution, centroid-mode, coarse-grain stage and a high-resolution, all-atom refinement stage that optimizes both rigid-body orientation and side-chain conformation. The algorithm, illustrated in Figure 1, roughly follows the biophysical theory of an encounter complex formation followed by a transition to a bound state. Typically the algorithm starts from either a random initial orientation of the two partners (global docking), or an initial orientation that is randomly perturbed from a user-defined starting pose (local perturbation). From there, the partner proteins are represented coarsely, where side chains are replaced by a single unified pseudo-atom, or centroid. During this phase, a 500-step Monte Carlo search is made with adaptive rotation and translational steps adjusted dynamically to achieve an acceptance rate of 25%. The score function used in this stage primarily consists of a ‘bump’ term, a contact term, and docking-specific statistical residue environment and residue-residue pair-wise potentials (Table 1) [13]. [Figure: The RosettaDock algorithm.RosettaDock is a multi-scale Monte-Carlo based algorithm that roughly models encounter complex formation followed by a transition to a bound state.] [Table: RosettaDock scoring components and weights.] Once the centroid-mode stage is complete, the lowest energy structure accessed during that stage is selected for high-resolution refinement. During high-resolution refinement, centroid pseudo-atoms are replaced with the side-chain atoms at their initial unbound conformations. Then 50 MC steps are made in which the (1) rigid-body position is perturbed by a random direction and magnitude specified by a Gaussian distribution around 0.1 Å and 3.0°, (2) the rigid-body orientation is energy-minimized, and (3) side-chain conformations are optimized with RotamerTrials [27], followed by a test of the Metropolis criteria. Every eight steps, an additional combinatorial side-chain optimization is carried out using the full side-chain packing algorithm, followed by an additional Metropolis criteria check. To reduce the time devoted to the computationally expensive energy-minimization for unproductive rigid-body moves, minimization is skipped if a rigid-body move results in a change in score of greater than +15. The all-atom score function used in this stage primarily consists of Van der Waals attractive and repulsive terms, a solvation term, an explicit hydrogen bonding term, a statistical residue-residue pair-wise interaction term, an internal side-chain conformational energy term, and an electrostatic term (Table 1) [13]. For particular targets, a variety of RosettaDock sampling strategies are often used to improve the chance of achieving an accurate structure prediction [28]. If no prior structural or biochemical information is known about the protein interaction of interest, global docking is used to randomize the initial docking poses. From there, score filters and clustering are used to identify clusters of acceptable low-energy structures for further docking and refinement. In most cases, there is some known information about the complex, either in the form of related protein complexes or in biochemical or bioinformatics data which identify probable regions of interaction on the protein partners. In these cases users manually arrange the starting docking pose to a configuration that is compatible with the information and carry out a local docking perturbation. Additionally, users can set distance-based filters that bias sampling towards those docking poses that are compatible with specified constraints [28]. If backbone conformational changes are anticipated, appropriate backbone sampling strategies are prescribed [6], [8], [14], [15]. Additional capabilities of Rosetta v3.2 Rosetta v3.2 represents a complete bottom-up re-implementation of the Rosetta software. The protein docking module of Rosetta v3.2 was intended to reproduce the core docking functionality of RosettaDock v2.3, and it can be used in conjunction with a number of new Rosetta v3.2 capabilities both through changes in source code and command-line options. These capabilities include automated parameterization of non-protein moieties such as small molecules [25] as well as non-canonical amino acids and post-translational modifications, precise control over the degrees of freedom available during conformational search using the pose, fold tree, and movemap functionalities,[6] expanded side-chain packing options for side-chain optimization and design, and control over sampling and decoy generation through constraints or filters. Finally, a number of algorithmic improvements in sampling and score calculations have led to an overall speed-up. For more information on the capabilities of Rosetta v3.2, see Leaver-Fay et al.[21] Implementation of docking in Rosetta v3.2 The RosettaDock code was restructured in Rosetta v3.2 with two goals: first, to allow for easier use of built-in Rosetta functionality, such as constraints or ligand modeling, and second to give developers greater flexibility when developing their own protocols that use docking functions. Figure 2 shows a diagram of the structure of the major classes associated with docking. Docking has been split into three major classes: DockingProtocol, DockingLowRes and DockingHighRes. DockingProtocol is responsible for handling user-specified docking-options, appropriately configuring various objected associated with docking, and applying DockingLowRes and DockingHighRes objects. [Figure: Structure of major classes associated with docking.A shaded diamond indicates composition (the object the diamond points towards is responsible for the lifecycle of the other object); an open diamond indicates aggregation (the object the diamond points towards has an instance of the other object but it may not be solely responsible for that instance's lifecycle); and an open triangle indicates a class hierarchy with the triangle pointing towards the parent class.] DockingLowRes and DockingHighRes contain all the data and functions associated with the low-resolution docking and high-resolution refinement stages, respectively, including the score functions, sampling functions (including translation/rotation parameters and side-chain packing), and Monte Carlo data. Both objects are independent of the Rosetta options system and can be called directly within the Rosetta source code or through Rosetta interfaces such as PyRosetta[23] and RosettaScripts[22]. Given the wide range of minimization and side-chain packing strategies that might be utilized in the high-resolution docking stage, DockingHighRes is designed as an abstract class that underlies a diverse set of high-resolution docking functions including standard high-resolution docking, pre-packing, as well as extensions of docking such as peptide docking and protein interface design. This versatility is achieved through the DockTaskFactory class within DockingHighRes, which handles all docking side-chain packing options and allows subclasses of DockingHighRes to be able to create a tailored set of packing instructions (Figure 2). All docking objects contain default parameters that allow them to be run with minimal setup; users only need to specify docking parameters for non-default behavior. Benchmarking Rosetta with Protein Docking Benchmark 3.0 The expanded Protein Docking Benchmark 3.0, curated by Huang et al., [26] is a non-redundant set of bound protein complex structures and their respective unbound structures from the Protein Data Bank.[29] There are 35 complexes classified as ‘enzyme-inhibitor’ (E/I), 25 classified as ‘antibody-antigen’ (Ab/Ag), and 64 classified as ‘other’ (O). According to docking difficulty criteria described by Mintseris et al., 88 are ‘rigid-body’ targets, 19 are ‘medium’ targets, and 17 are ‘difficult’ targets. [30] We applied RosettaDock v2.3 and RosettaDock v3.2 to the entire benchmark set. To prepare the structures for docking, the unbound structures were superimposed over the bound complex and the resulting superposed structure was used as the starting structure for local docking. For consistency, all chain identifiers were switched to that of the bound structure and all hetero-atoms were denoted with a chain identifier of ‘X’. In RosettaDock v2.3 the docking partners are defined by placing a ‘TER’ in the appropriate position in the starting structure PDB file. In RosettaDock v3.2, the docking partners are identified by chain identifier in the command-line. We first prepared each docking partner in isolation, optimizing their side-chain conformations prior to docking (“pre-packing”),[13] and then carried out local docking perturbations [13] using both RosettaDock v2.3 and RosettaDock v3.2 to generate 1000 decoys, or candidate structures, from each method. As described in [14], we set the docking perturbation parameter to 3 Å translation and 8° rotation. For side-chain packing, extra rotamers were used for χ1 for all residues and for χ2 for aromatic residues, and unbound rotamers were included as well [27]. We assessed docking performance by sorting these respective decoy sets by interface energy [6]. We ran all docking simulations on our local cluster; creation of each decoy required, on average, 2.5 minutes in RosettaDock v2.3 and 0.8 minutes on RosettaDock v3.2. The RosettaDock v3.2 command line used for pre-packing and docking structures in this benchmark is shown below. The inputs for docking include the pre-packed starting structure ($name.prepack.pdb), the original starting structure for loading unbound rotamer conformations ($name.unboundrot.pdb), the chain designations for the first and second partner ($chains1 and $chains2), and the desired number of decoys ($nstruct). The starting structure was the unbound components superimposed on the bound complex structure. The output includes all decoys in PDB format, and a score file that includes the total energy (total score), interface energy (I_sc) for each decoy, as well as docking metrics to the starting structure, including Lrmsd (rms), Irmsd (Irms), and f nat (Fnat). #for pre-packing docking_protocol.linxgccrelease –database $database_path / -s $name.pdb / -partners $chains1_$chains2 / -dock_ppk / -ex1 -ex2aro -unboundrot $name.unboundrot.pdb #for docking docking_protocol.linxgccrelease –database $database_path / -s $name.prepack.pdb / –nstruct $nstruct / -partners $chains1_$chains2 / -dock_pert 3 8 –spin / -ex1 -ex2aro -use_input_sc -unboundrot $name.unboundrot.pdb #for refinement of native complexes docking_protocol.linxgccrelease –database $database_path / -s $name.prepack.pdb / –nstruct $nstruct / -partners $chains1_$chains2 / -docking_local_refine / -ex1 -ex2aro -use_input_sc -unboundrot $name.unboundrot.pdb For analysis and classification of docking failures we refined native crystal co-complexes using the docking flag –docking_local_refine which carries out only the high-resolution refinement stage of docking. When incorporating non-protein moieties such as ligands or co-factors, additional parameters are input using the flags –extra_res_fa and –extra_res_cen to load the full-atom and centroid mode ligand connectivity and atom typing. We used the script molfile_to_params.pl packaged with Rosetta to generate the ligand parameters.[25] Metrics for structural accuracy and docking performance A number of measurements of structural accuracy are regularly used to measure docking performance, as defined by the CAPRI evaluators.[31] L_rmsd is the root mean squared deviation (RMSD) of the Cα atoms of the smaller partner in the complex (ligand) to its coordinates in bound structure after superposition of the larger partner in the complex (receptor). I_rmsd is defined as the RMSD of the heavy atoms in the interface residues after superposition of those same residues where the interface is defined as all residues with an intermolecular distance of at most 4.0 Å. Finally, the fraction of native contacts (f nat) is defined as the fraction of residue-residue contacts in the bound structure that are recovered in a given decoy, where a contact is defined by any two residues with any pair of atoms that are within 5.0 Å. L_rmsd describes the overall ligand-receptor position, I_rmsd describes the atomic accuracy of the interface between the two partners, and f nat describes the degree to which specific residue-residue interactions across the interface are recovered. As a qualitative description of accuracy, we use the term ‘near-native’ to refer to a decoy with I_rmsd of at most 4.0 Å. The presence of a ‘docking funnel’, in which near-native decoys consistently have better scores than non-native decoys, is considered to be the most robust measure of success in a docking simulation. For each target we count the number of near-native decoys among the top five scorers (N5) and classify a docking result as having a funnel if it has N5≥3. Classification of successes and failures Analyzing docking successes and failures is critical to understanding the strengths and weaknesses of the docking algorithm. Typically structure prediction algorithms are evaluated on the basis of sampling, discrimination, and prediction accuracy. We classified our docking results as successes or failures based on whether they achieved a docking funnel. The quality of a docking success was classified according to the accuracy of the closest decoy in the top five scoring decoys as high, medium, and acceptable quality according to CAPRI-defined criteria.[32] Generally, a decoy with I_rmsd<1.0 Å was considered high quality, 1.0 Å