MotifBench: A standardized protein design benchmark for motif-scaffolding problems

Zhuoqi Zheng, Bo Zhang, Kieran Didi, Kevin K. Yang, Jason Yim, Joseph L. Watson, Hai-Feng Chen, Brian L. Trippe · 2025

Read MotifBench: A standardized protein design benchmark for motif-scaffolding problems by Zhuoqi Zheng, Bo Zhang, Kieran Didi, Kevin K. Yang, Jason Yim, Joseph L. Watson, Hai-Feng Chen, Brian L. Trippe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12479.txt.

Introduction

Zhuoqi Zheng and co-authors present a 2025 paper on a standard benchmark for computational protein design. One core task is motif scaffolding: given a small piece of protein geometry that does a chemical job, find full protein backbones that hold it in place. Recent methods improved fast, but each team graded its work differently.

Their benchmark, MotifBench, fixes one pipeline and one set of metrics for everyone. It ships with thirty design problems and a public leaderboard. The new cases are harder than earlier sets. Some have known solutions that today's best methods still fail to find, in the authors' checks. A shared yardstick should make progress in protein design easier to trust and reproduce.

Catalog record

CREATORZhuoqi Zheng, Bo Zhang, Kieran Didi, Kevin K. Yang, Jason Yim, Joseph L. Watson, Hai-Feng Chen, Brian L. Trippe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12479

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MotifBench: A standardized protein design benchmark for motif-scaffolding problems Zhuoqi Zheng, Bo Zhang, Kieran Didi, Kevin K. Yang, Jason Yim, Joseph L. Watson, Hai-Feng Chen, Brian L. Trippe Abstract The motif-scaffolding problem is a central task in computational protein design: Given the coordinates of atoms in a geometry chosen to confer a desired biochemical function (a motif), the task is to identify diverse protein structures (scaffolds) that include the motif and maintain its geometry. Significant recent progress on motif-scaffolding has been made due to computational evaluation with reliable protein structure prediction and fixed-backbone sequence design methods [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] . However, significant variability in evaluation strategies across publications has hindered comparability of results, challenged reproducibility, and impeded robust progress. In response we introduce MotifBench, comprising (1) a precisely specified pipeline and evaluation metrics, (2) a collection of 30 benchmark problems, and (3) an implementation of this benchmark and leaderboard at github.com/blt2114/MotifBench . The MotifBench test cases are more difficult compared to earlier benchmarks (e.g. [ 4 ] ), and include protein design problems for which solutions are known but on which, to the best of our knowledge, state-of-the-art methods fail to identify any solution. 1 The motif-scaffolding problem: task specification and evaluation metrics A motif-scaffolding method takes a motif as input and returns a set of putatively compatible scaffolds as output. This section details how motifs and scaffolds in MotifBench are specified, proposes metrics by which a scaffold set is evaluated, and describes how these metrics are computed. Appendix A describes considerations upon which these specifications and metrics were chosen. Motif specification (inputs): A motif is specified by the coordinates of the backbone atoms of several residues and (in some cases) the amino acid types of a subset of those residues. • Motif atom coordinates are extracted from experimental structures deposited in the Protein Data Bank (PDB) [ 18 ] . • Each problem comprises the N, C \alpha , and C backbone coordinates of several residues. • The residues may be part of a single sequence-contiguous segment, or they may be divided across multiple segments. • When the amino acid types at particular positions are thought to be important to the biochemical function of the motif, these types and positions are specified and may not be modified in designed scaffolds. Side-chain heavy atom coordinates for these positions are also provided as optional inputs. Scaffold set specification (outputs): For each motif, the motif-scaffolding method outputs a collection of 100 scaffolds , each specified through the coordinates of backbone atoms (including those in the motif) of a single protein chain. • Coordinates for C \alpha atoms must be given, and additional backbone atoms (N, C and O) may be specified as well. • All scaffolds must contain the same number of residues. This fixed overall length of scaffolds is part of the problem specification. • Each scaffold is accompanied by metadata specifying the placement of the motif segment(s) in the scaffold sequence. These placements may be chosen in any way (e.g. to match the order and spacing in the experimental structures, algorithmically, or manually). Evaluation: Figure 1: MotifBench evaluation pipeline. Three metrics comprise the evaluation of each scaffold set: the number of unique solutions , the novelty of solutions, and overall success rate . The “number of unique solutions” counts the number of substantively distinct scaffolds in the set that are predicted to maintain the geometry of the motif within an experimentally realizable backbone. This metric emphasizes that a method for the motif-scaffolding problem should ideally provide a variety of solutions; protein design problems often have additional difficult-to-specify constraints and so benefit from further computational filtering and experimental screening. This metric is computed as follows: 1. For each scaffold, eight amino acid sequences are generated using a fixed backbone sequence design method. MotifBench uses ProteinMPNN [ 19 ] as default. If not all backbone atoms (N, C \alpha , C and O) are specified, then C \alpha -only ProteinMPNN is used. 2. For each generated sequence, a backbone structure is predicted using ESMFold [ 20 ] . 3. A scaffold backbone is a “success” if at least one of the eight sequences satisfies both of the following creteria: • Motif maintenance: The root mean squared distance (RMSD) between the backbone atoms (N, C \alpha , and C) of the input motif and corresponding atoms of the predicted structure (the motifRMSD ) is at most 1.0 Angstrom (Å); • Scaffold validity: The RMSD between corresponding backbone atoms (C \alpha only) of the generated and predicted structures (the self-consistency RMSD , or scRMSD [ 3 ] ) is at most 2.0 Å. For each of the above steps, the Kabsch algorithm [ 21 ] is first used to align structures so that the minimum RMSD across all possible alignments is returned. 4. The “number of unique solutions” is the number of clusters into which “successful” backbones are assigned by Foldseek-Cluster [ 22 ] . Secondarily, a scaffold set is evaluated by two additional metrics: • “Novelty” quantifies the typical distance of solutions to any structure in the PDB [ 18 ] . To compute the novelty, for each success we use FoldSeek-Search [ 23 ] to approximate the highest TM-score [ 24 ] to any structure in the PDB. Then for each cluster of successes we compute the mean of one minus this TMscore within the cluster. The “Novelty” is the mean of this score across clusters, or 0 if there are no successful scaffolds. • The “Success rate” is the fraction of scaffolds in the set that are successes, independent of their diversity and novelty. The MotifBench score for ranking performance across a set of problems. To enable ranking the relative ability of a motif-scaffolding method to provide unique solutions across a collection of test cases we introduce the “MotifBench score”. The MotifBench score has the form \textrm{MotifBench score}=\frac{1}{\#\mathrm{\ test\ cases}}\sum_{i=1}^{\#\mathrm{\ test\ cases}}(100+\alpha)\frac{\#\mathrm{\ unique\ solutions\ for\ case\ }i}{\alpha+\#\mathrm{\ unique\ solutions\ for\ case\ }i}\ ,\quad\mathrm{with}\ \alpha=5. We choose the form of this score to capture that the marginal value of an additional solution for a given problem is much larger when the number of unique solutions is low. If there were just one motif, one could rank on the number of unique solutions. However, simply averaging this across misses that the marginal value of an additional solution is much larger when the number of solutions is low. We note two properties of this metric: • It ranges from 0 to 100; the score is 0 if no solutions are found and 100 every backbone is a unique solution for every problem. • The choice of \alpha=5 gives high weight to first solutions and increases much more slowly as the number of solutions grows larger. As examples, with 1, 5, or 50 solutions the MotifBench score is 17.5, 52.5 or 95.5, respectively. As a consequence of the second property, the MotifBench score reflects that it is preferable for a method to provide few solutions to more motifs rather than to provide more

Read the complete text