ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning

Ziwen Wang University of IllinoisUrbana-Champaign [email protected], Jiajun Fan University of IllinoisUrbana-Champaign [email protected] Ruihan GuoUniversity of IllinoisUrbana-Champaign [email protected] Thao NguyenUniversity of IllinoisUrbana-Champaign [email protected] Heng JiUniv · 2025

Read ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning by Ziwen Wang University of IllinoisUrbana-Champaign [email protected], Jiajun Fan University of IllinoisUrbana-Champaign [email protected] Ruihan GuoUniversity of IllinoisUrbana-Champaign [email protected] Thao NguyenUniversity of IllinoisUrbana-Champaign [email protected] Heng JiUniv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07459.txt.

Introduction

Wang and co-authors let a protein-design model improve itself through online trial and reward, in this 2025 paper. Design models trained on fixed data often miss real goals like stability. Checking designs with physics is too slow to steer learning live.

Their system, ProteinZero, scores its own output with a fast structure predictor plus a self-taught stability estimator. A diversity guard at the embedding level stops the model from collapsing onto a few safe sequences. On a standard benchmark it beat three leading design tools, cutting failure rates by roughly forty percent and pushing success above ninety percent.

Catalog record

CREATORZiwen Wang University of IllinoisUrbana-Champaign [email protected], Jiajun Fan University of IllinoisUrbana-Champaign [email protected] Ruihan GuoUniversity of IllinoisUrbana-Champaign [email protected] Thao NguyenUniversity of IllinoisUrbana-Champaign [email protected] Heng JiUniv
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learning Ziwen Wang University of IllinoisUrbana-Champaign [email protected], Jiajun Fan University of IllinoisUrbana-Champaign [email protected] Ruihan GuoUniversity of IllinoisUrbana-Champaign [email protected] Thao NguyenUniversity of IllinoisUrbana-Champaign [email protected] Heng JiUniversity of IllinoisUrbana-Champaign [email protected] Ge LiuUniversity of IllinoisUrbana-Champaign [email protected] Abstract Protein generative models have shown remarkable promise in protein design but still face limitations in success rate, due to the scarcity of high-quality protein datasets for supervised pretraining. We present ProteinZero, a novel framework that enables scalable, automated, and continuous self-improvement of the inverse folding model through online reinforcement learning. To achieve computationally tractable online feedback, we introduce efficient proxy reward models based on ESM-fold and a novel rapid ddG predictor that significantly accelerates evaluation speed. ProteinZero employs a general RL framework balancing multi-reward maximization, KL-divergence from a reference model, and a novel protein-embedding level diversity regularization that prevents mode collapse while promoting higher sequence diversity. Through extensive experiments, we demonstrate that ProteinZero substantially outperforms existing methods across every key metric in protein design, achieving significant improvements in structural accuracy, designability, thermodynamic stability, and sequence diversity. Most impressively, ProteinZero reduces design failure rates by approximately 36% - 48% compared to widely-used methods like ProteinMPNN, ESM-IF and InstructPLM, consistently achieving success rates exceeding 90% across diverse and complex protein folds. Notably, the entire RL run on CATH-4.3 can be done with a single 8 × \times GPU node in under 3 days, including reward computation. Our work establishes a new paradigm for protein design where models evolve continuously from their own generated outputs, opening new possibilities for exploring the vast protein design space. 1 Introduction Protein design and engineering represent one of the most promising frontiers in computational biology, with applications spanning from drug discovery to developing novel enzymes ( proteinmpnn ; esm_if ; StructTrans ) . A central challenge in this field is protein inverse folding—generating amino acid sequences that will fold into desired three-dimensional structures ( GVP_GNN ; zhang2005tm ) , which serves as the foundation for fixed backbone sequence design. This task is particularly crucial as protein backbone structure and side-chain conformation jointly determine functionalities such as binding and catalytic interactions. Recent deep learning approaches such as ProteinMPNN ( proteinmpnn ) , ESM-IF ( esm_if ) , and graph-based methods ( GVP_GNN ; StructTrans ) have significantly improved inverse folding accuracy. However, current state-of-the-art methods are trained with paired sequence-structure data from the Protein Data Bank (PDB) which, while valuable, represent only a tiny fraction of the vast protein sequence space ( proteinmpnn ; esm_if ; instruct_plm ) and are subject to limited diversity and natural biases. This data scarcity creates a ceiling for model performance and limits the exploration of novel protein designs beyond what exists in nature or laboratory settings ( offline_3 ; ai_collapse_nature ) . Moreover, there is a misalignment between the supervised learning task of inverse folding and the actual objectives in real-world protein design, where high designability, thermal stability, and diversity are desired ( rfdiffusion ; chroma ) . There is limited effort in addressing this alignment task, where existing work has primarily focused on RL-finetuning structural generative models ( Multiflow ; foldflow2 ; abdpo ; capempnn ; iclr_dpo_fail ) and has only managed to do single- or few-round alignment with carefully curated offline designability datasets, failing to unlock the potential in self-evolving protein design models with their own generated outputs. We propose ProteinZero, a highly scalable online RL finetuning framework that, for the first time, enables fully automated, multi-round and stable self-improvement of the inverse folding model, and effectively optimizes multiple key objectives in protein sequence design, including designability, thermal stability, and sequence diversity while improving the inverse folding accuracy. The contributions of our work are as follows: 1. We develop ProteinZero, the first successful and practical online reinforcement learning framework for automated self-improvement of protein sequence design models without reliance on carefully curated preference datasets. 2. We propose a fast thermodynamic stability ddG estimator based on inverse folding sequence likelihoods and unconditional sequence priors. Combining it with lightweight designability rewards provided by ESMFold, we rapidly reduce the evaluation cost (Table 1 ), making online RL fine-tuning feasible for protein design for the first time. 3. We develop a novel diversity-promoting regularizer based on protein embeddings that effectively prevents mode collapse ( model_collapse ; ai_collapse_nature ; collapse2 ; collapse3 ) and improves the generated sequence diversity, a desired property for protein design. 4. We elucidate the design space of online RL fine-tuning frameworks by examining different algorithms (GRPO, RAFT, DPO), rewards, divergence, and diversity regularization, and develop the best strategy that stably optimizes multiple objectives without mode collapse. 5. Through extensive experiments on inverse folding tasks, we demonstrate that ProteinZero substantially outperforms existing methods across every key metric, achieving significant improvements in structural accuracy, thermodynamic stability, and sequence diversity while reducing design failure rates by approximately 36% - 48% compared to widely-used methods like ProteinMPNN ( proteinmpnn ) , ESM-IF ( esm_if ) and InstructPLM ( instruct_plm ) . Notably, our model is able to design better sequences across diverse and complex protein folds, and significantly improves the success rate on long-chain proteins. 2 Related Work Protein Inverse Folding Models. Inverse protein folding has been transformed by deep learning approaches that generate amino acid sequences for target structures. ProteinMPNN ( proteinmpnn ) pioneered message passing neural networks for this task, while ESM-IF ( esm_if ) leveraged pretrained protein language models to enhance structure-conditioned sequence generation. Other notable approaches include GVP-GNN ( GVP_GNN ) , which incorporated geometric vector perceptrons, and StructTrans ( StructTrans ) , which employed specialized transformer architectures. Recently, InstructPLM ( instruct_plm ) established a new state-of-the-art performance across diverse inverse folding benchmarks—we therefore adopt this model as our base architecture for fine-tuning. However, despite these advances, current models remain constrained by their reliance on fixed offline datasets, limiting exploration of novel sequence space. While they perform well on benchmark tasks, they lack mechanisms to learn from their own generations due to the prohibitive computational costs of quality assessment, creating a fundamental bottleneck for continuous improvement over multi-objectives. Table 1: Total wall-clock time (including MSA and template search) required to generate reward for eight inverse-folding sequences conditioned on the same structural backbone. Best results are highlighted in blue . The wall-clock time without MSA search can be found in Appx. Table 4 Length range Structural and Designability Reward Thermal Stability Reward (ddG) ESMFold AlphaFold 2 ColabFold OpenFold AlphaFold 3 Pred-ddG(ours) FoldX 0–150 aa 18.7 s 1632.6 s ( \sim 87.3 \times ) 576.2 s ( \sim 30.8 \times ) 674.9 s ( \sim 36.1 \times ) 705.4 s ( \sim 37.7 \times ) \sim 2 s (GPU) 472.3 s ( \sim 236.2 \times ) 150–300 aa 47.5 s 4112.5 s ( \sim 86.6 \times ) 1272.8 s ( \sim 26.8 \times ) 1424.7 s ( \sim 30.0 \times ) 1920.5 s ( \sim 40.4 \times ) \sim 2 s (GPU) 1520.6 s ( \sim 760.3 \times ) RLHF of Generative Models. Reinforcement Learning from Human Feedback (RLHF) has transformed generative models through two main approaches: online methods like PPO ( ppo ) , GRPO ( grpo ) , and RAFT ( Raft ) that learn through direct reward optimization,

Read the complete text