Boosting In-Silicon Directed Evolution with Fine-Tuned Protein Language Model and Tree Search
Yaodong Yang, Yang Wang, Jinpeng Li, Pei Guo, Da Han, Guangyong Chen, Pheng-Ann Heng · 2025
Read Boosting In-Silicon Directed Evolution with Fine-Tuned Protein Language Model and Tree Search by Yaodong Yang, Yang Wang, Jinpeng Li, Pei Guo, Da Han, Guangyong Chen, Pheng-Ann Heng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09900.txt.
Introduction
Yaodong Yang and six co-authors propose AlphaDE, a framework for evolving proteins inside the computer with help from protein language models. Directed evolution improves proteins through mutation, and language models trained on sequences have absorbed rich evolutionary patterns. Earlier computational methods mostly used heuristic search and did not combine these models with strong optimization.
AlphaDE first fine-tunes a pretrained protein language model on relatives of the target protein, using masked language modeling to wake up family-specific plausibility. It then searches at test time with Monte Carlo tree search, guided by the fine-tuned model. Benchmarks showed clear gains over previous state-of-the-art methods even with few-shot fine-tuning, and a case study condensed the sequence space of the fluorescent protein avGFP.
Catalog record
| CREATOR | Yaodong Yang, Yang Wang, Jinpeng Li, Pei Guo, Da Han, Guangyong Chen, Pheng-Ann Heng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-09900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Boosting In-Silicon Directed Evolution with Fine-Tuned Protein Language Model and Tree Search Yaodong Yang, Yang Wang, Jinpeng Li, Pei Guo, Da Han, Guangyong Chen, Pheng-Ann Heng Abstract Protein evolution through amino acid sequence mutations is a cornerstone of life sciences. While current in-silicon directed evolution algorithms largely focus on designing heuristic search strategies, they overlook how to integrate the transformative protein language models, which encode rich evolutionary patterns, with reinforcement learning to learn to directly evolve proteins. To bridge this gap, we propose AlphaDE, a novel framework to optimize protein sequences by harnessing the innovative paradigms of large language models such as fine-tuning and test-time inference. First, AlphaDE fine-tunes pretrained protein language models using masked language modeling on homologous protein sequences to activate the evolutionary plausibility for the interested protein class. Second, AlphaDE introduces test-time inference based on Monte Carlo tree search, which effectively evolves proteins with evolutionary guidance from the fine-tuned protein language model. Extensive benchmark experiments show that AlphaDE remarkably outperforms previous state-of-the-art methods even with few-shot fine-tuning. A further case study demonstrates that AlphaDE supports condensing the protein sequence space of avGFP through computational evolution. 1 Introduction Proteins are essential components of living systems, exhibiting a great diversity of functions among biological macromolecules ( 23 ) . They play critical roles in a wide range of biochemical processes, including enzyme catalysis, cellular metabolism, immune responses, and signal transduction ( 25 ) . Protein engineering through directed evolution enables optimization of protein functions by seeking potential protein variants with improved properties such as the expression level and catalytic activity ( 60 ) . Traditional in vitro or in vivo experimental directed evolution approaches, such as deep mutational scanning ( 16 ) and orthogonal DNA replication system ( 40 ) , directly measure the functional effects of protein mutations but are limited to exploring only a fraction of the possible protein space, and are usually expensive and laborious. To reduce the burden of the expensive wet experiments, recent advances in machine learning-guided approaches focus on building a surrogate sequence-function landscape ( 60 ) , and a lot of in-silicon directed evolution algorithms ( 8 ; 50 ; 41 ; 56 ) have been proposed to strategically explore the protein fitness landscape to identify optimal sequence mutations via an iterative search process. On the other hand, protein language models ( 33 ; 30 ) , which encapsulate millions of years of evolutionary information through unsupervised pretraining in massive protein databases, are rapidly changing the domain of protein design ( 32 ; 26 ; 21 ) . As protein language models implicitly learn complex evolutionary and structural dependencies from natural protein sequences, researchers have been increasingly employing protein language models for protein engineering tasks such as zero-shot inference of the functional effects of sequence substitutions to find high-fitness variations ( 22 ; 46 ) . More recently, some pioneering works have begun to introduce pretrained protein language models in the process of directed evolution ( 26 ; 59 ; 55 ) . However, although employing protein language models to recommend mutations, these works use simple heuristic search methods such as greedy selection ( 26 ) or beam search ( 55 ) to search in directed evolution, leaving a large room to explore how to integrate protein language models into advanced optimization techniques such as reinforcement learning. To fill this gap, in this work, we propose a novel framework named AlphaDE to directly evolve protein sequences following the technical paradigm of natural large language models ( 19 ) . Specifically, AlphaDE consists of a fine-tuning step and a Monte Carlo tree search (MCTS) inference step built on the protein language model. In the fine-tuning step, AlphaDE fine-tunes the pretrained protein language model with homologous protein sequences to activate its evolutionary plausibility for the interested protein class. In the MCTS-assisted inference step, similar to test-time MCTS in large language models to boost reasoning ( 61 ; 18 ) , AlphaDE conducts an iterative tree search to efficiently optimize protein function via residue mutations guided by the fine-tuned protein language model. Through the two synergic steps, AlphaDE harvests superior directed evolution ability for proteins. To evaluate AlphaDE, we conduct computational benchmark experiments on eight distinct tasks. Impressively, AlphaDE substantially outperforms various in-silicon directed evolution methods. Further few-shot fine-tuning experiments reveal that AlphaDE’s evolution ability can be activated by fine-tuning with dozens of homologous sequences. Lastly, to show the broader applicability of AlphaDE, we conduct a proof-of-concept task to computationally condense the sequence space of the functional avGFP protein. 2 Background 2.1 Protein Language Models In analogy to large language models in natural language processing, protein language models such as ESM ( 42 ; 30 ) , ProteinBERT ( 7 ) , and ProGen ( 32 ) have surged in modeling protein sequences. Pretrained by masked language modeling ( 14 ) or autoregressive language modeling ( 10 ) on evolutionary-scale protein databases, protein language models have demonstrated outstanding potential in protein design. For example, recently, 21 prompt a protein language model (i.e., ESM3) to generate a bright fluorescent protein far different from known fluorescent proteins, which is estimated to simulate five hundred million years of evolution. Another notable example is that 6 utilize contrastive language modeling to design peptide binders to conformationally diverse targets using only the amino acid sequence of the target protein. At the same time, some works ( 57 ; 58 ) utilize labeled sequence-fitness pairs to steer protein language models for de novo protein sequence design to generate novel and high-quality sequences, showing the superior alignment ability of protein language models. 2.2 Protein Directed Evolution Directed evolution is a classical paradigm for protein sequence design, where a plenty of algorithms are developed to accelerate the in-silicon directed evolution process. AdaLead ( 50 ) is an advanced implementation of model-guided directed evolution with iteratively recombined and mutated operations for seed sequences. CMA-ES ( 20 ) is a second-order evolutionary search algorithm that estimates the covariance matrix to adaptively adjust the search strategy of the upcoming generations. Bayesian optimization (BO) ( 51 ) is a classical paradigm for the sequential design problem ( 34 ) , which estimates the uncertainty and constructs the acquisition function for exploration. DbAS ( 8 ) establishes a probabilistic framework that trains a variational autoencoder (VAE) ( 27 ) to model the distribution of high-fitness sequences and adaptively samples sequences from this trained VAE to explore the fitness landscape. Following DbAS, CbAS ( 9 ) estimates the probability distribution conditioned on the desired properties with model-based adaptive sampling and additionally considers a regularization to stabilize the model-guided search process. DyNA-PPO ( 2 ) formulates protein sequence design as a sequential decision-making problem and uses proximal policy optimization (PPO) ( 45 ) to perform sequence generation. PEX ( 41 ) aims to search for effective candidates of low-order mutants near the wild-type, and formulates this process as a proximal optimization problem to solve. EvoPlay ( 56 ) uses the self-play reinforcement learning inspired by AlphaZero ( 49 ) to optimize protein sequences. Recently, TreeNeuralTS and TreeNeuralUCB ( 38 ) combine tree search with bandit machine learning for directed evolution, which expands a tree starting from the initial sequence with