Exploring Large Protein Language Models in Constrained Evaluation Scenarios within the FLIP Benchmark Journal: arXiv

Manuel F. Mollon, Joaquin Gonzalez-Rodriguez, Alicia Lozano-Diez, Daniel Ramos, Doroteo T. Toledano · 2025

Read Exploring Large Protein Language Models in Constrained Evaluation Scenarios within the FLIP Benchmark Journal: arXiv by Manuel F. Mollon, Joaquin Gonzalez-Rodriguez, Alicia Lozano-Diez, Daniel Ramos, Doroteo T. Toledano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18223.txt.

Introduction

Mollon and four co-authors test big protein language models on small, specialized tasks in this two thousand twenty-five paper. These models, trained on millions of protein sequences, promise to predict how mutations change a protein's fitness, but most benchmarks use large and varied datasets.

The FLIP benchmark instead stresses constrained settings where only a little task-specific data exists, closer to many real lab problems. The team evaluated leading models including ESM-2 and SaProt across FLIP's tasks. Their findings show how much the recent jumps in model scale actually help when data is scarce. That answer matters for biologists deciding whether a giant general model can stand in for experiments in niche protein projects.

Catalog record

CREATORManuel F. Mollon, Joaquin Gonzalez-Rodriguez, Alicia Lozano-Diez, Daniel Ramos, Doroteo T. Toledano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Large Protein Language Models in Constrained Evaluation Scenarios within the FLIP Benchmark Journal: arXiv Manuel F. Mollon, Joaquin Gonzalez-Rodriguez, Alicia Lozano-Diez, Daniel Ramos, Doroteo T. Toledano Abstract In this study, we expand upon the FLIP benchmark—designed for evaluating protein fitness prediction models in small, specialized prediction tasks—by assessing the performance of state-of-the-art large protein language models, including ESM-2 and SaProt on the FLIP dataset. Unlike larger, more diverse benchmarks such as ProteinGym, which cover a broad spectrum of tasks, FLIP focuses on constrained settings where data availability is limited. This makes it an ideal framework to evaluate model performance in scenarios with scarce task-specific data. We investigate whether recent advances in protein language models lead to significant improvements in such settings. Our findings provide valuable insights into the performance of large-scale models in specialized protein prediction tasks. 1 Introduction Protein fitness prediction, the process of forecasting the functional impact of mutations on protein behavior, is a critical task in computational biology. Accurate prediction of protein fitness has significant implications for various fields, including drug design, protein engineering, and directed evolution. Directed evolution [ 1 ] , in particular, relies on the ability to predict how mutations will influence protein properties, enabling the design of proteins with desired traits. However, traditional methods of protein fitness prediction, such as early machine learning (ML)-based approaches often require large datasets, limiting their application to certain scenarios. To address this challenge, the FLIP (Functional Landscape of Interacting Proteins) [ 2 ] benchmark was developed as a specialized dataset tailored to predict protein fitness under constrained conditions. Unlike larger datasets, such as ProteinGym, which encompass a wide array of protein-related tasks, the FLIP dataset focuses on smaller, more specific tasks. FLIP uses data splits like two vs many and low vs high, providing a unique opportunity to study model performance in scenarios with lower mutation levels during training and higher mutation levels during testing. This differs from the ProteinGym [ 3 ] approach, which involves random sampling across tasks. The FLIP setup is important for real-world applications where models must generalize well to more complex, higher-mutation scenarios. This structure makes overfitting a bigger concern in FLIP compared to random sampling scenarios, as the model is trained on lower mutations and tested on higher mutations. While this does not inherently make one approach better or worse, it is a crucial factor to consider when analyzing training and evaluation metrics in this paper, especially for models with limited data and a need for generalization across diverse protein functions. Recent advancements in state-of-the-art models like ESM-2 [ 4 ] and SaProt [ 5 ] have shown promising results in addressing the challenge of requiring large datasets by using self-supervised pretraining. These models are trained on vast amounts of unlabeled protein sequence data, enabling them to capture patterns and structural properties of proteins without needing labeled datasets. This allows them to generalize well in low-data scenarios or zero-shot tasks. These models are increasingly being explored for protein fitness prediction tasks, where their ability to generalize from limited training data is highly valuable. In this study, we expand on the FLIP baseline by testing the performance of these recent high-performance models within the context of the FLIP benchmark, aiming to understand how they perform in small, task-specific datasets. 2 Dataset Within the FLIP benchmark, several activities provide the foundational tasks for protein fitness prediction. Among them, three key activities are GB1, Meltome, and AAV. These three tasks used in this paper and summarized in this section represent distinct challenges in protein fitness prediction, each contributing to our understanding of how different protein features influence function under mutation. For evaluation, we selected a subset of the splits used in the FLIP benchmark. The selected splits and their sample distributions across train and test sets are shown in Table 1 . For detailed information on each split, please refer to the FLIP paper [ 2 ] . These activities within the FLIP benchmark provide a rich and diverse set of tasks that require accurate protein fitness predictions. The tasks vary in their focus, from stability and melting temperature predictions to the functional efficiency of viral vectors, making the FLIP benchmark ideal for testing protein language models in small, task-specific scenarios. 2.1 AAV The AAV task focuses on predicting the efficiency of adenovirus-associated virus (AAV) vectors used in gene therapy. The task involves predicting how mutations in the AAV capsid affect its ability to deliver genetic material to target cells. This is an important application in gene therapy, as optimizing the viral vector’s efficiency can lead to better therapeutic outcomes. The AAV dataset consists of mutations that either enhance or reduce the viral vector’s transduction efficiency, and the challenge is to predict these effects based on sequence data. 2.2 Meltome (Thermostability) The Meltome activity involves predicting the melting temperature ( T_{m} ) of proteins, which serves as a proxy for protein stability. The melting temperature is an important characteristic that reflects the protein’s ability to maintain its functional conformation under different conditions. The Meltome dataset includes a wide range of mutations across various proteins, and the task is to predict how these mutations will shift the T_{m} value, indicating changes in protein stability. This task is particularly relevant for protein design, where maintaining stability is often a key goal. 2.3 GB1 The GB1 task focuses on the prediction of the stability of a single-domain antibody, GB1. This is a critical task for understanding antibody engineering. The dataset for GB1 contains mutations that are known to either stabilize or destabilize the protein, and the challenge lies in determining how these mutations influence the overall structural integrity and binding affinity. Landscape Split Total Train Test AAV 2-vs-rest 82,583 31,807 50,776 7-vs-rest 82,583 70,002 12,581 low-vs-high 82,583 47,546 35,037 Meltome Human 10,093 8,148 1,945 Human-cell 7,156 5,792 1,366 GB1 2-vs-rest 8,733 427 8,306 3-vs-rest 8,733 2,968 5,765 low-vs-high 8,733 5,089 3,644 sampled 8,733 6,961 1,772 Table 1 : Sample distribution across landscapes and splits 3 Evaluated Models In this study, we extend the evaluation of the FLIP benchmark to include additional state-of-the-art models and enhance training procedures to improve performance. Below, we summarize each model, including modifications made to ensure a comprehensive and fair assessment of model capabilities on the dataset. 3.1 ESM-1v [ 6 ] The ESM 1v model was previously used in the FLIP baseline; however, in this study, we make several improvements. The training process was extended by increasing the number of epochs and employing a learning rate (LR) scheduler to dynamically adjust the learning rate based on model performance, with early stopping applied via a patience parameter to prevent overfitting. These modifications aim to optimize the performance of the model in protein fitness prediction tasks. 3.2 ESM-2 We evaluate three configurations of the ESM-2 model: a 6, 33, and 48 layer variant. Table 2 provides details regarding the parameter count for each model. The 6-layer and 48-layer models were selected to assess the smallest and largest ESM-2 configurations, allowing us to analyze how model size impacts learning capabilities and generalization power. Additionally, we included the 33-layer model as an intermediate option to compare the performance of ESM-2 with its predecessor, ESM-1v, since

Read the complete text