Continued domain-specific pre-training of protein language models for pMHC-I binding prediction

Sergio E. Mares · 2025

Read Continued domain-specific pre-training of protein language models for pMHC-I binding prediction by Sergio E. Mares (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13077.txt.

Introduction

Sergio Mares and co-authors test in 2025 whether extra training in a narrow domain helps protein language models predict immune binding. The task is to say how strongly a peptide binds a class I immune presenting molecule, a step in how the body flags threats. Tens of thousands of variants exist, data are scarce for most, and measures are noisy.

Starting from a protein model, the team kept training it on peptides tied to these immune molecules, trying the peptides alone and paired with the presenting chain. They then fine tuned the model only on high quality numeric binding data, avoiding biases that mass based methods can carry in. The question is practical: can focused pre training lift accuracy where older methods struggle most, on rare variants and exact binding strength.

Catalog record

CREATORSergio E. Mares
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continued domain-specific pre-training of protein language models for pMHC-I binding prediction Thanks: Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA Thanks: Center for Computational Biology and Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA, USA; Chan Zuckerberg Biohub, San Francisco, CA, USA Sergio E. Mares Abstract Predicting peptide–major histocompatibility complex I (pMHC-I) binding affinity remains challenging due to extreme allelic diversity ( ∼ \sim 30,000 HLA alleles), severe data scarcity for most alleles, and noisy experimental measurements. Current methods particularly struggle with underrepresented alleles and quantitative binding prediction. We test whether domain-specific continued pre-training of protein language models is beneficial for their application to pMHC-I binding affinity prediction. Starting from ESM Cambrian (300M parameters), we perform masked-language modeling (MLM)-based continued pre-training on HLA-associated peptides (epitopes), testing two input formats: epitope sequences alone versus epitopes concatenated with HLA heavy chain sequences. We then fine-tune for functional IC 50 binding affinity prediction using only high-quality quantitative data, avoiding mass spectrometry biases that are inherited by existing methods. Key Results: After continued pre-training and fine-tuning, our resulting model (ESMCBA) achieves a median Spearman correlation of 0.62 for predicting binding affinity across 25 common HLA alleles, outperforming NetMHCpan (0.56), MHCflurry (0.49), and other state-of-the-art predictors. Continued pre-training provides consistent gains relative to ESM Cambrian models that are directly fine-tuned without the continued pre-training step, particularly for alleles with a moderate amount of available binding data (500–2000 peptides), improving correlations by ∼ \sim 0.10 over models without continued pre-training. Limitations: The benefits of continued pre-training drop significantly for data-scarce alleles ( < < 500 peptides), where models without continued pre-training outperform continued pretraining models. In addition, the method requires substantial computational resources (300M parameters), and the fine-tuning step remains limited by the inherent noise in binding affinity measurements. Binding prediction shows variable performance across alleles, highlighting ongoing challenges for generalization over data-scarce alleles. Impact: This work has important potential application to neoantigen vaccine prioritization and provides a framework for improving protein language model performance on specialized tasks through domain-specific continued pre-training. 1 Introduction Protein language models (PLMs) trained on large protein corpora have become foundational tools for structure and function prediction [ Rives et al. 2021 , Lin et al. 2023 ] . However, most applications of these models to downstream tasks involve a standard fine-tuning approach. In natural language processing, domain-specific continued pre-training—where models undergo additional unsupervised training on task-relevant data before supervised fine-tuning—often yields substantial performance gains [ Gururangan et al. 2020 ] . Whether this strategy translates effectively to protein modeling remains largely unexplored. We investigate this question using peptide–major histocompatibility complex class I (pMHC-I) binding affinity prediction. This task represents an important test case for several reasons. First, accurate pMHC-I binding prediction is critical for vaccine design and personalized immunotherapy [ Vita et al. 2019 ] , making performance improvements directly clinically relevant. Second, the task suffers from extreme data scarcity and imbalance: while humans express approximately 30 000 different HLA class I alleles, quantitative binding data exist for fewer than 200 alleles, with most alleles having fewer than 1000 measured peptide binding affinities. Third, the task requires joint modeling of highly polymorphic HLA chains and diverse peptide sequences, creating a stringent benchmark for cross-sequence generalization. Current pMHC-I binding predictors face three fundamental challenges. Allelic diversity: The extreme polymorphism of HLA genes creates a long-tail distribution where most alleles lack sufficient training data for robust supervised learning. Experimental bias: Mass spectrometry-based datasets systematically over-represent peptides with canonical anchor residues, creating training distributions skewed toward specific motifs while under-sampling weak binders [ Bruno et al. 2023 ] . Label heterogeneity: Binding measurements come from diverse experimental protocols (competitive binding, mass spectrometry, fluorescence polarization) with varying quality and interpretation, complicating model training and evaluation. 1.1 Hypothesis and Approach We hypothesize that domain-specific continued pre-training can improve protein language model representations of peptide sequences bound to MHC-I, improving downstream performance on binding affinity prediction. Specifically, we test whether additional masked-language modeling pre-training on HLA-associated peptides—before supervised fine-tuning—enables models to learn generalizable binding motifs across alleles. Starting from the 300M-parameter ESM Cambrian model [ Nijkamp and Team 2024 , Hayes et al. 2025 ] , we implement a two-stage training protocol: Stage 1 (Unsupervised): Continued masked-language modeling pre-training on two domain-specific corpora: (i) epitope sequences alone and (ii) epitopes concatenated with their corresponding HLA heavy chains. Stage 2 (Supervised): Fine-tuning of the continued pre-training models for half-maximal inhibitory concentration (IC 50 ) binding affinity prediction. To mitigate experimental bias, we train exclusively on high-quality functional antagonist assays, avoiding mass spectrometry data. We evaluate our approach—termed ESMCBA (ESM Cambrian Binding Affinity)—on the hypothesis that continued pre-training should: (1) improve performance over baseline ESM models without additional pre-training, (2) enhance data efficiency for low-resource alleles, and (3) match or exceed current state-of-the-art predictors. 1.2 Related Work Early pMHC-I binding predictors relied on position-weight matrices and linear models [ Chen et al. 2019 ] . Modern neural approaches, including MHCflurry [ O’Donnell et al. 2020 ] , HLAthena [ Sarkizova et al. 2020 ] , MHCnuggets [ Shao et al. 2020 ] , NetMHCpan [ Reynisson et al. 2020 ] , and HLApollo [ Thrift et al. 2024 ] , have achieved substantial improvements. However, all of these methods train on the same types of experimental datasets and thus inherit the systematic biases present in mass spectrometry-derived training data [ Bruno et al. 2023 ] . Recent work has begun exploring protein language models for immunological applications. However, these efforts have primarily focused on standard feature-extraction approaches without investigating domain-specific continued pre-training Thrift et al. 2024 . Our work fills this gap by systematically evaluating whether additional unsupervised learning on immunological sequences can improve downstream task performance. In this work, we demonstrate that domain-specific continued pre-training significantly enhances pMHC-I binding prediction and outperforms existing methods. Additionally, we provide the first systematic analysis of how continued pre-training influences protein language models across varying conditions, showing that improvements are most pronounced for alleles with moderate data availability (500–2000 peptides). We introduce ESMCBA as a practical tool designed for neoantigen prioritization, effectively addressing critical limitations of existing predictors, particularly for underrepresented alleles. Finally, we establish a methodological framework for applying continued pre-training to specialized biological prediction tasks, providing guidance for future protein language model applications. 2 Results Figure 1: (A) Distribution of available pMHC-I training data across 121 HLA alleles from classes A, B, and C. (B) IC 50 binding affinity distribution (nM). (C) Two-stage training workflow: unsupervised continued pretraining on epitope sequences (E) or HLA+epitope concatenations (H+E), followed by supervised fine-tuning for binding affinity prediction. 2.1 IC 50 data is scarce across alleles We first extracted quantitative peptide–MHC binding affinity measurements from the Immune Epitope Database (IEDB) across available HLA-A, HLA-B, and HLA-C alleles, filtering entries to exclude sequences containing non-canonical residues. We show (Fig. 1 a) that most alleles have fewer than 1000 associated peptide measurements, highlighting the data scarcity for many alleles. We also observe substantial variability and notable outliers in the measured IC 50 binding affinities (Fig. 1 b); therefore, we log-transform the data to stabilize variance and improve downstream modeling performance. 2.2 Continued pre-training leads to more

Read the complete text