g-DPO: Scalable Preference Optimization for Protein Language Models

Constance Ferragu, Jonathan D. Ziegler, Nicolas Deutschmann, Eli Bixby, Cradle ML Team · 2025

Read g-DPO: Scalable Preference Optimization for Protein Language Models by Constance Ferragu, Jonathan D. Ziegler, Nicolas Deutschmann, Eli Bixby, Cradle ML Team (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19474.txt.

Introduction

Constance Ferragu and five co-authors introduce g-DPO, a scalable way to align protein language models with experimental goals through preference optimization. Direct preference methods work well for protein design. But the number of possible training pairs grows with the square of the labeled sequences, making training prohibitively slow even for modest datasets.

Clustering sequences prunes redundant pairs while keeping the training signal, and group-based approximations cut the cost of repeated likelihood calculations. Across three protein engineering tasks, the method matches standard training in both computer and laboratory performance while converging several times faster, with speedups that grow as datasets expand.

Catalog record

CREATORConstance Ferragu, Jonathan D. Ziegler, Nicolas Deutschmann, Eli Bixby, Cradle ML Team
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

g-DPO: Scalable Preference Optimization for Protein Language Models Constance Ferragu, Jonathan D. Ziegler, Nicolas Deutschmann, Eli Bixby, Cradle ML Team Abstract Direct Preference Optimization (DPO) is an effective approach for aligning protein language models with experimental design goals. However, DPO faces a scalability bottleneck: the number of possible training pairs grows quadratically with the number of labeled sequences, leading to prohibitive training times even for modestly sized datasets. We introduce g-DPO, a framework that (i) uses sequence space clustering to prune redundant pairs while preserving training signal, and (ii) amortizes likelihood computations with group-based approximations. Across three protein engineering tasks, g-DPO maintains in-silico and in-vitro performance that is statistically indistinguishable from standard DPO, while converging 1.8 1.8 to 3.7 3.7 times faster, with greater gains expected as the size of the dataset increases. 1 Introduction Protein language models (PLMs), trained with self-supervised learning on large sequence datasets, capture relevant structure and function signals [ 17 , 33 , 25 , 14 ] . To use these models for protein engineering tasks, such as optimizing binding affinity, thermostability, or catalytic activity, they are typically fine-tuned on labeled experimental datasets, many of which naturally define preferences (e.g., higher fluorescence is better, lower toxicity is better). Direct Preference Optimization (DPO) is a natural fit for this setting. DPO fine-tunes a language model using pairwise preferences, maximizing the likelihood of preferred sequences relative to dispreferred ones [ 24 ] , without requiring a separate reward model as in RLHF [ 6 ] . Recent studies have applied DPO to protein sequence design [ 2 , 32 , 30 , 37 ] , however, unlike in NLP, where explicit human preferences are available [ 22 , 3 ] , experimental protein datasets often provide scalar labels. Constructing preferences by exhaustively comparing all samples is infeasible because the number of pairs grows quadratically with dataset size. Moreover, not all comparisons are equally informative. Distant comparisons in sequence space tend to collapse into coarse binary signals, while local comparisons capture subtle and often non-additive effects of a few mutations [ 12 , 10 ] , providing a more valuable learning signal, which is crucial in late-stage optimization. We introduce group-DPO (g-DPO), a DPO framework for experimentally labeled protein data that addresses these challenges. This study contributes: (1) Scalable preference sampling: Sequence-space clustering prunes redundant comparisons and focuses training on informative local neighborhoods in input space. (2) Efficient training: Building on this clustering, we exploit union masking to amortize log-likelihood computations across groups of sequences, further improving convergence time. (3) Empirical validation: Across three protein mutational landscapes, g-DPO maintains the in-silico and in-vitro performance of standard DPO, while converging 1.8 to 3.7 times faster, with larger convergence time gains expected as the size of the dataset increases. 2 Related work: Constructing pairs for DPO with PLMs Direct Preference Optimization (DPO). DPO fine-tunes language models directly from pairwise comparisons, avoiding the need for a separate reward model [ 24 ] . Given a distribution \mathcal{D} of pairs of sequences (y_{w},y_{l}) , where y_{w} (winner) is preferred over y_{l} (loser), a policy \pi_{\theta} is optimized to increase the likelihood of sampling y_{w} over y_{l} under a regularization constraint formulated as a bound on the KL divergence between \pi_{\theta} and a reference policy \pi_{\mathrm{ref}} . This objective is expressed as a loss function: \displaystyle\mathcal{L}_{\mathrm{DPO}}(\pi_{\theta},\pi_{\mathrm{ref}})=-\mathbb{E}_{(y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\Big(\beta\log\frac{\pi_{\theta}(y_{w})}{\pi_{\mathrm{ref}}(y_{w})}-\beta\log\frac{\pi_{\theta}(y_{l})}{\pi_{\mathrm{ref}}(y_{l})}\Big)\right], (1) where \beta>0 is the regularization hyperparameter and \sigma is the sigmoid function. A complete derivation is provided in A.1 . A primary challenge in applying DPO to PLMs is constructing a preference distribution \mathcal{D} from scalar labels, as exhaustively pairing labeled sequences does not scale. 2.1 Preference sampling for DPO with PLMs Output space partitioning. A common strategy is to partition labeled sequences by thresholds defined by design objectives (e.g., "get sequences above x stability") or quantiles, and assign sequences above as "preferred" and sequences below as "dispreferred" [ 20 , 35 , 34 , 26 , 36 ] . In terms of complexity, subsampling schemes, such as fixing k losers per winner, can be used and scale effectively with dataset size. Although this aligns well for coarse optimization goals, treating all pairs within the same partition as equivalent reduces resolution among high-performing sequences, which is limiting in late-stage optimization. Rank-space sampling. Unlike thresholding, which collapses labels into binary partitions, rankings preserve relative relationships across all sequences. Methods such as RRHF [ 38 ] and PRO [ 28 ] show that training on rankings (or their induced pairs) leads to a more stable convergence in aligning LMs with human preferences. Applied to proteins, CtrlProt [ 15 ] shows that a rank-wise preference objective improves controllability over pairwise methods for multi-objective optimization. Widatalla et al. [ 32 ] evaluate a ‘gap level’ heuristic that down-samples pairs of sequences at fixed distances in rank space, and find that most gap levels have comparable performance to randomly sampling pairs. This suggests that not all pairs are equally informative and that randomly pruning pairs in the output space risks diluting the training signal. Informative pairs. Other methods have been proposed to prioritize preference pairs that provide a stronger training signal. Maru et al. [ 18 ] propose a hybrid sampling strategy that mixes global and local perturbations, selecting pairs with large predicted stability differences for the former and pairs ten mutations apart for the latter. They find no improvement over randomly selecting pairs. Beyond proteins, other strategies have been proposed to improve informativeness, such as stratified sampling to ensure coverage across score bins, oversampling rare but informative regions, or curriculum learning to gradually introduce harder comparisons as the model improves [ 13 , 23 , 7 , 4 , 11 ] . 3 Methods: Group-DPO (g-DPO) In NLP, preferences are inherently structured: multiple outputs are generated from the same prompt, and annotators decide which is better [ 22 , 3 ] . This provides a shared context that makes likelihood ratios between outputs meaningful. On the other hand, comparing outputs from different prompts yields likelihoods ratios that are dominated by prompt differences rather than output preference. Proteins have a related dynamic: even when variants come from the same wild type, comparing distant variants conflates the effects of many mutations, collapsing into coarse signals, whereas local comparisons better capture subtle, non-additive effects [ 12 , 10 ] . Our work, therefore, leverages proximity in sequence space to construct preference pairs, enabling the model to learn fine-grained mutational effects while reducing the number of training pairs. Our g-DPO training framework is conducted in three stages. We begin with unsupervised fine-tuning of a PLM on evolutionarily related sequences of a wild type protein (evo-tuning) [ 1 , 8 ] . Next, given an experimental mutant dataset of the relevant wild type, we apply union mask clustering, a greedy agglomerative procedure that groups sequences based on shared mutational positions. Figure 1 : Clustering of mutants with overlapping mutations. Finally, we sample groups of sequences from each cluster and evaluate the DPO loss over all preferences within each group to further fine-tune our evo-tuned model. Although some assay variants could appear within the evo-tuning training data, this does not constitute data leakage as evo-tuning uses no assay labels. This unsupervised design choice is intended to

Read the complete text