Structure-Aligned Protein Language Model

Can (Sam) Chen, David Heurtel-Depeiges, Robert M. Vernon, Christopher James Langmead, Yoshua Bengio, Quentin Fournier · 2025

Read Structure-Aligned Protein Language Model by Can (Sam) Chen, David Heurtel-Depeiges, Robert M. Vernon, Christopher James Langmead, Yoshua Bengio, Quentin Fournier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16896.txt.

Introduction

Models that read protein chains have learned a lot from vast banks of sequences, but they often miss the shape those chains fold into. Can Chen and co-authors add shape sense to such a model in 2025. Their method leans on a second model type that already works with protein shapes as graphs.

Two tasks do the teaching. One aligns the chain model's inner codes with the shape model's codes across many proteins. The other trains the chain model to name shape tokens for parts of its own protein. A small filter then picks which spots give fair but hard lessons, since stored shapes vary in quality. The mix aims to give sequence models a firmer grasp of form.

Catalog record

CREATORCan (Sam) Chen, David Heurtel-Depeiges, Robert M. Vernon, Christopher James Langmead, Yoshua Bengio, Quentin Fournier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16896

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structure-Aligned Protein Language Model Can (Sam) Chen, David Heurtel-Depeiges, Robert M. Vernon, Christopher James Langmead, Yoshua Bengio, Quentin Fournier Abstract Protein language models (pLMs) pre-trained on vast protein sequence databases excel at various downstream tasks but often lack the structural knowledge essential for some biological applications. To address this, we introduce a method to enrich pLMs with structural knowledge by leveraging pre-trained protein graph neural networks (pGNNs). First, a latent-level contrastive learning task aligns residue representations from pLMs with those from pGNNs across multiple proteins, injecting inter-protein structural information. Additionally, a physical-level task integrates intra-protein information by training pLMs to predict structure tokens. Together, the proposed dual-task framework effectively incorporates both inter- and intra-protein structural knowledge into pLMs. Given the variability in the quality of protein structures in PDB, we further introduce a residue loss selection module that uses a small model trained on high-quality structures to select reliable yet challenging residue losses for the pLM to learn. Applying our structure alignment method as a simple, lightweight post-training step to the state-of-the-art ESM2 and AMPLIFY yields notable performance gains. These improvements are consistent across a wide range of tasks, including substantial gains in deep mutational scanning (DMS) fitness prediction and a 59% increase in P@L for ESM2 650M contact prediction on CASP16. Furthermore, we demonstrate that these performance gains are robust, scaling with model sizes from 8M to 650M and extending to different downstream tasks. Model Weights : huggingface.co/chandar-lab/structure-alignment Code Repository : github.com/chandar-lab/AMPLIFY 1 Introduction Building on recent progress in natural language processing ( Brown et al., 2020 ; Devlin et al., 2019 ) , researchers have focused on pre-training protein language models (pLMs) on vast databases of protein sequences with masked language modeling ( Rives et al., 2019 ; Hayes et al., 2024 ; Fournier et al., 2024 ) and next token prediction ( Ferruz et al., 2022 ) . These pLMs learn representations that researchers have demonstrated hold substantial potential across a variety of biological applications, including protein function annotation, enzyme-catalyzed reaction prediction, and protein classification ( Hu et al., 2022 ) . Additionally, Rives et al. (2019) observed that structural information emerged in the models’ latent representations without supervision. Nonetheless, while the sequence-only nature of pLMs contributes to their widespread adoption, they often struggle in tasks requiring detailed structural insights. For instance, the structured informed ESM-GearNet outperforms ESM2 by 9.7\% on the Human Protein-Protein Interaction classification task ( Xu et al., 2022 ; Su et al., 2024 ) . In this paper, we aim to develop a pLM that preserves its sequence-only nature for broader applicability yet is augmented with structural insights. Given the availability of open-source pre-trained protein graph neural networks (pGNNs) ( Zhang et al., 2023 ; Chen et al., 2023 ; Jumper et al., 2021 ) , we investigate integrating pGNN-derived structural insights into pLMs. Specifically, we introduce a latent-level contrastive learning task for the structural alignment of pLMs. As illustrated in Figure 1 , this task aligns residue hidden representations from the pLM ( \bm{h}_{a} ) with those from the pGNN ( \bm{h}_{g} ) across a batch of B proteins. During this process, the pGNN is frozen while the pLM is optimized to minimize the contrastive learning loss, enriching the pLM with inter-protein 1 1 1 Note that “inter-protein” and “intra-protein” refer to tasks involving multiple proteins and within a single protein, respectively. This usage differs from the biological definition, where “inter-protein” refers to interactions between two proteins, and “intra-protein” refers to interactions within a single protein chain. structural knowledge. However, pure contrastive alignment may overemphasize differing residue-level patterns across the dataset, neglecting intra-protein 1 1 footnotemark: 1 structural context ( Zheng and Li, 2024 ) . To address this, we add a physical-level task that trains the pLM to predict structural tokens \bm{z} (representing physical conformations ( van Kempen et al., 2022 ) ) from its residue representations \bm{h}_{a} . This reinforces the encoding of each residue within its protein, thereby enriching the pLM with intra-protein structural knowledge. We combine latent and physical tasks, yielding three residue loss types for a batch of proteins with a total length N : (i) N sequence-to-structure contrastive losses from the latent-level task, (ii) N structure-to-sequence contrastive losses from the latent-level task, and (iii) N structure token prediction losses from the physical-level task. The dual-task framework effectively integrates inter-protein and intra-protein residue-level structural knowledge (§ 3.1 ). The masked language modeling loss is additionally incorporated to preserve the sequential knowledge of pLMs. Figure 1 : Overview of the dual-task framework . (a) Latent-level task: contrastively aligns residue representations from the pLM and pGNN, allowing the pLM to learn inter-protein structural knowledge. (b) Physical-level task: trains the pLM to predict structural tokens, incorporating intra-protein knowledge. (c) Residue embedding visualization: UMAP colored by secondary structure, showing that alignment improves separation. (d) Downstream task performance: structural knowledge improves contact map prediction, thermostability estimation, and fitness landscape modeling. Given that some protein structure regions in the PDB are ambiguous or inaccurate ( Burley et al., 2019 ) , we propose a residue loss selection module that prioritizes residue losses aligned with high-quality protein structures across the 3\times N total residue losses (§ 3.2 ). First, we use resolution and R-free metrics ( Morris et al., 1992 ) to curate a high-quality reference set and train a small reference model on the set. Next, we compute the excess loss , defined as the difference between the residue loss of the current model and that of the reference model ( Mindermann et al., 2022 ) . Residue losses with high excess loss are selectively included in each loss type as they exhibit greater learnable potential. This module filters out inaccurate residues with high reference loss and easy residues with low current loss. By focusing on challenging yet reliable residue losses, the module improves both training effectiveness and efficiency. We conducted 10 ablations to validate our design choices. Our analysis demonstrates that the proposed dual-task framework improves performance, with residue loss selection providing further gains. The models were evaluated on a comprehensive suite of benchmarks. We assess performance on deep mutational scanning (DMS) fitness prediction using ProteinGym ( Notin et al., 2023 ) and on direct structural validation using a contact prediction task on withheld data from CASP16 ( Yuan et al., 2025 ) . We further test generalization on 9 tasks from xTrimoPGLM ( Chen et al., 2024 ) and 9 from SaProt ( Su et al., 2023 ) , and evaluate language modeling fidelity using pseudo-perplexity on the high-quality held-out validation set from Fournier et al. (2024) . We find that structure alignment is a computationally lightweight post-training step that, depending on the downstream task, either matches or exceeds the performance of the original model. To summarize, our contributions are three-fold: • We propose a dual-task framework that integrates inter-protein and intra-protein residue-level structural knowledge into pLMs, while fully retaining their language modeling capabilities. • We develop a residue loss selection module that prioritizes challenging yet reliable residue losses, enhancing the learning process of pLMs. • We conduct extensive experiments that demonstrate the effectiveness of our method across model sizes, model families and downstream tasks. 2 Preliminaries In this section, we

Read the complete text