Towards Understanding the Shape of Representations in Protein Language Models

Kosio Beshkov, Anders Malthe-Sørenssen · 2025

Read Towards Understanding the Shape of Representations in Protein Language Models by Kosio Beshkov, Anders Malthe-Sørenssen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24895.txt.

Introduction

Beshkov and Malthe-Sørenssen ask what protein language models keep inside their hidden layers. Such models read amino acid chains and turn them into number patterns. Knowing what those patterns hold could help design new proteins. Their 2025 paper studies the shape of the whole pattern space.

The authors use two math tools to compare proteins and their patterns. They find that averages and spread in the pattern space change in a curved way across layers. The models store nearby links between chain parts best, and the fit fades for far apart parts. The most true patterns sit near, but not at, the last layer. Protein sets came from the SCOP dataset, a standard source for protein structure classes.

Catalog record

CREATORKosio Beshkov, Anders Malthe-Sørenssen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24895

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Understanding the Shape of Representations in Protein Language Models Kosio Beshkov, Anders Malthe-Sørenssen Abstract While protein language models (PLMs) are one of the most promising avenues of research for future de novo protein design, the way in which they transform sequences to hidden representations, as well as the information encoded in such representations is yet to be fully understood. Several works have attempted to propose interpretability tools for PLMs, but they have focused on understanding how individual sequences are transformed by such models. Therefore, the way in which PLMs transform the whole space of sequences along with their relations is still unknown. In this work we attempt to understand this transformed space of sequences by identifying protein structure and representation with square-root velocity (SRV) representations and graph filtrations. Both approaches naturally lead to a metric space in which pairs of proteins or protein representations can be compared with each other. We analyze different types of proteins from the SCOP dataset and show that the Karcher mean and effective dimension of the SRV shape space follow a non-linear pattern as a function of the layers in ESM2 models of different sizes. Furthermore, we use graph filtrations as a tool to study the context lengths at which models encode the structural features of proteins. We find that PLMs preferentially encode immediate as well as local relations between residues, but start to degrade for larger context lengths. The most structurally faithful encoding tends to occur close to, but before the last layer of the models, indicating that training a folding model ontop of these layers might lead to improved folding performance. 1 Introduction Protein language models (PLMs) are a novel and powerful approach for the modeling and design of new proteins with desired structural or functional features ( Ferruz & Höcker 2022 ) . Using PLMs one can predict the folding of proteins in 3D space ( Lin et al. 2023 ) , generate new candidate sequences for viral vectors ( Lyu et al. 2024 ) , enzymes ( Madani et al. 2023 ) , biosensors ( Hayes et al. 2025 ) and functional binders ( Chen et al. 2025 ; Bryant & Elofsson 2023 ) . A particular setting in which PLMs have been found to be useful is in finding high-dimensional representations of protein sequences that are thought to reflect the physical, evolutionary or functional properties of a protein. This is especially useful since it allows one to efficiently evaluate and compare newly generated proteins without the need for expensive modeling or experiments. One of the most widespread models of this type are the Evolutionary Scale Modeling (ESM) models ( Lin et al. 2023 ; Hayes et al. 2025 ) , which we will analyze in this work. It is empirically understood that the representations in these models form a good initialization for folding models ( Lin et al. 2023 ) and can also be used as a reward function to guide other generative approaches ( Wang et al. 2025 ) . However, the precise features that these representations encode are still not fully understood. In a previous work by Zhang et al. 2024 , the authors develop a categorical Jacobian approach to suggest that PLMs encode the pairwise statistics of coevolving residues. In another work by Simon & Zou 2024 , the authors use sparse autoencoders to suggest that PLMs encode human-interpretable features such as binding, structural motifs and functional domains. In yet another work that leverages the power of sparse autoencoders ( Gujral et al. 2025 ) , the authors show that PLMs and the individual neurons in them also encode features related to biologically relevant terms within the Gene Ontology hierarchy ( Aleksander et al. 2023 ) . What all of these works study is the transformation of individual sequences to high-dimensional latent representations. However, such approaches ignore how different proteins or their representations relate to each other in the latent space of a PLM. If structure determines function, then it is reasonable to assume that similar structure determines similar function. Furthermore, if protein representations in PLMs characterize structural, evolutionary and functional features, then one might expect that similar representations share such features. In addition, the representation of a protein in a PLM is a tensor of shape amino acids \times model dimension , however for many applications it is standard to take the average over the first dimension, which ignores the shape of the representation, thereby missing the full richness of the information present in PLM representations. To state our motivation more formally, it is important to understand how the metric space of proteins compares to the metric space of complete shape representations in PLMs. Metric space approaches for the structural analysis of proteins have previously been considered within the rich field of shape analysis ( Liu et al. 2010 ) . The essential realization of these approaches is that two proteins can be structurally compared by finding the optimal way to superimpose them, allowing different types of transformations. This is the backbone for many standard tools in the field of rigid and flexible structural alignment such as root mean square deviation (RMSD), TM score ( Zhang & Skolnick 2004 ) and FATCAT ( Li et al. 2020 ) among others. Despite their popularity, to our knowledge, such methods have yet to be applied to study the hidden representations of PLMs. In this work, we adapt and extend the shape-analysis framework to study the layerwise metric representation spaces of eight different classes in the SCOP ( Chandonia et al. 2022 ) dataset pushed through several ESM2 models. We consider different features such as the Fréchet radius and the effective dimensionality of these spaces and show that they follow a peculiar pattern as a function of layer, which is especially prominent for larger models. Furthermore, we introduce a graph-filtration method that allows us to separate and study the scale at which PLMs best maintain the structure of proteins. Using this analysis, we show that while structure is always encoded better than chance in deep PLM representations, it is optimally encoded at very short context lengths of 2 or at slightly longer context lengths at \sim 8 amino-acid neighbors. 2 Background In this section, we fix our notation and introduce the mathematical machinery necessary for comparing proteins and their corresponding representations in protein language models. We define two different ways to compare proteins with PLM representations. In the first one, each protein is a point in a high-dimensional (sometimes infinite-dimensional) space, and we define a metric that can be consistently applied to proteins made up of different numbers of amino acids. In the second approach, we define a filtration of metric spaces that only allows us to compare proteins of the same size. As we will argue, this is useful for understanding the context length that current PLMs are sensitive to, as well as the degree to which protein structure is encoded in PLMs. We start by outlining several ways to mathematically formulate what proteins are. For each such definition of a ”protein” we also propose a metric space in which such proteins can be sensibly embedded in. 1. Proteins as sequences : In this case, we define

Read the complete text