PLM-eXplain: Divide and Conquer the Protein Embedding Space

Jan van Eck · 2025

Read PLM-eXplain: Divide and Conquer the Protein Embedding Space by Jan van Eck (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07156.txt.

Introduction

Jan van Eck and co-authors published a paper in 2025 on opening the black box of protein language models. These models turn amino acid sequences into rich number vectors that predict protein behavior well, but nobody can read why. The authors built an adapter that splits each vector into two parts: one phrased in classic biochemical features, the other a leftover that keeps the model's power.

Anchored on a widely used model, the readable part speaks in terms like secondary structure and water affinity, while the residual quietly carries everything else. Across three tasks, spotting membrane helices, vesicle association, and clumping risk, accuracy held while decisions became interpretable. The adapter generalizes across tasks, offering biologists predictions they can actually reason about.

Catalog record

CREATORJan van Eck
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07156

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PLM-eXplain: Divide and Conquer the Protein Embedding Space Jan van Eck Abstract Abstract Protein language models (PLMs) have revolutionised computational biology through their ability to generate powerful sequence representations for diverse prediction tasks. However, their black-box nature limits biological interpretation and translation to actionable insights. We present an explainable adapter layer - PLM-eXplain (PLM-X), that bridges this gap by factoring PLM embeddings into two components: an interpretable subspace based on established biochemical features, and a residual subspace that preserves the model’s predictive power. Using embeddings from ESM2, our adapter incorporates well-established properties, including secondary structure and hydropathy while maintaining high performance. We demonstrate the effectiveness of our approach across three protein-level classification tasks: prediction of extracellular vesicle association, identification of transmembrane helices, and prediction of aggregation propensity. PLM-X enables biological interpretation of model decisions without sacrificing accuracy, offering a generalisable solution for enhancing PLM interpretability across various downstream applications. This work addresses a critical need in computational biology by providing a bridge between powerful deep learning models and actionable biological insights. keywords: Protein language model, Explainable AI, Embeddings, Protein Property Prediction, Protein Aggregation. Introduction The field of computational biology has expanded rapidly, supported by the development of large-scale protein language models (PLMs) trained on extensive sequence databases [ 1 , 2 ] . These models quickly outperformed existing tools across a variety of protein prediction tasks [ 1 , 3 , 2 , 4 , 5 , 6 ] , enabling highly accurate predictions of properties ranging from secondary structure [ 7 , 8 ] and subcellular location [ 9 ] to protein aggregation [ 10 ] . A key innovation behind these PLMs is the transformer architecture [ 11 ] , which uses multi-head self-attention mechanisms to process entire protein sequences. This allows the model to learn context-dependent representations for each amino acid. This process captures patterns in the sequence that are stored in a numerical representation. The resulting dense embeddings integrate both local and global sequence information, making them valuable for various downstream tasks. A critical challenge remains that the representations learned by PLMs are not interpretable, in contrast to traditional shallow learning approaches that rely on hand-engineered features [ 12 ] . Although traditional methods may be limited in their predictive power, their use of carefully crafted features, such as physicochemical properties and various experimental annotations, provides clear biological meaning to their predictions [ 13 ] . PLMs, however, transform protein sequences into high-dimensional amino acid-based representations without specific biological meaning attached, offering limited insight into the underlying principles driving their predictions. This lack of interpretability limits scientific understanding of how PLMs capture biological mechanisms, potentially hindering their integration into experimental workflows where model decisions need clear biological rationales [ 14 , 15 ] . Efforts to address these issues have focused on internal feature importance, attention-weight analyses [ 16 ] , and post hoc interpretation methods [ 17 ] . However, these approaches do not offer a direct mapping between the learned latent representations and biological understanding. As a result, PLMs still lack a bridge between sequence data and biological interpretation. In this study, we present a new explainable adapter approach, PLM-eXplain (PLM-X) that retains the prediction power of protein language models while including interpretability of the representations. Our method utilises embeddings derived from ESM2 [ 2 ] , one of the most advanced PLMs currently available. We factored these embeddings into two complementary components: a subspace composed of crafted, interpretable physicochemical features and a compressed residual subspace capturing information not explicitly described by these known attributes. By anchoring a fraction of the embedding space in known descriptors, such as structure classifications (SS3 and SS8) [ 18 ] and hydropathy (GRAVY) [ 19 ] , we empower researchers to better rationalise the contributions of fundamental chemical and structural factors. The remaining subspace ensures that the model retains its full predictive power, preserving more subtle patterns that contribute to performance but are not captured by the predefined feature set. Our explainable adapter is reusable as a flexible layer that can be integrated into various downstream tasks without requiring retraining the adapter. To illustrate this versatility, we applied our semi-explainable embeddings to three distinct protein-level (global) classification problems: (i) prediction of extracellular vesicle (EV) proteins, which are crucial disease biomarkers occuring in all domains of life [ 20 , 21 , 22 , 23 ] ; (ii) identification of transmembrane proteins, a well-characterised task with state-of-the-art solutions [ 24 ] ; (iii) prediction of protein aggregation propensity (amyloidogenicity), which remains a critical challenge in clinical and biotechnological applications [ 25 , 26 , 27 ] . In each of these cases, we demonstrate that our explainable adapter not only preserves the high accuracy characteristic of black-box PLM embeddings but also provides a better method to explain the model’s decisions. Methods Our method employs a two-step approach to enhance the interpretability of protein language models while maintaining their predictive power (Figure 1 ). First, we train an adapter layer for ESM2 that serves as encoder and transforms traditional PLM embeddings into partitioned representations. This process splits the embedding space into two complementary components: an informed subspace grounded in established biochemical features and a residual subspace that preserves additional predictive information not captured by known attributes. Second, to demonstrate the versatility and effectiveness of these adapted embeddings, we evaluate them across three distinct protein classification tasks: aggregation propensity, EV association, and transmembrane helices classification. For each task, we explore two different architectural approaches: a protein-level (global) analysis method that pools amino acid embeddings by averaging and a local analysis method using convolutional neural networks (CNNs) to capture local patterns of crafted features. Partitioning PLM embeddings To transform the ESM2 PLM embeddings (t12_35M_UR50D) into partitioned representations, we create two complementary subspaces. The combined embeddings consists of 480 features, matching the size of the original ESM2 embedding. An informed subspace that explicitly captures established biochemical features (N X 34), and a residual subspace (N X 446) that captures the residual predictive information from the original embedding (Figure 1 ). The informed subspace is designed to represent well-understood protein characteristics in a transparent manner. By incorporating (hand)crafted features based on fundamental biochemical properties, this subspace provides direct interpretability for a portion of the model’s decision-making process. These features include the following metrics: hydrophobicity scales (GRAVY), aromaticity, secondary structure components (SS3, SS8), accessible surface area (ASA) [ 28 ] and standard amino acid types (Table S1 describes crafted features in detail). Each latent feature within this subspace corresponds to a distinct known element. Figure 1: Our encoder-decoder model architecture splits protein language model embeddings into two complementary subspaces: one capturing explicit physicochemical features and another containing the residual predictive information. An adversarial component stimulates this separation while preserving the original embeddings’ prediction capabilities. We evaluate the partitioned embeddings using XGBoost and CNN models on three downstream protein-level tasks. In contrast, the residual subspace preserves information that cannot be readily explained by biochemical features analysed in this work. This subspace is trained adversarially to minimise redundancy with the knowledge-informed subspace while maintaining the predictive power of the model. This design ensures that subtle patterns and complex relationships in the protein sequence data are not lost

Read the complete text