Layer Probing Improves Kinase Functional Prediction with Protein Language Models
Ajit Kumar, Indra Prakash Jha · 2025
Read Layer Probing Improves Kinase Functional Prediction with Protein Language Models by Ajit Kumar, Indra Prakash Jha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00376.txt.
Introduction
Protein language models learn from amino acid sequences, and their layers may store different kinds of knowledge. Ajit Kumar and Indra Prakash Jha probe those layers for kinase functional prediction in 2025. Most applications use only the final layer's embeddings, possibly missing useful signals.
The authors systematically evaluate embeddings from the ESM-2 model with unsupervised clustering and supervised learning. Kinases are enzymes central to cell signaling and drug design, so predicting their function matters. If intermediate layers carry extra functional information, practitioners can improve predictions simply by reading more of a model they already use. Kinases are heavily studied drug targets, so even small prediction gains have wide impact.
Catalog record
| CREATOR | Ajit Kumar, Indra Prakash Jha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layer Probing Improves Kinase Functional Prediction with Protein Language Models Ajit Kumar, Indra Prakash Jha Abstract Protein language models (PLMs) like ESM-2 have shown remarkable success in learning protein sequence representations. However, most applications use only the final layer embeddings, potentially missing functionally relevant information encoded in intermediate layers. In this study, we systematically evaluated ESM-2 embeddings for kinase functional classification using both unsupervised clustering and supervised learning. Our results show that mid-to-late transformer layers (layers 20–33) outperform the final layer by 32% (ARI: 0.268 → \rightarrow 0.354) in unsupervised settings and significantly improve supervised classification accuracy to 75.7% under rigorous homology-aware evaluation. We also incorporate domain extraction, calibrated confidence estimates, and reproducible benchmarking, demonstrating a practical and scalable pipeline for kinase functional annotation. keywords Protein Language Models, Kinase Functional Classification, Layer Selection in Transformers, Domain-Specific Embeddings † † equal-contributors: These authors contributed equally to this work. 1 Introduction 1.1 Background Protein kinases are critical enzymes for cell signaling and major drug targets, but predicting their specific function from the sequence remains challenging. Advances in protein language models (PLMs), particularly the Evolutionary Scale Modeling (ESM) family, have made it possible to extract biologically meaningful features directly from amino acid sequences. These models, trained on millions of sequences using masked language modeling objectives, have achieved state-of-the-art results in a range of protein prediction tasks. Yet, a common oversight is the over-reliance on the final transformer layer, which may not contain the most functionally relevant information. 1.2 The Layer Selection Problem Most studies default to using the final transformer layer for downstream tasks, assuming it encapsulates the richest representation. However, transformer-based PLMs are hierarchical, and prior work in natural language processing (NLP) has shown that intermediate layers often encode more transferable or semantically relevant features peters2018deep ; jawahar2019bert ; rogers2020primer . This discrepancy raises a fundamental question: Which transformer layers best capture biologically meaningful information for protein function prediction? 1.3 Objectives Protein kinases play a central role in cellular signaling, and accurate functional classification is critical for understanding disease mechanisms and drug development. Although protein language models (PLMs) offer a powerful route to sequence-based functional inference, most existing applications rely solely on the final transformer layer—potentially overlooking biologically relevant information distributed across intermediate layers. This study investigates the following questions: 1. Can intermediate transformer layers in ESM-2 improve functional classification of kinase domains compared to the final layer? 2. What is the optimal strategy for selecting and aggregating layer embeddings for kinase classification? 3. How do different embedding strategies affect unsupervised clustering and supervised classification under homology-aware conditions? To answer these, we build a reproducible pipeline incorporating domain-level extraction, layer probing, calibrated classification, and rigorous evaluation. Our goal is to provide a practical framework for improving functional predictions of protein kinases using PLM representations. 2 Related Work 2.1 Traditional Approaches to Kinase Classification Previous methods have relied on homology-based annotation using tools like BLAST or HMMER , or motif-based heuristics curated from the literature. While these methods perform well for known families, they struggle with novel sequences and require significant expert intervention. 2.2 Deep Learning and PLM-Based Protein Function Prediction With the rise of protein language models (PLMs) like UniRep , TAPE , ProtBERT , and ProtT5 , the focus has shifted to unsupervised representation learning. Embedding-based models have shown promise in capturing global and local sequence properties relevant to function. However, most applications simply extract the final-layer embedding—a design choice that may not be optimal. 2.3 Layer Selection in PLMs Layer probing in natural language processing (e.g., BERT , GPT ) has shown that semantic features often peak at intermediate layers. In protein ML, recent studies hint at similar trends, but systematic evaluations are rare. Our work contributes the first comprehensive analysis of layer-wise embedding utility in kinase classification. 3 Methods 3.1 Data Collection We retrieved a curated dataset of protein kinase sequences from the UniProt SwissProt database (release October 2025) using the following query: reviewed:true AND (keyword:KW-0418 OR name:kinase*) . Only canonical isoforms with sequence length greater than 100 amino acids were retained. Fragmentary sequences were excluded using UniProt flags. To focus on functionally relevant regions, Pfam domains PF00069 (Protein kinase domain) and PF07714 (Protein tyrosine kinase) were extracted using HMMER 3.3 with an E-value threshold of 0.001. CD-HIT 4.8.1 was used to remove redundancy at various identity thresholds (70%, 50%, 40%) to generate homology-aware splits. All tool versions and parameters were recorded for reproducibility. 3.2 Model Architecture We used the ESM-2 650M model ( esm2_t33_650M_UR50D ), a 33-layer transformer encoder pretrained on UniRef50 using a masked language modeling (MLM) objective. Each residue is mapped to a 1280-dimensional vector. For sequences exceeding the model’s maximum input length (1,022 residues), we applied a sliding window approach with overlap stitching. We compared different pooling strategies including mean over residues and [CLS] token extraction, and experimented with embeddings from different layer ranges: final layer (33), mid-layers (20–30), and mid-to-final layers (20–33). Model Selection Rationale Why ESM-2 over other protein language models? We selected ESM-2 (650M parameters) lin2023 as our primary model for five reasons: 1. State-of-the-art performance on protein tasks : ESM-2 achieves the highest accuracy among publicly available protein language models on CATH structure prediction (CATH 4.2: 87% top-1) and protein-protein interaction prediction. Meta AI’s 2023 benchmark shows ESM-2 outperforms ESM-1b (+8%), ProtBERT (+12%), and ProtTrans (+6%) on functional annotation tasks. 2. Evolutionary-scale training data : ESM-2 was trained on UniRef50 (2020) with ~50M sequences spanning diverse protein families, ensuring broad coverage of kinase evolutionary space. In contrast, ProtBERT (~200K sequences) and earlier ESM-1b (~250M sequences) have more limited diversity. 3. Appropriate architecture depth for layer probing : ESM-2’s 33-layer transformer provides sufficient depth to explore intermediate representations. This is critical for our research question (layer selection). Shallower models (ProtBERT: 12 layers, ESM-1b: 33 layers) or deeper models (ESM-2 15B: 48 layers) would either limit or complicate layer exploration. 4. Computational feasibility : The 650M parameter variant balances performance and accessibility. It runs on single consumer GPUs (e.g., NVIDIA RTX 3090/4090 with 24GB VRAM) and processes ~20 sequences/minute, enabling iteration on ~6,500 kinases within reasonable time (~6 hours). Larger models (3B, 15B) require multi-GPU setups and 10–50 \times longer processing times. 5. Established baseline for reproducibility : ESM-2 is the current de facto standard in protein ML (¿2,000 citations in 2 years), with well-documented APIs ( fair-esm ) and extensive community adoption. Using ESM-2 ensures our layer selection findings are immediately applicable to ongoing research and directly comparable to other studies. Alternative models considered but not used: • ESM-1b (33 layers, 650M): Predecessor to ESM-2, superseded by improved training. • ProtBERT (12 layers): Too shallow for layer probing experiments. • ProtTrans-XLNet-BFD (24 layers): Comparable but less widely adopted. • ESM-2 3B/15B : Prohibitively expensive for our dataset size (~100–500 GPU-hours). • AlphaFold2 embeddings : Optimized for structure, not function; requires MSAs. • Ankh (2023): Promising but very recent (unclear stability/generalization). Variant selection (650M vs 3B vs 15B): We chose the 650M parameter variant to balance performance and computational cost. Based on Meta AI’s ablation studies lin2023 , the 3B and 15B variants are expected to improve downstream task performance by 3–5% on average, but require: • 5 \times GPU memory (3B: ~40GB,