TurboESM: Ultra-Efficient 3-Bit KV Cache Quantization for Protein Language Models with Orthogonal Rotation and QJL Correction

Yue Hu, Junqing Wang, Yingchao Liu, [6pt] School of Bioengineering, Qilu University of Technology (Shandong Academy of Sciences),No. 3501 Daxue Road, Jinan, Shandong, China, [2pt] Shandong Provincial Hospital, Shandong First Medical University [4pt] https://github.com/YueHuLab/TurboESM · 2026

Read TurboESM: Ultra-Efficient 3-Bit KV Cache Quantization for Protein Language Models with Orthogonal Rotation and QJL Correction by Yue Hu, Junqing Wang, Yingchao Liu, [6pt] School of Bioengineering, Qilu University of Technology (Shandong Academy of Sciences),No. 3501 Daxue Road, Jinan, Shandong, China, [2pt] Shandong Provincial Hospital, Shandong First Medical University [4pt] https://github.com/YueHuLab/TurboESM (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26110.txt.

Introduction

Hu, Wang, and Liu present TurboESM, a way to shrink the memory cache of protein language models. These models predict protein structure well, but their key-value cache grows fast. Three-bit compression fails because activations have harsh outliers. TurboESM adapts a rotation-based quantizer to this domain. A head-wise calibration tunes it to amino acid patterns. A small correction table fixes the largest errors. On ESM-2, cache size drops from 330 to 47 megabytes. Decoding similarity stays above 0.96 across many protein families.

Catalog record

CREATORYue Hu, Junqing Wang, Yingchao Liu, [6pt] School of Bioengineering, Qilu University of Technology (Shandong Academy of Sciences),No. 3501 Daxue Road, Jinan, Shandong, China, [2pt] Shandong Provincial Hospital, Shandong First Medical University [4pt] https://github.com/YueHuLab/TurboESM
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39750
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26110

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TurboESM: Ultra-Efficient 3-Bit KV Cache Quantization for Protein Language Models with Orthogonal Rotation and QJL Correction Yue Hu, Junqing Wang, Yingchao Liu, [6pt] School of Bioengineering, Qilu University of Technology (Shandong Academy of Sciences),No. 3501 Daxue Road, Jinan, Shandong, China, [2pt] Shandong Provincial Hospital, Shandong First Medical University [4pt] https://github.com/YueHuLab/TurboESM Abstract The rapid scaling of Protein Language Models (PLMs) has unlocked unprecedented accuracy in protein structure prediction and design, but the quadratic memory growth of the Key-Value (KV) cache during inference remains a prohibitive barrier for single-GPU deployment and high-throughput generation. While 8-bit quantization is now standard, 3-bit quantization remains elusive due to severe numerical outliers in activations. This paper presents TurboESM , an adaptation of Google’s TurboQuant to the PLM domain. We solve the fundamental incompatibility between Rotary Position Embeddings (RoPE) and orthogonal transformations by deriving a RoPE-first rotation pipeline. We introduce a head-wise SVD calibration method tailored to the amino acid activation manifold, a dual look-up table (LUT) strategy for asymmetric K/V distributions, and a 1-bit Quantized Johnson-Lindenstrauss (QJL) residual correction. All experiments are conducted on ESM-2 650M, where our implementation achieves a 7.1 × \times memory reduction (330 MB → \to 47 MB) while maintaining cosine similarity > 0.96 >0.96 in autoregressive decoding across diverse protein families including short peptides, transmembrane helices, enzyme active site fragments, and intrinsically disordered regions. We further implement a Triton-based fused decode attention kernel that eliminates intermediate dequantization memory allocations, achieving a 1.96 × \times speedup over the PyTorch two-step path for the KV fetch operation alone; however, TurboESM incurs a prefill overhead of 21–27 ms relative to the original model due to KV quantization and packing, making it most suitable for memory-bound scenarios rather than latency-critical short-sequence workloads. Analysis reveals that PLMs exhibit sharper outlier profiles than large language models (LLMs) due to amino acid vocabulary sparsity, and our method effectively addresses these distributions. 1 Introduction Protein Language Models (PLMs) have revolutionized the field of computational biology by learning the “grammar” of life directly from amino acid sequence data. ESM-2, developed by Meta FAIR [ 1 ] , has become the industry standard for generating high-quality protein embeddings, supporting downstream tasks ranging from secondary structure prediction to protein–protein interaction docking. As these models scale to tens of billions of parameters, deployment efficiency becomes a first-class research concern alongside accuracy. During autoregressive generation or long-sequence processing, the Key-Value (KV) cache stores the attention states computed at each layer for all past tokens, allowing them to be reused without recomputation. This mechanism is essential for efficient inference, but its memory footprint grows quadratically with context length, making large-scale PLM deployment increasingly challenging as model size grows. Quantization is the natural solution to this memory pressure. By representing activations in fewer bits, the KV cache size can be reduced proportionally. 8-bit (INT8) quantization is now widely adopted with minimal accuracy degradation. However, 3-bit quantization—which would yield a theoretical \sim 10 \times compression—remains elusive for Transformer models because their activations are far from uniformly distributed. They contain “outlier” dimensions with values several orders of magnitude larger than the mean, which consume the entire dynamic range of a coarse quantizer and force the remaining 99% of values into a tiny cluster of quantization bins, effectively destroying the information they carry. In PLMs, these outliers are even more pronounced than in natural language LLMs. The amino acid vocabulary contains only 20 standard residues (compared to > 32,000 tokens in typical LLMs). This sparsity produces “spiky” activation patterns where certain channels consistently encode biologically critical features—conserved motifs, hydrophobic patches, or global sequence properties such as overall charge or length. Quantizing over such distributions without special treatment inevitably leads to catastrophic information loss at critical biological loci. Google recently introduced TurboQuant [ 2 ] , which addresses the outlier problem using an orthogonal matrix \Pi to rotate the activation space. This transformation spreads the energy of outliers uniformly across all dimensions, resulting in a distribution that closely approximates an isotropic Gaussian—the ideal target for any fixed-point quantizer. However, PLMs like ESM-2 use Rotary Position Embeddings (RoPE) [ 3 ] , which encode positional information by applying a position-dependent rotation R_{\theta,i} to each query-key pair before computing attention scores. The interplay between RoPE’s position-dependent rotation and TurboQuant’s data-driven feature rotation creates a non-trivial mathematical compatibility problem that, to our knowledge, has not been previously addressed. In this paper, we present TurboESM , which makes the following concrete contributions: 1. RoPE-invariant orthogonal transformation : We derive the correct operation ordering (RoPE before \Pi ) and prove that it preserves exact attention equivalence via the inner-product invariance of orthogonal matrices. 2. Head-wise SVD calibration : We compute a unique \Pi matrix per layer per attention head using SVD on real protein activations, capturing the distinct biological features encoded by each head. 3. Dual K/V LUT design : We calibrate independent 8-point Lloyd-Max look-up tables for keys (in the rotated space) and values (in the original space), recovering 1.2 dB of SNR over a shared LUT. 4. QJL 1-bit residual correction : We store the sign of the quantization residual (1 bit per element) and apply a first-order correction at decode time, reaching 4-bit-equivalent accuracy at 3.125-bit effective cost. 5. Triton fused decode kernel : We implement and validate a single CUDA kernel that merges 3-bit unpacking, QJL residual correction, and Flash-Attention-style online softmax into one pass, achieving a 1.96 \times speedup over PyTorch for the KV fetch operation. We note that TurboESM adds prefill latency overhead of 21–27 ms and is primarily beneficial in memory-bound, not latency-bound, settings. 6. Comprehensive empirical validation : We provide end-to-end accuracy measurements across six biologically diverse protein families on both Mac MPS (CPU-compatible) and NVIDIA GPU (CUDA) platforms, with cosine similarity consistently exceeding 0.96 at decode. The remainder of this paper is organized as follows. Section 2 reviews background on ESM-2 and the outlier problem in protein activations. Section 3 details the TurboESM methodology. Section 4 describes implementation details including vectorized quantization and the Triton kernel. Section 5 presents experimental results. Section 6 discusses the broader implications and differences between PLMs and LLMs. Section 7 concludes with future directions. 2 Background and Motivation 2.1 The ESM-2 Architecture ESM-2 [ 1 ] is a family of protein language models trained on the UniRef50 and UniRef90 databases using a masked language modeling objective. The models range from 8M to 15B parameters and follow the standard Transformer encoder architecture with a key modification: Rotary Position Embeddings (RoPE) replace the conventional learned absolute position embeddings. In this work we focus on ESM-2 650M as our experimental platform. In RoPE, the query vector q_{i} and key vector k_{j} at positions i and j , respectively, are transformed by position-dependent rotation matrices before computing attention: \text{Attention}(q_{i},k_{j})=\frac{(R_{\theta,i}\,q_{i})^{T}(R_{\theta,j}\,k_{j})}{\sqrt{d_{k}}} (1) where R_{\theta,i} is a block-diagonal matrix composed of 2\times 2 rotation blocks parameterized by frequency \theta and position i : R_{\theta,i}=\text{diag}\left(\begin{bmatrix}\cos i\theta_{1}&-\sin i\theta_{1}\\ \sin i\theta_{1}&\cos i\theta_{1}\end{bmatrix},\ldots,\begin{bmatrix}\cos i\theta_{d/2}&-\sin i\theta_{d/2}\\ \sin i\theta_{d/2}&\cos i\theta_{d/2}\end{bmatrix}\right) (2) The key mathematical property exploited by RoPE is that the inner product (R_{\theta,i}q_{i})^{T}(R_{\theta,j}k_{j})=q_{i}^{T}R_{\theta,i}^{T}R_{\theta,j}k_{j} depends only on the relative position j-i , providing translational equivariance. Any quantization scheme that modifies the KV cache must preserve this property to maintain fidelity in long-sequence scenarios. For

Read the complete text