Latent-Condensed Transformer for Efficient Long Context Modeling

Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan · 2026

Read Latent-Condensed Transformer for Efficient Long Context Modeling by Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12452.txt.

Introduction

Zeng You and seven colleagues shrink the cost of long-context language modeling. Long inputs inflate the key-value cache linearly and attention cost quadratically. Existing fixes attack only one bottleneck at a time. Their Latent-Condensed Attention condenses context inside a compressed latent space. Semantic vectors are pooled by query relevance, while positional keys survive through anchor selection. The design adds no parameters and extends to other attention types. Tests show up to 2.5 times faster prefilling and 90 percent less cache at 128,000 tokens.

Catalog record

CREATORZeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62260
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12452

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent-Condensed Transformer for Efficient Long Context Modeling Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan Abstract Large language models (LLMs) face significant challenges in processing long contexts due to the linear growth of the key-value (KV) cache and quadratic complexity of self-attention. Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation. However, sparse methods cannot operate natively on MLA’s compressed latent structure, missing opportunities for joint optimization. In this paper, we propose Latent-Condensed Attention (LCA), which directly condenses context within MLA’s latent space, where the representation is disentangled into semantic latent vectors and positional keys. LCA separately aggregates semantic vectors via query-aware pooling and preserves positional keys via anchor selection. This approach jointly reduces both computational cost and KV cache without adding parameters. Beyond MLA, LCA’s design is architecture-agnostic and readily extends to other attention mechanisms such as GQA. Theoretically, we prove a length-independent error bound. Experiments show LCA achieves up to 2.5 × \times prefilling speedup and 90% KV cache reduction at 128K context while maintaining competitive performance. 1 Introduction Efficient long-context modeling in large language models (LLMs) is essential for applications spanning full-document comprehension and extended multi-turn dialogues 29 ; 14 ; 15 . However, transformer-based LLMs face two challenges: 1) the linear growth of key-value (KV) cache during decoding and 2) the quadratic computational complexity of self-attention 34 , together hindering long - context deployment. To alleviate the memory overhead (challenge 1), Multi-head Latent Attention (MLA) 9 projects tokens into a shared low-dimensional latent space, significantly reducing the per-token KV cache size. During inference, MLA caches only the compressed latent vectors \mathbf{C}^{KV} and the positional keys \mathbf{K}^{R} , where \mathbf{K}^{R} encodes precise positional information via Rotary Position Embedding (RoPE) 31 . Owing to its favorable trade-off between memory efficiency and model capacity, MLA has been widely adopted in recent long-context models 9 ; 26 ; 15 ; 21 ; 22 . Nevertheless, MLA still retains all L latent vectors and performs dense attention, preserving the quadratic computational bottleneck. Separately, efficient attention methods mitigate quadratic computation (challenge 2) via sparsification. Early approaches used fixed sparse patterns 4 ; 39 , which lack adaptability and risk discarding important information. Recent dynamic sparse methods 20 ; 24 ; 37 selectively skip computation blocks or evict less relevant tokens to reduce computation. However, these approaches remain susceptible to potentially important information loss. Crucially, these two lines of work cannot be directly combined to address both challenges simultaneously. Most existing sparse methods operate on the computation of attention scores between original queries and keys, which require full-dimensional representations. When applied to MLA, these methods must first reconstruct full-dimensional KV matrices before performing sparsification. Consequently, these methods fail to leverage the compressed latent structure for further efficiency gains, particularly in reducing the KV cache beyond what MLA already achieves. This reveals a significant yet overlooked gap: existing methods lack a mechanism to perform efficient computation reduction natively within MLA’s latent space. In this paper, we bridge this gap by directly condensing redundant context within the MLA’s latent space into a compact set of representatives, thereby reducing the number of vectors participating in attention computation and stored in the KV cache. However, MLA’s disentangled representation presents a unique challenge: the semantic component \mathbf{C}^{KV} and the positional component \mathbf{K}^{R} have distinct functional properties due to their different encodings. Semantic information, which captures the continuous and often smooth content representations, can be aggregated across tokens without significant loss of fidelity 5 ; 12 ; while positional encoding requires careful preservation to maintain accurate relative positional relationships. Thus, an effective condensation strategy must treat these two components differently. Based on the above insights, we propose Latent-Condensed Attention ( LCA ), an efficient attention mechanism that performs structured context condensation natively within latent space. LCA partitions context into groups and compresses each group into a representative latent vector via weighted pooling for the semantic component \mathbf{C}^{KV} , while preserving positional information by selecting the token with the highest relevance within each group for \mathbf{K}^{R} . In this way, LCA reduces both the KV cache size and the attention complexity, without introducing additional parameters. We further provide a theoretical guarantee that the approximation error is uniformly bounded, independent of context length. Crucially, the decoupled condensation strategy is not specific to MLA; it generalizes to other attention mechanisms such as GQA, as we validate in Section 5.4 . We summarize our contributions as follows: • We propose Latent-Condensed Attention (LCA), an efficient attention mechanism that performs structured context condensation directly in MLA’s latent space, reducing both KV cache and attention computations, instead of sparsification only on reconstructed KV. • We design a latent-space condensation strategy that decouples semantic and positional processing: semantic information is adaptively aggregated via query-aware weighted pooling, while positional fidelity is preserved through hard anchor selection. This design avoids the signal-blending issue in conventional approaches. • LCA can be easily integrated into pretrained models via lightweight fine-tuning, requiring no additional parameters. Experiments demonstrate that LCA delivers up to 2.5 \times prefill speedup and 90% KV cache reduction at 128K context length while maintaining comparable performance. 2 Related Works Efficient Attention . To reduce the memory footprint of the KV cache, several attention variants have been proposed. Multi-Query Attention (MQA) 30 shares a single key and value head across all query heads, while Grouped-Query Attention (GQA) 1 groups query heads to share key-value heads, improving efficiency while maintaining quality. More recently, Multi-head Latent Attention (MLA) 9 compresses KV states into a low-dimensional latent space, significantly reducing per-token cache size. It has been widely adopted in state-of-the-art long-context models 9 ; 26 ; 15 ; 21 ; 22 . However, these methods retain the quadratic computational complexity of standard attention. Besides, sparse attention methods 39 ; 4 ; 40 ; 6 reduce the computational cost by skipping some tokens or computation blocks. Early methods employ fixed sparse patterns 39 ; 11 ; 4 ; 36 , which lack adaptability. To introduce adaptability, DuoAttention 35 learns head-specific static sparse patterns during an additional offline training phase. However, static patterns cannot adjust to the varying importance of tokens across different contexts. Recent dynamic sparse attention methods 24 ; 20 ; 35 selectively compute only a subset of attention blocks. Unlike prior work that treats KV cache reduction and attention sparsity separately, LCA operates directly on MLA’s compressed latent codes. By condensing information in the low-rank space, it avoids the overhead of full-dimensional reconstruction while simultaneously reducing both memory footprint and computational complexity. System-Level Optimizations for Long-Context Inference . Another line of work focuses on kernel-level optimizations and cache management for long-context inference. FlashAttention 8 and its successors speed up attention by optimizing memory access patterns. PagedAttention 23 manages the KV cache more efficiently by allowing non-contiguous storage, reducing memory fragmentation. Token-level eviction policies 25 ; 16 ; 27 retain only the most salient tokens based on attention scores, while query-aware cache selection methods 32 learn to select a compact set of KV pairs for each query. These techniques are largely orthogonal to our approach. 3

Read the complete text