GTA: Grouped-head latenT Attention

Luoyang Sun, Cheng Deng, Jiwen Jiang, Xinjian Wu, Haifeng Zhang, Lei Chen, Lionel M. Ni, Jun Wang · 2025

Read GTA: Grouped-head latenT Attention by Luoyang Sun, Cheng Deng, Jiwen Jiang, Xinjian Wu, Haifeng Zhang, Lei Chen, Lionel M. Ni, Jun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17286.txt.

Introduction

Luoyang Sun and co-authors slim the attention step in large language models by sharing maps and compressing values, in work from twenty twenty-five. Attention drives model quality but eats memory and compute, and the stored key value cache grows fast with text length. Much of that storage is redundant.

Their method, grouped head latent attention, reuses one attention map across several heads, shrinking the key cache. A learned decoder squeezes the value cache into a compact latent form. Against a common grouped query baseline, compute fell by up to about sixty percent and the cache by up to seventy percent. End to end generation ran about twice as fast, with no extra machinery.

Catalog record

CREATORLuoyang Sun, Cheng Deng, Jiwen Jiang, Xinjian Wu, Haifeng Zhang, Lei Chen, Lionel M. Ni, Jun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GTA: Grouped-head latenT Attention Luoyang Sun, Cheng Deng, Jiwen Jiang, Xinjian Wu, Haifeng Zhang, Lei Chen, Lionel M. Ni, Jun Wang Abstract Attention mechanisms underpin the success of large language models (LLMs), yet their substantial computational and memory overhead poses challenges for optimizing efficiency and performance. A critical bottleneck arises as KV cache and attention computations scale rapidly with text length, challenging deployment on hardware with limited computational and memory resources. We observe that attention mechanisms exhibit substantial redundancy, since the KV cache can be significantly compressed and attention maps across heads display high similarity, revealing that much of the computation and storage is unnecessary. Leveraging these insights, we propose G rouped-Head Laten T A ttention (GTA), a novel attention mechanism that reduces memory usage and computational complexity while maintaining performance. GTA comprises two components: (1) a shared attention map mechanism that reuses attention scores across multiple heads, decreasing the key cache size; and (2) a nonlinear value decoder with learned projections that compresses the value cache into a latent space, further cutting memory needs. GTA cuts attention computation FLOPs by up to 62.5% versus Grouped-Query Attention and shrink the KV cache by up to 70% , all while avoiding the extra overhead of Multi-Head Latent Attention to improve LLM deployment efficiency. Consequently, GTA models achieve a 2× increase in end-to-end inference speed, with prefill benefiting from reduced computational cost and decoding benefiting from the smaller cache footprint. 1 Introduction Large language models (LLMs) have revolutionized natural language processing, driving breakthroughs in text generation, reasoning, and contextual understanding brown2020language ; touvron2023llama . The attention mechanism, a core component of these models, enables selective focus on relevant parts of the input sequence, underpinning their expressive power vaswani2017attention . However, the memory and computational demands of attention, particularly the key-value (KV) cache in autoregressive generation, pose significant challenges for long-context scenarios and resource-constrained environments dao2022flashattention ; liu2023lostmiddlelanguagemodels . These bottlenecks limit the scalability of LLMs in practical applications, where memory efficiency and low-latency inference are critical. Prior efforts to mitigate attention-related challenges in large language models (LLMs) have led to several innovations. Multi-Head Attention (MHA) vaswani2017attention , the foundation of modern transformers, projects input sequences into multiple query, key, and value representations to capture diverse contextual patterns, but its KV cache scales poorly with sequence length, limiting long-context applicability. Multi-Query Attention (MQA) shazeer2019fast reduces memory usage by sharing a single key-value pair across heads, yet sacrifices expressivity. Grouped-Query Attention (GQA) ainslie2023gqa groups heads to balance efficiency and performance, but compromises attention granularity. Multi-head Latent Attention (MLA) liu2024deepseek compresses the KV cache while preserving representational capacity, but its high computational overhead restricts use in resource-constrained settings. Other methods, such as differential attention ye2025differentialtransformer and convolution-augmented attention golovneva2025multitokenattention , improve contextual focus, but often increase complexity. These approaches are limited by high computational overhead, inefficient KV cache storage, and compromised model performance, with no method optimizing all three simultaneously. To address this limitations, we propose Grouped-head latenT Attention (GTA) , a novel attention framework that optimizes memory usage and computational efficiency while preserving the expressive power of MHA. GTA introduces two key innovations, as detailed in our method. First, it employs a shared attention map mechanism, grouping query and key projections to reuse computations across heads, thereby reducing computational overhead while maintaining fine-grained attention patterns. Second, it leverages a nonlinear value decoder that compresses the value cache into a compact latent space, using a context-adaptive sigmoid gate to dynamically generate head-specific values shazeer2020gluvariantsimprovetransformer . This design, illustrated in our architectural diagrams, significantly reduces memory requirements compared to traditional attention mechanisms, enabling efficient inference without sacrificing model quality. By combining grouped projections with nonlinear decoding, GTA achieves robust expressivity, overcoming the trade-offs observed in GQA and MLA. In this paper, we show the design roadmap of GTA, and present experiments on GTA models ranging from 160M to 1B parameters. Not only the statistical validation of GTA’s efficiency is provided the practical evaluations of cache footprint and latency are also carried out. The contributions of this work are as follows: • Proposal of GTA, a novel attention mechanism that reduces self-attention computation by up to 62.5% and KV cache size by up to 70% while preserving expressive power through shared attention maps and nonlinear decoding. • Training of GTA models on large-scale corpora and validation of their performance, matching or surpassing GQA on benchmarks across model scales from 160M to 1B parameters. • Analysis of GTA’s inference speed in prefill and decode stages, demonstrating 2× throughput compared to GQA, validating its effectiveness for low-latency LLM deployment. By breaking the conventional trade-off between efficiency and expressivity, GTA paves the way for scalable, sustainable, and high-performance LLM deployment in various devices. • This paper record the attention mechanism design process, including detailed design introduction, analysis methods, and evaluation procedures, guiding future efficient attention designs. 2 Related Work Attention mechanisms are central to LLMs, enabling effective modeling of contextual dependencies vaswani2017attention . However, the KV cache in standard attention mechanisms scales linearly with sequence length, creating memory and computational bottlenecks dao2022flashattention . Recent research has developed dense attention variants to optimize KV cache usage through sharing or compression, aligning with GTA. We review these approaches, focusing on methods that share KV caches across heads or layers and those that use latent compression, positioning GTA’s contributions. Shared KV cache methods. Several methods reduce memory usage by sharing KV caches across heads or layers. MHA vaswani2017attention , the transformer baseline, uses independent KV caches for each head, resulting in high memory demands. MQA shazeer2019fast shares a single KV pair across all heads, significantly reducing memory but limiting expressivity. GQA ainslie2023gqa groups heads and shares KV pairs within each group, balancing efficiency and performance, as seen in LLaMA touvron2023llama . You Only Cache Once (YOCO) sun2024you employs a decoder-decoder architecture to cache KV pairs once, sharing them across layers via cross-attention, reducing memory while maintaining global attention. These methods trade off some expressivity for efficiency, which GTA addresses through its design. Latent attention mechanisms. Another approach compresses the KV cache using latent representations. MLA used in DeepSeek-V3 liu2024deepseek and PLM deng2025plmefficientperipherallanguage , compresses keys and values into a latent vector, achieving significant memory savings while preserving performance. Similarly, GTA uses a compressed latent value representation with a nonlinear decoder to generate head-specific values, enhancing expressivity with low computational costs. GTA’s nonlinear decoding, inspired by gated mechanisms like GLU shazeer2020glu and GLA yang2024gatedlinearattentiontransformers , distinguishes it by maximizing information density. Performance-focused attention. Some methods prioritize performance over efficiency. Multi-Token Attention (MTA) golovneva2025multitokenattention uses convolutions to enhance contextual interactions, and the Differential Transformer ye2025differentialtransformer employs dual softmax maps for sharper focus. These approaches improve accuracy but often increase computational complexity, unlike GTA’s efficiency-driven design. Comparison with zadouri2025hardwareefficientattentionfastdecoding The paper zadouri2025hardwareefficientattentionfastdecoding introduces Grouped Tied Attention, which reduces cache requirements by sharing key and value components, thereby increasing arithmetic intensity. Building on this, Grouped Latent Attention is proposed to enhance model parallelism through grouped operations on latent variables within the MLA framework. In contrast, G rouped-Head Laten T A ttention (GTA) proposed in this paper adopts a novel attention matrix sharing strategy combined with a nonlinear value decoding process. To our knowledge, this is the first

Read the complete text