Tucker Attention: A generalization of approximate attention mechanisms

Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer · 2026

Read Tucker Attention: A generalization of approximate attention mechanisms by Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-30033.txt.

Introduction

Klein and co-authors generalize how attention layers get approximated. Methods like grouped-query and latent attention use odd low-rank factorizations. What they truly approximate stays murky under classic theory. Their Tucker Attention views attention weights through one tensor factorization frame. It needs about ten times fewer parameters at matched validation scores. Grouped-query, latent, and standard attention all appear as special cases. The frame also exposes the real ranks those methods reach. MLA gets useful simplifications from the view.

Catalog record

CREATORTimon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75513
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.30033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tucker Attention: A generalization of approximate attention mechanisms Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer Abstract The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA. † † affiliationnotice: Equal contribution 1 Introduction The development of the attention mechanism ( Bahdanau et al. 2014 ) and its use in transformer architectures ( Vaswani et al. 2017 ) has significantly impacted deep learning. Transformer architectures, relying on multi-head attention (MHA) and related methods, however, incur substantial computational costs. MHA introduces a large number of parameters in the key, query, value, and output weight matrices, which induces substantial memory costs during pre-training when storing optimizer states and gradients. These memory costs manifest at multiple stages: during training through optimizer states and gradients, during inference through the KV-cache (whose size scales with the number of heads and head dimension), and during attention computation through GPU cache pressure in kernels like flash-attention ( Dao et al. 2022 ) . Designing compressed, parameter-efficient representations of attention weights that address all three bottlenecks simultaneously remains an open challenge. A main approach to reduce KV cache overhead and the overall parameter count in transformer architectures is to leverage parameter redundancies and the limited utility of individual attention heads. A large amount of work has studied such redundancies, which are summarized below. ( Voita et al. 2019 ) shows that many heads in multi-head attention contribute little and can be pruned, implying the existence of low-rank structure across heads. ( Brunner et al. 2020 ) demonstrates that portions of the attention weights may not affect the model output. ( Bhojanapalli et al. 2020 ) identifies head dimensions that induce a low-rank bottleneck preventing the representation of arbitrary context vectors. ( Behnke & Heafield 2020 ) proposes head pruning during training. ( Li et al. 2018 ) mitigates redundant learning across heads through disagreement regularization. ( Li et al. 2021 ) applies differentiable subset pruning to remove redundant head information. ( Michel et al. 2019 ) observes that some layers contain many redundant heads, whereas others rely heavily on a large head dimension. Several methods have been proposed to leverage parameter redundancies. Among the most widely used approaches is Multi-Query Attention (MQA), which employs a single head for the key and value projections ( Shazeer 2019 ) . MQA lowers the parameter count and enables efficient KV caching, where the cache size is independent of the number of attention heads. To trade off parameter efficiency and accuracy, Grouped-Query Attention (GQA) ( Ainslie et al. 2023 ) adopts a limited number of heads for the key and value projections, achieving improved accuracy over MQA. A further direction is Multi-Head Latent Attention (MLA) ( Liu et al. 2024 ) , which compresses the query, key, and value weights in low-rank format, resulting in reduced parameter counts. Moreover, the KV-cache of MLA that scales only with the low-rank latent dimension, instead of the per-head embedding dimension and number of heads. In recent years, several works have investigated the low-rank structure of attention weights through approaches similar to low-rank adapters (LoRA) for fine-tuning ( Hu et al. 2022 ) . ( Cordonnier et al. 2020 ) leverages low-rank structure in multi-head attention to enforce information sharing by learning a mixing vector that selects relevant information from shared projection maps. Related to the MLA approach, in ( Zhang et al. 2025 ) query, key, and value weights are individually represented through a low-rank canonical polyadic (CP) decomposition while maintaining compatibility with position encodings like rotational position embedding (RoPE) ( Su et al. 2023 ) . Similarly, ( Gu et al. 2025 ) represents query, key, value, and output weights as an order three tensor, and then stacks these as an order four tensor before factorizing the object in low-rank Tucker format. ( Meng et al. 2025 ) successfully represented GQA layers as MLA layers, which motivates further generalization. We remark that there is a wealth of approximate attention methods that avoid the quadratic cost in sequence length induced by MHA and similar methods, e.g., linear attention ( Wang et al. 2020 ) . These methods invoke a different efficiency to accuracy trade-off discussion that is out of scope for this paper. This includes sparse attention mechanisms ( Beltagy et al. 2020 ; Zaheer et al. 2020 ) that focus on sub-selecting chunks of the input sequence to enforce sparsity of the pre-softmax attention matrix. These methods are orthogonal to the proposed work and can be combined. 1.1 Contribution In this paper, we provide a principled, interpretable, and generalized view on the attention weight representation. Instead of inspecting and manipulating the query, key, value and output matrices individually , we analyze the pre-softmax attention weights (query and key) and the post-softmax attention weights (value and output) as standalone objects. The former constitutes a bilinear form on the token sequence that yields the pre-softmax attention matrix per head, and the latter a linear form to combine the post-softmax attention matrices to the attention output, relative to the current input sequence. Thus, pre- and post-softmax attention weights are each structured in an order three tensor object each. As a corollary we observe that MHA itself is a special factorization of this object. We ask the question: • Which tensor ranks does MHA assume in this context? • Instead of approximating MHA, as the listed related works do, is it more efficient to approximate the tensor structure directly? We find that MHA is the most efficient factorization if we assume pre- and post- softmax tensors to be of full rank. Under the reasonable assumption ( Voita et al. 2019 ; Brunner et al. 2020 ) that there is low-rank structure in the pre- and post-softmax tensors, we propose a direct low-rank Tucker decomposition of those objects, simply called Tucker Attention. This proposed Tucker Attention has two key features: • Expressive : MHA, MQA, GQA, and MLA are all special cases of Tucker Attention corresponding to specific Tucker ranks. This unifying view reveals that none of these methods compress the head mode of the attention tensor, leaving exploitable redundancies ( Section 2.2 ). • Informative : Unlike prior methods that target individual weight matrices, Tucker Attention enables independent

Read the complete text