A 3 : an A nalytical Low-Rank A pproximation Framework for A ttention
Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, Christos-Savvas Bouganis, George Anthony Constantinides, Wayne Luk, Yiren Zhao · 2025
Read A 3 : an A nalytical Low-Rank A pproximation Framework for A ttention by Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, Christos-Savvas Bouganis, George Anthony Constantinides, Wayne Luk, Yiren Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12942.txt.
Introduction
Shrinking giant language models by low-rank cuts usually trims each layer blind to the whole design. Jeffrey Wong and co-authors add that design sense in a 2025 paper. Their framework, A3, splits each layer into three working parts.
Closed-form cuts shrink the hidden width inside each part. The cuts cost no extra run time or memory moves. A shrunk 70-billion model hit a perplexity of 4.69. The prior best method sat at 7.87 under the same budget. The trick also stacks with quantization and cache cuts. Mixed ranks per layer add one more dial.
Catalog record
| CREATOR | Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, Christos-Savvas Bouganis, George Anthony Constantinides, Wayne Luk, Yiren Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12942
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A 3 : an A nalytical Low-Rank A pproximation Framework for A ttention Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, Christos-Savvas Bouganis, George Anthony Constantinides, Wayne Luk, Yiren Zhao Abstract Large language models have demonstrated remarkable performance; however, their massive parameter counts make deployment highly expensive. Low-rank approximation offers a promising compression solution, yet existing approaches have two main limitations: (1) They focus on minimizing the output error of individual linear layers, without considering the architectural characteristics of Transformers, and (2) they decompose a large weight matrix into two small low-rank matrices. Consequently, these methods often fall short compared to other compression techniques like pruning and quantization, and introduce runtime overhead such as the extra GEMM kernel launches and memory operations for decomposed small matrices. To address these limitations, we propose A 3 , a post-training low-rank approximation framework. A 3 splits a Transformer layer into three functional components, namely QK , OV , and MLP , and provides analytical solutions that reduces the hidden dimension size inside each component while minimizing the component’s functional loss. This approach directly reduces model sizes, KV cache sizes, and FLOPs without introducing any runtime overheads. Through extensive experiments, we show that A 3 maintains superior performance compared to SoTAs. For example, under the same reduction budget in computation and memory, our low-rank approximated LLaMA 3.1-70B achieves a perplexity of 4.69 on WikiText-2, outperforming the previous SoTA’s 7.87 by 3.18. We also showcase versatile applications of A 3 in KV cache compression, integration with quantization, fine-tuning and mixed-rank assignments. We open-sourced our framework and code at https://github.com/DeepWok/a3 . † † affiliationnotice: Equal contribution 1 Introduction Large language models (LLMs) have shown exceptional performance in various applications, including language understanding, code completion, and reasoning tasks ( Vaswani et al., 2017 ; Brown et al., 2020 ; Chen et al., 2021a ; Wei et al., 2022 ) . However, these models usually contain billions of parameters, resulting in high computational costs and memory requirements. Linear layers and the attention mechanism contribute significantly to the model size and computational complexity, while the KV cache produced during generation further exacerbates the memory burden. Low-rank approximation is a promising technique that breaks down a matrix into smaller sub-matrices, directly reducing computational complexity and memory usage without the need of additional specialized hardware support. Usually a trained linear layer {\bm{W}}\in{\mathbb{R}}^{m\times n} is approximated by \widetilde{{\bm{W}}}_{r}={\bm{A}}_{r}{\bm{B}}_{r} , where {\bm{A}}_{r}\in{\mathbb{R}}^{m\times r} and {\bm{B}}_{r}\in{\mathbb{R}}^{r\times n} are two rank- r matrices with r\ll m,n . At inference time, the original GEMM operation {\bm{X}}{\bm{W}} is replaced by two smaller GEMM operations {\bm{X}}{\bm{A}}_{r} and ({\bm{X}}{\bm{A}}_{r}){\bm{B}}_{r} . The challenge is to construct the optimal {\bm{A}}_{r} and {\bm{B}}_{r} that maintains the end-to-end model performance. Recent studies show that minimizing the layer output error instead of the weight error gives better model performance ( Zhang et al., 2024 ; Zhang et al., ; Mozaffari and Dehnavi, 2024 ) , thus various activation-aware methods have been proposed, such as SVD-LLM ( Wang et al., 2025c ) , ASVD ( Yuan et al., 2023 ) , FWSVD ( Hsu et al., 2022 ) . However, these methods usually target general linear layers, which ineffectively save the FLOPs and memory proportional to \frac{m+n}{mn}r (Note that \frac{m+n}{mn}r<r for any m,n>2 ). Moreover, these methods rarely consider the architectural characteristics of Transformer, and suffer from severe performance degradation when compared to pruning and quantization. To address these limitations, we propose A 3 , a new analytical framework for post-training low-rank approximation. A 3 splits the Transformer architecture into three functional components: query-key ( QK ) component, output-value ( OV ) component, and multi-layer perceptron ( MLP ) component, and minimizes the functional loss of each component. This enables better end-to-end model performance than minimizing the error of individual linear layer outputs. We highlight the following contributions of A 3 : • We propose a three-part low-rank approximation setup for multi-head attention (MHA), which formulates the problem as three separate objectives: minimizing the functional loss of (1) QK ’s attention score, (2) OV ’s attention output, and (3) MLP ’s layer output. • We derive closed-form solutions for the three objectives, which reduces the hidden dimensions shared within each component: QK head dimension, OV head dimension, and MLP intermediate size. This naturally reduces model sizes, KV cache sizes, FLOPs, and avoids runtime overheads like extra GEMM operations. Moreover, A 3 trims both FLOPs/memory and information energy proportionally to the rank r , enabling a more effective trade-off between model performance and hardware efficiency. • We have adapted A 3 for use with diverse Transformer architectures, including group query attention (GQA) and rotary position embedding (RoPE), which allows the application of A 3 across a broad spectrum of models. This overcomes the limitation of existing low-rank approximation methods, which can only be applied on the vanilla MHA architecture. • We conduct extensive experiments on various LLMs, and show that A 3 outperforms SoTA low-rank methods by a significant margin. For example, our compressed LLaMA 3.1-70B achieves a perplexity of 4.69 on WikiText-2, outperforming SoTA method’s 7.87 by 3.18. We also demonstrate further applications of A 3 , such as its effectiveness in KV cache reduction, combination with quantization, and mixed-rank assignments for better performance. 2 Related Work Transformer and its variants The attention layer and the multi-layer perceptron layer (MLP) are two main modules of the Transformer architecture. In vanilla multi-head attention (MHA) ( Vaswani et al., 2017 ) , QK component computes the attention scores as follows: {\bm{A}}^{\prime}_{i}=\mathrm{softmax}({{\bm{A}}_{i}}/{\sqrt{{d_{\text{qk}}}}})=\mathrm{softmax}({{\bm{Q}}_{i}{\bm{K}}_{i}^{T}}/{\sqrt{{d_{\text{qk}}}}})\;. (1) where {\bm{A}}_{i} is the pre-softmax attention score of i -th head, and {d_{\text{qk}}} is the head dimension shared by {\bm{Q}}_{i} and {\bm{K}}_{i} . The post-softmax attention score is then multiplied with the value and summed over all heads to form the attention output: {\bm{O}}=\sum_{i}^{{h_{\text{q}}}}{\bm{A}}^{\prime}_{i}{\bm{V}}_{i}{{\bm{W}}_{\text{o},i}}\;. (2) Note that there is a head dimension {d_{\text{vo}}} shared by {\bm{V}}_{i} and {{\bm{W}}_{\text{o},i}} . In practice, {{\bm{W}}_{\text{o},i}} are usually concatenated as a single linear layer, {\bm{W}}_{o}=\left[{\bm{W}}_{o,1}^{T},{\bm{W}}_{o,2}^{T},\dots,{\bm{W}}_{o,{h_{\text{q}}}}^{T}\right]^{T} . The classic MLP in a Transformer has two linear layers with a ReLU activation function in between: {{\bm{X}}_{\text{d}}}=\text{ReLU}({{\bm{X}}_{\text{mlp}}}{{\bm{W}}_{\text{u}}}),\quad{\bm{Y}}_{\text{mlp}}={{\bm{X}}_{\text{d}}}{{\bm{W}}_{\text{d}}}\;. (3) {{\bm{W}}_{\text{u}}} and {{\bm{W}}_{\text{d}}} scale the input dimension {d_{\text{m}}} to the intermediate dimension {d_{\text{inter}}} and back to {d_{\text{m}}} . Following the vanilla MHA, numerous Transformer variants have been proposed ( Ainslie et al., 2023 ; Shazeer, 2019 ; Liu et al., 2024 ) . In recent models, the 2-layer MLP is usually replaced with a 3-layer variant ( Shazeer, 2020 ) : \displaystyle{\bm{Y}}_{\text{g}} \displaystyle={{\bm{X}}_{\text{mlp}}}{{\bm{W}}_{\text{g}}},\quad{\bm{Y}}_{\text{u}}={{\bm{X}}_{\text{mlp}}}{{\bm{W}}_{\text{u}}}, (4) \displaystyle{{\bm{X}}_{\text{d}}} \displaystyle=\mathrm{SiLU}({\bm{Y}}_{\text{g}})\otimes{\bm{Y}}_{\text{u}},\quad{\bm{Y}}_{\text{mlp}}={{\bm{X}}_{\text{d}}}{{\bm{W}}_{\text{d}}}\;. where \otimes denotes element-wise multiplication. {{\bm{W}}_{\text{u}}} and {{\bm{W}}_{\text{g}}} upscale the input dimension {d_{\text{m}}} to {d_{\text{inter}}} and {{\bm{W}}_{\text{d}}} downscale it back to {d_{\text{m}}} . Group query attention (GQA) ( Ainslie et al., 2023 ) is another widely adopted variant sharing a reduced number of key and value heads among groups of query heads. Besides, many LLMs adopt rotary positional embedding (RoPE) ( Su et al., 2024 ) . Figure 1 : High-level overview of A 3 . A 3 performs a low-rank approximation on each QK , OV , and MLP component, reducing the head dimensions in QK and OV , and the intermediate dimension in MLP . Inspired by ( Elhage et al., 2021 ) , we split the Transformer layer into three key components: QK