Large Language Model Compression via the Nested Activation-Aware Decomposition

Jun Lu, Tianyi Xu, Bill Ding, David Li, Yu Kang · 2025

Read Large Language Model Compression via the Nested Activation-Aware Decomposition by Jun Lu, Tianyi Xu, Bill Ding, David Li, Yu Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17101.txt.

Introduction

Lu, Xu, Ding, Li, and Kang compress large language models with a nested decomposition method in twenty twenty-five. Smaller models are cheaper to run and easier to deploy. Their method splits weight matrices into low rank parts after training. Two problems arise: activation patterns vary, and unseen data can surprise the model.

Their framework, called NSVD, reshapes weights using the activation distribution first. Outliers in the activations are absorbed into the transformed weights. That improves the accuracy of the low rank split. Tests covered eight data sets and six models from three model families. The method beat current state of the art approaches. Gains were largest at high compression and in multilingual or multitask settings.

Catalog record

CREATORJun Lu, Tianyi Xu, Bill Ding, David Li, Yu Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17101

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Model Compression via the Nested Activation-Aware Decomposition Jun Lu, Tianyi Xu, Bill Ding, David Li, Yu Kang Abstract In this paper, we tackle the critical challenge of compressing large language models (LLMs) to facilitate their practical deployment and broader adoption. We introduce a novel post-training compression paradigm that focuses on low-rank decomposition of LLM weights. Our analysis identifies two main challenges in this task: the variability in LLM activation distributions and handling unseen activations from different datasets and models. To address these challenges, we propose a nested activation-aware framework (NSVD) for LLMs, a training-free approach designed to enhance the accuracy of low-rank decompositions by managing activation outliers through transforming the weight matrix based on activation distribution and the original weight matrix. This method allows for the absorption of outliers into the transformed weight matrix, improving decomposition accuracy. Our comprehensive evaluation across eight datasets and six models from three distinct LLM families demonstrates the superiority of NSVD over current state-of-the-art methods, especially at medium to large compression ratios or in multilingual and multitask settings. \sidecaptionvpos figuret 1 Introduction By training on vast amounts of textual data, large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks. These models can learn rich linguistic structures and contextual information, performing excellently in various applications such as natural language understanding, text generation, question-answering systems, machine translation, sentiment analysis, and more (Brown et al., 2020 ; Thirunavukarasu et al., 2023 ; Achiam et al., 2023 ; Merchant et al., 2023 ; Lu et al., 2024 ) . As technology advances, the scale of LLMs continues to grow, leading to performance improvements. However, this also raises the requirements for computational resources and storage. Consequently, an important area of current research focuses on effectively compressing these models to reduce their resource consumption while maintaining high performance. This involves studying and applying various compression methods, such as neural network and transformer pruning (Frantar & Alistarh, 2023 ; Mirzadeh et al., 2023 ) , knowledge distillation (Tunstall et al., 2023 ) , weight quantization (Dettmers et al., 2022 ; Frantar et al., 2022 ; Lin et al., 2024 ) , and others, aiming to find the optimal balance between efficiency and effectiveness. Among these techniques, low-rank matrix decomposition remains relatively unexplored but holds great promise (Hsu et al., 2022 ; Yuan et al., 2023 ; Wang et al., 2024 ) . This technique involves approximating the weight matrices in neural networks or transformer structures with lower rank matrices, thereby reducing the overall model size. Given the vast number of parameters in LLMs, low-rank decomposition can significantly decrease memory usage. Moreover, it can enhance the efficiency of already compressed models by further compressing quantized or pruned models. Despite its potential, singular value decomposition (SVD) or other matrix decomposition for LLM compression has not been fully realized. Although some SVD-based methods like FWSVD, ASVD, and SVD-LLM have been proposed (Hsu et al., 2022 ; Yuan et al., 2023 ; Wang et al., 2024 ) , they still suffer from performance degradation at medium to high compression ratios or datasets with significantly different activations (i.e., “overfitting” happens during the post-training procedure). This limitation stems from two main issues: imprecise data preprocessing, where existing strategies fail to strike a balance between the activation-aware compression loss and the original matrix compression loss; and the unawareness of different activations for different model architectures, different datasets, or different tasks. This paper introduces the nested activation-aware framework (NSVD) , a nested SVD-based compression method for LLMs that addresses the aforementioned issues. NSVD features two key components: truncation-aware data whitening, which ensures a direct correlation between singular values and compression loss, allowing for minimal loss when truncating singular values; and a two-way decomposition for adhering to the original weight matrix. We evaluate NSVD against three other SVD-based LLM compression methods—standard SVD, and two other ASVD approaches (ASVD-0 and ASVD-I; which will be introduced in the sequel)—across eight datasets and six models from three LLM families (LLaMA, OPT, and Mistral) at different scales. Our findings highlight that NSVD consistently outperforms these methods across all tested scenarios, especially at medium to large compression ratios (30% to 50%) or in multilingual and multitask settings. 2 Related Work Low-rank approximation. In the context of low-rank matrix approximation, two types of problems emerge due to the interaction between rank and error: the fixed-precision approximation problem and the fixed-rank approximation problem . In the fixed-precision approximation problem, given a matrix \bm{A} and a tolerance \epsilon , the goal is to find a matrix \widetilde{\bm{A}} with rank r=r(\epsilon) such that \big{\lVert}\bm{A}-\widetilde{\bm{A}}\big{\rVert}\leq\epsilon in an appropriate matrix norm. Conversely, in the fixed-rank approximation problem, one seeks a matrix \widetilde{\bm{A}} with a fixed rank k that minimizes the error \big{\lVert}\bm{A}-\widetilde{\bm{A}}\big{\rVert} (Kishore Kumar & Schneider, 2017 ; Martinsson, 2019 ; Lu, 2022b ) . This paper focuses on the latter. Pruning and quantization. The compression of LLMs has become an essential area of research aimed at mitigating their significant computational and memory demands (Yuan et al., 2023 ; Wang et al., 2024 ) . Driven by this need, various strategies have emerged, broadly categorized into weight quantization (Dettmers et al., 2022 ; Frantar et al., 2022 ; Lin et al., 2024 ) , network and transformer pruning (Frantar & Alistarh, 2023 ; Mirzadeh et al., 2023 ) , knowledge distillation (Tunstall et al., 2023 ) , and low-rank approximation (Hsu et al., 2022 ; Yuan et al., 2023 ; Wang et al., 2024 ) . Among these, post-training methods that avoid resource-intensive retraining processes are particularly noteworthy, including unstructured pruning, structured pruning, quantization, and low-rank approximation. Unstructured pruning sets individual weights to zero without changing the model’s shape, exemplified by SparseGPT (Frantar & Alistarh, 2023 ) , which prunes less important weight elements using the inversion of the Hessian matrix. However, the irregular sparsification achieved through unstructured pruning often fails to achieve the desired speedup or memory savings. Structured pruning, in contrast, removes entire channels or components from LLMs, making it easier to implement on hardware but potentially leading to significant accuracy degradation, especially under high compression ratios as seen with LLM-Pruner (Ma et al., 2023 ) . Quantization techniques reduce the precision of weight matrices, offering limited compression options typically ranging from 3 to 8 bits, which may not fully utilize available memory budgets. GPTQ is a notable example that uses layer-wise quantization and updates weights with inverse Hessian information (Frantar et al., 2022 ) . Low-rank approximation for LLMs. Despite the popularity of low-rank factorization as a neural network compression technique, its application in LLMs remains relatively underexplored. This gap is addressed through the introduction of novel low-rank decomposition methods designed specifically for LLMs, such as ASVD (Yuan et al., 2023 ) . To be more specific, standard SVD focuses on compressing original weight matrices without considering parameter importance, which can result in larger compression errors. To address this, FWSVD incorporates Fisher information to weigh parameter importance, ASVD scales the weight matrix by a diagonal matrix representing input channel impact to account for activation distribution effects, while SVD-LLM further scales the weight matrix by a Cholesky decomposition of the activation matrix to upper bound the reconstruction error (Hsu et al., 2022 ;

Read the complete text