MGAA: Multi-Granular Adaptive Allocation for Low-Rank Compression of LLMs

Guangyan Li, Yongqiang Tang, Wensheng Zhang · 2025

Read MGAA: Multi-Granular Adaptive Allocation for Low-Rank Compression of LLMs by Guangyan Li, Yongqiang Tang, Wensheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03294.txt.

Introduction

Guangyan Li, Yongqiang Tang, and Wensheng Zhang present a smarter way to compress large language models by shrinking their weight matrices unevenly. Their paper came out in 2025. Compression matters because these models are too large to run cheaply, and low-rank approximation, which replaces a matrix with a simpler near copy, is a leading tool.

Most methods compress every matrix by the same ratio, ignoring that some matter more. Their method assigns ratios between layers based on how much each layer changes its input. Inside each layer, it assigns ratios by how energy is distributed across the matrix, keeping a consistent share of that energy. The method needs no task-specific testing during compression. It beat rivals across several models and also improved a multimodal model.

Catalog record

CREATORGuangyan Li, Yongqiang Tang, Wensheng Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MGAA: Multi-Granular Adaptive Allocation for Low-Rank Compression of LLMs Guangyan Li, Yongqiang Tang, Wensheng Zhang Abstract The enormous parameter scale of large language models (LLMs) has made model compression a research hotspot, which aims to alleviate computational resource demands during deployment and inference. As a promising direction, low-rank approximation technique has made remarkable achievements. Nevertheless, unfortunately, the vast majority of studies to low-rank approximation compression generally apply uniform compression ratios across all weight matrices, while disregarding their inherently differentiated impacts on the model’s performance. Although a few recent work attempts to employ heuristic search strategies to achieve the optimal parameter allocation, such strategies are computationally inefficient and lose the generalization ability in the era of LLMs. In this study, we propose a novel parameter Multi-Granular Adaptive Allocation (MGAA) method, which can adaptively allocate parameters between and within sublayers without task-specific evaluations in the compression process. MGAA consists of two components: 1) Among different sublayers, it assigns compression ratios based on their cosine similarity between inputs and outputs, allowing for a more tailored compression in sublayers with varying degrees of importance, and 2) Within each sublayer, it allocates different compression ratios to weight matrices based on their energy distribution characteristics, ensuring a consistent energy retention ratio while optimizing compression efficiency. Comprehensive evaluations of MGAA across multiple LLMs backbone models and benchmark datasets demonstrate its superior performance. Additionally, we apply our MGAA to multimodal model LLaVA, exhibiting remarkable performance improvements. More importantly, MGAA could serve as a versatile plug-and-play solution that can be seamlessly integrated with various low-rank approximation techniques. Its consistent performance improvements across existing low-rank compression methods further validate its broad applicability and strong generalization capability. I Introduction The advent of Large Language Models (LLMs) [ 1 , 2 , 3 ] has revolutionized natural language processing, achieving unprecedented performance across various tasks [ 4 ] . However, this remarkable performance comes with substantial computational costs: modern LLMs require hundreds of billions of parameters and massive computational resources for inference. Such resource demands pose significant challenges for real-world deployment, particularly in resource-constrained environments and edge devices. To address these computational challenges, model compression for LLMs has emerged as a critical research direction. Current approaches primarily fall into four categories: 1) quantization [ 5 , 6 , 7 , 8 , 9 ] , which reduces numerical precision while maintaining model performance; 2) pruning [ 10 , 11 , 12 , 13 , 14 ] , which eliminates redundant parameters; 3) knowledge distillation [ 15 , 16 , 17 ] , which transfers knowledge to smaller models; and 4) low-rank approximation [ 18 , 19 , 20 , 21 ] , which decomposes weight matrices into lower-rank matrices. These approaches offer unique advantages and challenges in terms of compression ratio, inference speed, and deployment complexity. Among these approaches, low-rank approximation has gained significant attention due to its unique combination of theoretical guarantees and task-agnostic nature. This method not only provides clear mathematical insights into the compression process but also achieves substantial parameter reduction. The application of low-rank approximation to LLMs has evolved through several key stages. Initial efforts focused on basic matrix decomposition, where SVD was applied to compress BERT models [ 22 ] . Subsequently, task-specific approaches emerged: Drone [ 23 ] introduced simultaneous decomposition of weight matrices and input activations, while FWSVD [ 24 ] leveraged task-specific gradient information to identify and preserve critical weights. However, these methods’ dependence on specific tasks limited their applicability to LLMs, where preserving general capabilities is crucial. To address this limitation, recent methods have shifted towards task-agnostic approaches. ASVD [ 19 ] , AWSVD [ 25 ] , and SVD-LLM [ 20 ] utilize task-agnostic input activations to weight matrices before applying low-rank approximation. In contrast to these weight-space methods, AFM [ 18 ] proposed a novel feature-space compression paradigm using PCA on output features, demonstrating scalability through successful compression of a 16B model in LORD [ 26 ] . Despite the significant advances in low-rank approximation compression techniques, the challenge of efficient parameter allocation across the entire model during compression remains largely unaddressed. Actually, recent studies [ 10 , 27 , 19 , 18 ] have revealed that different layers and weight matrices within the model exhibit varying degrees of importance to model performance. This observation suggests that a differentiated parameter allocation strategy could substantially enhance both the performance and generalization capability of compressed model. Unfortunately, current research to low-rank approximation compression typically apply uniform compression ratios across all weight matrices, while disregarding their inherent differences in importance. More recently, a few works [ 18 , 19 ] attempt to employ the heuristic search strategy, which determine differentiated compression ratios based on performance degradation in downstream validation tasks: weight matrices inducing larger performance drops receive lower compression ratios, while those causing minimal impact are compressed more aggressively. Despite the impressive performance improvement, this approach presents two critical limitations. First, it requires extensive evaluations on validation sets, making it computationally prohibitive for LLMs. Second, the resulting compression strategy becomes inherently biased towards specific validation tasks, potentially compromising the model’s general capabilities, particularly its zero-shot performance across diverse tasks. To address these limitations, we propose a novel parameter Multi-Granularity Adaptive Allocation (MGAA) method for low-rank compression, which operates at both inter- and intra-sublayer levels. At the sublayer level, we employ cosine similarity to estimate the importance of each sublayer and strategically allocate compression ratios accordingly. This compression strategy preserves the model’s critical components during compression, allowing the compressed model to better maintain its performance. At the intra-sublayer level, we find that model performance remains consistent when different matrices maintain equivalent energy retention ratios. This insight leads us to propose an energy-balanced parameter allocation strategy based on energy distribution. We assign different compression ratios to weight matrices while ensuring consistent energy retention, determined by the eigenvalue distribution of each weight matrix. Note that, compared with the present parameter allocation rivals [ 18 , 19 ] , our MGAA owns several distinct advantages. First, MGAA effectively reduces compression time by only leveraging activation values with just two forward propagation. Besides, by using task-agnostic calibration data, MGAA avoids overfitting to specific downstream tasks, ensuring robust performance across diverse tasks and improving accuracy in zero-shot learning scenarios. More importantly, MGAA can serve as a plug-and-play parameter allocation module that can be directly combined with various low-rank approximation techniques. Our main contributions are summarized as follows: • We propose a sublayer-level compression ratio allocation strategy based on the cosine similarity between input and output features. This strategy adaptively allocates compression ratios to sublayers according to their importance, preserving the critical components of the model during compression. • Within each sublayer, we introduce an energy-balanced parameter allocation method based on energy distribution. This approach, founded on the relationship between energy preservation ratio and performance, ensures consistent energy retention proportions across different matrices during compression, resulting in superior performance. • We conduct comprehensive evaluations across multiple models (including LLaMA1, Vicuna, LLaMA3, Mistral) using zero-shot perplexity on the WikiText2, PTB, and C4 datasets, as well as seven common sense reasoning datasets. Additionally, we extend our approach to multimodal model LLaVA, which exhibits remarkable performance improvements across various compression ratios. • MGAA significantly

Read the complete text