Layer-wise dynamic rank for compressing large language models

Zhendong Mi Stevens Institute of Technology [email protected] Bian Sun Carnegie Mellon University [email protected] and Grace Li Zhang Technical University of Darmstadt [email protected] Shaoyi Huang Stevens Institute of Technology [email protected] · 2025

Read Layer-wise dynamic rank for compressing large language models by Zhendong Mi Stevens Institute of Technology [email protected] Bian Sun Carnegie Mellon University [email protected] and Grace Li Zhang Technical University of Darmstadt [email protected] Shaoyi Huang Stevens Institute of Technology [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25622.txt.

Introduction

Big language models are costly to store and run, so engineers often shrink them after training. Zhendong Mi and three co-authors present D-Rank in 2025, a way to decide how much each layer of a model can be squeezed. Most older methods squeezed every layer by the same share, though middle layers tend to hold richer information.

D-Rank scores each weight table for how much real content it carries, then hands the most room to the densest groups under one fixed budget. It also rebalances the shares across attention layers. In tests on several models, D-Rank beat three rival methods, cutting error sharply at mild squeeze rates. The gains held across model sizes and settings.

Catalog record

CREATORZhendong Mi Stevens Institute of Technology [email protected] Bian Sun Carnegie Mellon University [email protected] and Grace Li Zhang Technical University of Darmstadt [email protected] Shaoyi Huang Stevens Institute of Technology [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layer-wise dynamic rank for compressing large language models Zhendong Mi Stevens Institute of Technology [email protected] Bian Sun Carnegie Mellon University [email protected] and Grace Li Zhang Technical University of Darmstadt [email protected] Shaoyi Huang Stevens Institute of Technology [email protected] Abstract Large language models (LLMs) have rapidly scaled in size, bringing severe memory and computational challenges that hinder their deployment. Singular Value Decomposition (SVD)-based compression has emerged as an appealing post-training compression technique for LLMs, yet most existing methods apply a uniform compression ratio across all layers, implicitly assuming homogeneous information included in various layers. This overlooks the substantial intra-layer heterogeneity observed in LLMs, where middle layers tend to encode richer information while early and late layers are more redundant. In this work, we revisit the existing SVD-based compression method and propose D-Rank, a framework with layer-wise balanced D ynamic Rank allocation for LLMs compression. We first introduce effective rank as a principled metric to measure the information density of weight matrices, and then allocate ranks via a Lagrange multiplier-based optimization scheme to adaptively assign more capacity to groups with higher information density under a fixed compression ratio. Moreover, we rebalance the allocated ranks across attention layers to account for their varying importance and extend D-Rank to latest LLMs with grouped-query attention. Extensive experiments on various LLMs with different scales across multiple compression ratios demonstrate that D-Rank consistently outperforms SVD-LLM, ASVD, and Basis Sharing, achieving more than 15 lower perplexity with LLaMA-3-8B model on C4 datasets at 20% compression ratio and up to 5% higher zero-shot reasoning accuracy with LLaMA-7B model at 40% compression ratio while achieving even higher throughput. 1 Introduction As large language models (LLMs) expand in both scale and deployment, their associated computational and environmental costs continue to escalate ( Fernandez et al. 2025 ) . For example, a 30B-parameter model (e.g., LLaMA-30B) requires about 66GB for FP16 weights, which exceeds the capacity of a single GPU and forcing the adoption of model parallelism across multiple GPUs ( Touvron et al. 2023a ) . And a 176 billion parameter model BLOOM running on Google Cloud received 230,768 queries over 18 days, using an average of 40.32 kWh per day (roughly equivalent to 1,110 smartphone charges), demonstrating the substantial energy requirements for model inference at scale ( Luccioni et al. 2024 ; Luccioni et al. 2023 ) . To mitigate these costs, model compression techniques (e.g., pruning ( Sun et al. 2024 ; Zhang et al. 2024 ; Ling et al. 2024 ; Frantar & Alistarh 2023 ; Petri et al. 2023 ) , quantization ( Sun et al. 2023 ; Zhao et al. 2024 ; Xiao et al. 2023 ; Lin et al. 2024 ; Ashkboos et al. 2024b ) , knowledge distillation ( Gu et al. 2024 ; Magister et al. 2023 ; Jiang et al. 2023b ; Qiu et al. 2024 ) ) that reduce computational and storage demands while preserving model accuracy have been extensively employed to facilitate more efficient LLM deployment. Although effective, these approaches typically require time-consuming retraining process and specialized hardware configurations (e.g., 2:4 semi-structured deployment for GPU-based pruning), creating practical deployment bottlenecks ( Li et al. 2023 ; Ma et al. 2023 ) . As an effective solution to these limitations, compression techniques such as low-rank adaptation with Singular Value Decomposition (SVD) ( Meng et al. 2024 ) or QR decomposition ( Saha et al. 2024 ) have been extensively employed in LLM deployment ( Bałazy et al. 2025 ) . In SVD-based low-rank adaptation, each weight matrix is approximated through decomposition into three matrices with much smaller dimensions ( Yuan et al. 2025 ; Wang et al. 2025b ) . After the decomposition, matrix multiplications are performed on the lower-dimensional factors rather than the original full matrix, resulting in substantial parameter reduction and improved storage and computation efficiency while preserving model performance comparable to the original full-rank model. Typically, a weight matrix W\in\mathbb{R}^{m\times n} can be approximated as W\approx U_{k}\Sigma_{k}V_{k}^{\top} , where k<\min(m,n) denotes the retained rank. A larger compression ratio will lead to a smaller k . Despite the benefits and popularity of SVD-based low-rank adaptation in practical LLM deployment scenarios, there has been limited research on how to define an effective metric to quantify the information content in weight matrices, and subsequently attain dynamic ranks by leveraging the differences in information content between intra-layer attention matrices and cross-layer matrices, which therefore can maximize information preservation under a given overall compression ratio. In this work, we observe and identify several bottlenecks that hinder the efficiency of current SVD-based low-rank adaptation approaches: 1) limited effort has been devoted to designing effective metrics for measuring weight information content to determine optimal retained ranks k for each weight, leading to suboptimal compression performance; 2) existing approaches maintain uniform compression ratios across weight matrix types, failing to account for the substantial differences in their information density and inherent complexity; 3) in the latest LLMs with grouped-query attention, compression techniques such as grouping weight matrices across layers for joint compression may become ineffective due to substantial reduction in the column dimension of the W_{K} and W_{V} weight matrices compared with Multi-Head-Attention-based architectures (MHA), yet there is a lack of explanation for the underlying reasons as well as corresponding optimization strategies. To address the bottleneck , we develop the layer-wise dynamic rank for SVD-based LLMs compression. Specifically, we propose a metric, effective rank, for measuring information density of weight matrices. Subsequently, the effective rank will be employed to guide us in dynamically adjusting the retained rank for different types of weight matrices across different layers. To further improve the compression performance, we reallocate the preserved ranks across matrix types for attention layers by transferring part of the budget from matrices with lower information density to ones with higher information density while the same the overall target compression ratio. The main contributions of this work can be summarized as follows: • We propose D-Rank , a layer-wise d ynamic rank allocation approach for compressing LLMs. This approach enables us to preserve more information in large language models under the same compression budget, thereby achieving superior compression performance. • We introduce a novel metric, effective rank, to quantify the information density of each grouped layer in LLMs. Moreover, we develop a Lagrangian multiplier-based framework that dynamically allocates ranks across grouped layers according to their effective rank, aiming to improve the information preserved in the models. • Through effective rank analysis, we discover that the effective rank distribution among the attentions matrices is highly unbalanced: W^{Q},W^{K} have lower effective rank (less information) than W^{V} matrix. To address this issue, we propose a reallocation strategy that transfers part of the preserved rank budget from W^{Q},W^{K} to W^{V} . • Moreover, we analyze the reason why the performance of latest models (e.g., LLaMA-3) with grouped-query attention degrades in the state-of-the-art works, and we further demonstrate the effectiveness of D-Rank on the models. Extensive experiments on the LLaMA, LLaMA-2, LLaMA-3, and Mistral families show that D-Rank consistently outperforms baselines, achieving more than 15 lower perplexity on LLaMA-3-8B model with 20% compression ratio on C4 datasets, and up to 5% higher accuracy on zero-shot reasoning tasks with LLaMA-7B model at 40% compression ratio, while

Read the complete text