CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition

Lin Xv, Jingsheng Gao, Xian Gao, Ting Liu, Yuzhuo Fu · 2025

Read CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition by Lin Xv, Jingsheng Gao, Xian Gao, Ting Liu, Yuzhuo Fu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19385.txt.

Introduction

Lin Xv and three co-authors present Duo-SVD, a training-free way to compress large language models by low-rank decomposition that treats different parts of a weight matrix unequally. Standard singular value methods spread their error unevenly across columns and modules, yet they judge all pieces by the same rule, which wastes accuracy.

Duo-SVD optimizes at two levels: a column-preserving strategy keeps the columns whose loss would hurt most. A module-level score judges each weight matrix's importance directly. Together they approximate the original model more faithfully at the same size, helping compressed models run on smaller hardware without retraining.

Catalog record

CREATORLin Xv, Jingsheng Gao, Xian Gao, Ting Liu, Yuzhuo Fu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19385

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition Lin Xv, Jingsheng Gao, Xian Gao, Ting Liu, Yuzhuo Fu Abstract Abstract The rapid advancement of Large Language Models (LLMs) faces a critical bottleneck in their immense size, necessitating efficient compression techniques. While Singular Value Decomposition (SVD) is a promising approach, existing SVD-based methods treat the entire parameter matrix uniformly, overlooking that SVD approximation errors vary significantly across different matrix parts, which often leads to suboptimal compression. To address this, we propose C olumn- P reserving S ingular V alue D ecomposition (CPSVD), a novel method that refines SVD-based LLM compression by intelligently segmenting the parameter matrix. Unlike traditional SVD, CPSVD identifies and directly preserves matrix columns with high decomposition errors, applying SVD only to columns with low decomposition errors, while precisely determining the optimal balance point between these two strategies to minimize error. Furthermore, leveraging the inherent heterogeneity in decomposition errors across different matrices within an LLM, CPSVD adaptively allocates non-uniform compression rates to modules within that layer, while adhering to a target layer-wise compression ratio, thereby further enhancing compression performance. Extensive experiments demonstrate that CPSVD consistently outperforms state-of-the-art SVD-based LLM compression methods, achieving lower perplexity and higher accuracy on zero-shot tasks. 1 Introduction Large Language Models (LLMs) have rapidly emerged as foundational technologies, demonstrating unprecedented performance across diverse tasks ranging from complex text generation and translation to sophisticated reasoning and problem-solving ( Wan et al. 2024 ) . This remarkable progress is largely attributed to the scaling laws ( Kaplan et al. 2020 ) , which have driven a relentless increase in model size, with leading LLMs such as GPT ( Brown et al. 2020 ) and LLaMA ( Touvron et al. 2023a ) now comprising tens to hundreds of billions of parameters. While their colossal scale unlocks extraordinary capabilities, the immense computational and memory demands of these models severely restrict their widespread adoption ( Chavan et al. 2024 ) , creating an urgent need for efficient model compression techniques ( Zhou et al. 2024 ) . Mainstream compression methods for LLMs include quantization ( Frantar et al. 2022 ; Xiao et al. 2023 ; Mozaffari and Dehnavi 2024 ) , pruning ( Frantar and Alistarh 2023 ; An et al. 2024 ; Fang et al. 2024 ) , knowledge distillation ( Gu et al. 2024 ; Zhong et al. 2024 ) , and low-rank factorization ( Hsu et al. 2021 ; Wang et al. 2024 ) . Among these, Singular Value Decomposition (SVD) has gained significant favor within low-rank factorization due to its inherent ability to reveal parameter importance and facilitate straightforward truncation. Methods like ASVD ( Yuan et al. 2024 ) , SVD-LLM ( Wang et al. 2024 ) , AdaSVD ( Li et al. 2025 ) and ResSVD ( Bai et al. 2025 ) have achieved notable progress. Figure 1: Workflow of CPSVD. (a) illustrates the module compression rate allocation: it first computes individual losses at a uniform compression rate, then determines each module’s rate based on relative loss and the layer’s overall compression ratio. (b) presents the column-preserving strategy: a workflow that includes decomposition, calculating and ranking column errors, and searching for the optimal number of columns to retain. Despite their specific implementation differences, a common characteristic of current SVD-based methods is that they leverage SVD to perform a global decomposition and approximation of the entire parameter matrix, thereby effectively preserving important information. However, we observe a critical phenomenon: different parts of a parameter matrix exhibit significantly varying sensitivity to SVD decomposition. After decomposition and truncation, some parts of the matrix may incur only minor truncation errors, while others suffer from non-negligible errors. This implies that a uniform decomposition of the entire parameter matrix can be costly and suboptimal. Furthermore, we find that different parameter matrices within an LLM also display considerable variations in their sensitivity to SVD decomposition, yet existing methods often apply the same compression ratio to different parameter matrices within the same Transformer layer, overlooking the optimization potential offered by these inter-matrix differences. Building upon the aforementioned insights into the limitations of existing SVD-based compression methods, we propose CPSVD, a novel approach that differentiates in its treatment of various parts of the parameter matrix. CPSVD utilizes the ’columns of a parameter matrix’ as its fundamental processing unit, ensuring that the compressed results retain a sufficiently good structure for practical compression benefits. For the SVD decomposition of a matrix W , W=U\Sigma V^{T}+W_{e} (where W_{e} is the truncation error matrix), CPSVD innovatively analyzes the sensitivity of different columns to decomposition based on the W_{e} . Specifically, columns highly sensitive to decomposition (i.e., those that would incur larger approximation errors) are directly preserved, while the remaining, less sensitive columns undergo SVD. To determine the optimal balance point between these two processing strategies, we further leverage the monotonicity of truncation errors to efficiently determine the optimal proportion between these two parts, aiming to minimize the overall approximation error. Furthermore, we observe that at the same compression ratio, different parameter matrices within an LLM exhibit significant variations in their compression errors. Based on this finding, CPSVD, while maintaining a consistent overall compression ratio for a given Transformer layer, strategically transfers or compensates portions of the compression burden from matrices with larger intra-layer compression errors to those with smaller errors. This approach further optimizes the overall compression error at the layer level, thereby enhancing model performance. Our key contributions are summarized as follows: • Column-Preserving Strategy : We propose an innovative matrix compression method that effectively balances compression ratio and performance by directly preserving critical information columns and performing low-rank decomposition on the rest. • Module-Wise Compression : Based on compression loss, we design an intra-layer adaptive scheme for allocating compression ratios, which significantly reduces the overall compression error. • Performance Advantage : Experiments show that our method outperforms existing SVD-based compression techniques on multiple models and datasets. 2 Related Works 2.1 SVD-based Model Compression SVD-based compression techniques reduce the number of parameters by decomposing a large matrix into three smaller matrices and retaining only the most significant components. Native SVD decomposition only focuses on the parameter matrix, ignoring data information, which leads to large errors. To address this, FWSVD ( Hsu et al. 2021 ) attempts to leverage an approximate Fisher information matrix to gauge parameter importance, but this method is primarily designed for smaller models and usually requires retraining to recover performance. In the era of Large Language Models, ASVD ( Yuan et al. 2024 ) was the first to propose an activation-aware, training-free SVD-based compression method. Building upon ASVD, SVD-LLM ( Wang et al. 2024 ) theoretically provides a method to obtain the minimum error for the product of parameters and activations. AdaSVD ( Li et al. 2025 ) employs an iterative optimization approach, alternately solving for the two decomposed low-rank matrices. ResSVD ( Bai et al. 2025 ) , on the other hand, focuses on further decomposing the residual matrix formed by the truncation. While these methods strive to reduce truncation error from different perspectives, they share a common limitation: they are all premised on performing SVD decomposition on the entire parameter matrix, overlooking the crucial characteristic that certain parts of

Read the complete text