Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Yoonjun Cho Soeun Kim Dongjae Jeon Kyelim Lee Beomsoo Lee Albert No, [0.5em] Department of Computer Science, Yonsei University · 2025
Read Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition by Yoonjun Cho Soeun Kim Dongjae Jeon Kyelim Lee Beomsoo Lee Albert No, [0.5em] Department of Computer Science, Yonsei University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02077.txt.
Introduction
Yoonjun Cho and colleagues improve a standard recipe to compress large language models. The recipe splits each weight matrix into a quantized part plus a low-rank fix. Past joint methods switch between quantization and low-rank steps. In practice those methods favor one part at the cost of the other. That wastes the strengths of each part.
Their method, Outlier-Driven Low-Rank Initialization, gives the low-rank part a set job. Its job is to capture weights that are sensitive to activations. That set task blunts the harm outliers do to quantization. It also balances the two parts better. Added to the joint frame, it cut error and quantization scale in tests each time. Perplexity and zero-shot accuracy rose in low-bit settings across Llama and Mistral models.
Catalog record
| CREATOR | Yoonjun Cho Soeun Kim Dongjae Jeon Kyelim Lee Beomsoo Lee Albert No, [0.5em] Department of Computer Science, Yonsei University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition Yoonjun Cho Soeun Kim Dongjae Jeon Kyelim Lee Beomsoo Lee Albert No, [0.5em] Department of Computer Science, Yonsei University Abstract Decomposing weight matrices into quantization and low-rank components ( 𝐖 ≈ 𝐐 + 𝐋𝐑 \mathbf{W}\approx\mathbf{Q}+\mathbf{L}\mathbf{R} ) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component’s unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers’ negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings. 1 Introduction Quantization ( Polino et al. 2018 ; Jacob et al. 2018 ; Nagel et al. 2021 ) and weight matrix factorization ( Golub et al. 1987 ; Saha et al. 2023 ) are two widely used techniques for compressing large language models (LLMs) to enable efficient inference on resource-constrained hardware. Post-training quantization (PTQ) reduces model size and computational cost by mapping high-precision weights to lower-bit representations ( Tao et al. 2022 ; Bai et al. 2022 ; Dettmers et al. 2022 ; Shao et al. 2024 ) , while matrix factorization approximates weight matrices with compact factored representations ( Li et al. 2023 ; Gao et al. 2024b ) . Recent joint optimization approaches combine these methods to achieve extreme compression, representing weights as the sum of a quantization matrix and a low-rank component: \mathbf{W}\approx\mathbf{Q}+\mathbf{L}\mathbf{R} ( Saha et al. 2024 ; Li et al. 2024b ; Guo et al. 2024 ) . Joint optimization approaches minimize representation error by iteratively alternating between quantization and low-rank approximation. These methods typically adopt either a quantize-first strategy ( Saha et al. 2024 ; Li et al. 2024b ) or a low-rank-first strategy ( Guo et al. 2024 ) . While these differ in iteration ordering, they can be equivalently understood as distinct initialization choices for the low-rank components: the quantize-first approach initializes \mathbf{L}\mathbf{R} to zero, while the low-rank-first approach initializes them using factorized weights. Critically, existing methods treat these different orderings merely as preparatory steps, assuming iterative updates will naturally converge to optimal solutions. However, viewing this process through the lens of initialization reveals an underexplored aspect of how initialization strategies might fundamentally affect weight decomposition quality. In this work, we investigate the role of initialization in joint optimization. Our analysis shows that different initializations lead to distinct solution spaces, with components maintaining persistent roles throughout optimization. Quantize-first methods treat low-rank components as error correction terms, while low-rank-first methods preserve them as the primary weight representation. This finding highlights that initialization fundamentally determines the role assignment between quantization and low-rank components, raising a key question: What is the optimal initialization strategy for decomposing weights into quantized and low-rank matrices? Recent works have shown that quantization errors are pronounced for weights associated with activation outliers, as extreme activations amplify weight sensitivity ( Dettmers et al. 2023b ; Lin et al. 2024 ; Lee et al. 2024 ; Huang et al. 2024 ; Kim et al. 2024 ) . Building on this insight, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns a specific role to the low-rank component to capture these salient weights while using the quantized matrix to express the residuals. By handling outlier-sensitive weights through the low-rank component, our approach stabilizes quantization and enables more precise weight decomposition. Through extensive experiments on Llama2 (7B, 13B, 70B) ( Touvron et al. 2023 ) , Llama3-8B ( Dubey et al. 2024 ) , and Mistral-7B ( Jiang et al. 2023 ) , we demonstrate that incorporating ODLRI into the joint optimization framework consistently improves perplexity and zero-shot accuracy across extreme low-bit settings. Our analysis shows that ODLRI reduces activation-aware error, minimizes quantization scale, and enhances model performance, highlighting the importance of structured initialization in low-bit quantization. These results present a principled approach for stable and efficient LLM compression, and provide new insights into the effective decomposition of quantization and low-rank matrices. We summarize our contributions as follows: • We propose a unified framework for expressing iterative joint optimization algorithms by introducing the concept of initialization of low-rank components. • We analyze the impact of initialization on iterative joint optimization algorithms, revealing the suboptimality of conventional approaches. • We propose Outlier-Driven Low-Rank Initialization (ODLRI), which assigns a specific role to the low-rank component \mathbf{L}\mathbf{R} to capture salient weights while using the quantization matrix \mathbf{Q} for the remaining weights. • Comprehensive experiments demonstrate that incorporating ODLRI reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy. 2 Preliminaries 2.1 Post-Training Quantization Early post training quantization (PTQ) methods ( Jacob et al. 2018 ; Nagel et al. 2021 ; Dettmers et al. 2022 ; Shao et al. 2024 ) mainly focus on minimizing the direct quantization error for a given weight matrix \mathbf{W}\in\mathbb{R}^{m\times n} , optimizing: \displaystyle\argmin_{\mathbf{Q}}\|\mathbf{W}-\mathbf{Q}\|^{2}_{\text{F}}, where \mathbf{Q} is obtained by rounding weights to the nearest discrete values ( Banner et al. 2019 ; Stock et al. 2020 ; Wu et al. 2020 ) . However, this naive rounding approach often results in significant accuracy degradation. This problem is particularly pronounced in large-scale LLMs, where even small perturbations in weights can propagate across layers, gradually accumulating and worsening errors. To mitigate this issue, activation-aware PTQ methods ( Dong et al. 2019 ; Nagel et al. 2020 ; Li et al. 2021 ; Li et al. 2024a ) incorporate a calibration dataset \mathbf{X}\in\mathbb{R}^{n\times d} to account for the interaction between weight quantization and activations: \displaystyle\argmin_{\mathbf{Q}}\|(\mathbf{W}-\mathbf{Q})\mathbf{X}\|^{2}_{\text{F}}. This formulation ensures that weight quantization preserves the statistical behavior of activations, and leads to improved model performance. OPTQ ( Frantar et al. 2023 ) refines activation-aware PTQ by introducing error feedback, which reduces cumulative quantization errors for effective low-bit quantization. QuIP ( Chee et al. 2023 ) and QuIP# ( Tseng et al. 2024 ) further improve robustness through incoherence processing, applying orthogonal transformations to \mathbf{W} and its local Hessian to reduce correlations, making 2-bit quantization more effective. In addition, recent studies show that effectively controlling a small portion of activation outlier-sensitive weights can significantly enhance quantization performance. SpQR ( Dettmers et al. 2023b ) retains these critical salient weights in higher precision, while AWQ employs per-channel scaling to protect them, effectively mitigating the adverse effects of activation outliers during quantization. 2.2 Weight Matrix Factorization Matrix factorization decomposes a matrix into low-rank components for efficient representation and computation ( Golub et al. 1987 ; Saha et al. 2023 ) . Recently, it has been applied to LLM compression by approximating weight matrices with low-rank representations ( Li et al. 2023 ; Gao et al. 2024b ) . Given a weight matrix \mathbf{W}\in\mathbb{R}^{m\times n} , these methods decompose it into lower-dimensional matrices, \mathbf{L}\in\mathbb{R}^{m\times r} and \mathbf{R}\in\mathbb{R}^{r\times n} , by minimizing: \displaystyle\argmin_{\mathbf{L},\mathbf{R}}\|\mathbf{W}-\mathbf{L}\mathbf{R}\|^{2}_{\text{F}}. By reducing the rank r , low-rank decomposition, such as