CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
Yanxia Deng [email protected], Aozhong Zhang [email protected], Selcuk Gurses [email protected], Naigang Wang [email protected], Zi Yang [email protected], Penghang Yin [email protected] · 2025
Read CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization by Yanxia Deng [email protected], Aozhong Zhang [email protected], Selcuk Gurses [email protected], Naigang Wang [email protected], Zi Yang [email protected], Penghang Yin [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18475.txt.
Introduction
Yanxia Deng and five co-authors ease the tuning of slimmed language models in a 2025 paper. A popular tuning trick adds small trainable patches to a frozen model, which saves compute. Yet when the base model is first slimmed to fewer bits, its weights lose care, and the patches start from a poor place.
Their start plan, called CLoQ, uses a small set of sample data to set the patches so each layer of the slim model hugs the full one. That head start lifts later tuning gains. Tests show slim models tuned this way beat the stock start.
Catalog record
| CREATOR | Yanxia Deng [email protected], Aozhong Zhang [email protected], Selcuk Gurses [email protected], Naigang Wang [email protected], Zi Yang [email protected], Penghang Yin [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18475
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization Yanxia Deng [email protected], Aozhong Zhang [email protected], Selcuk Gurses [email protected], Naigang Wang [email protected], Zi Yang [email protected], Penghang Yin [email protected] Abstract Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ ( C alibrated Lo RA initialization for Q uantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at 2-bit. The code is available at https://github.com/AozhongZhang/CLoQ Reviewed on OpenReview: https://openreview.net/pdf?id=FHnTRAAdAZ 1 Introduction Large language models (LLMs) Achiam et al. (2023) ; Touvron et al. (2023) ; Jiang et al. (2023) ; Guo et al. (2024a) have achieved remarkable success across a wide range of domains and applications. With ongoing advancements, the size and complexity of LLMs have grown exponentially, with some models now exceeding billions or even trillions of parameters. Although this scaling has unlocked unprecedented capabilities, it also introduces significant challenges, particularly in efficiently adapting these models to downstream tasks. Traditionally, full fine-tuning has been the dominant approach for adapting pre-trained models, involving updates to all model parameters. While effective in achieving state-of-the-art results, full fine-tuning is resource-intensive, requiring substantial GPU memory to store both model weights and optimizer states. These memory demands grow with the size of the model, making full fine-tuning increasingly impractical for large-scale models in resource-constrained settings. To address these challenges, parameter-efficient fine-tuning (PEFT) Houlsby et al. (2019) , such as Low-Rank Adaptation (LoRA) Hu et al. (2021) , has emerged as a promising approach. PEFT updates only a small subset of parameters while keeping the majority of the model unchanged, enabling resource-efficient fine-tuning of large-scale models. LoRA, for instance, introduces small, learnable low-rank matrices into the model’s architecture. These matrices are optimized during fine-tuning while the original model weights remain frozen, significantly reducing memory and computational requirements. This design leverages the insight that weight updates often reside in a low dimensional subspace, allowing LoRA to achieve efficient adaptation with minimal overhead. In an orthogonal direction, model compression techniques , notably quantization Rastegari et al. (2016) ; Hubara et al. (2018) ; Choi et al. (2018) ; Wang et al. (2018) ; Yin et al. (2016) ; Yin et al. (2018) ; Yin et al. (2019b) ; Yin et al. (2019a) ; Li et al. (2023b) ; Shao et al. (2023) ; Zhang et al. (2023) ; Frantar et al. (2022a) ; Zhang et al. (2024a) , have been developed to minimize GPU memory usage by converting high-precision weights into low-precision representations. Initially designed for inference in memory-limited environments, quantization methods have since been adapted to support fine-tuning. A notable advancement in this regard is QLoRA Dettmers et al. (2023) , which combines LoRA with quantization techniques to reduce GPU memory requirements for fine-tuning significantly. By leveraging low-rank adaptation and quantized weights, QLoRA enables resource-efficient fine-tuning of LLMs without compromising performance, making it a powerful tool for adapting large-scale models to downstream tasks. However, extending LoRA to quantized LLMs introduces additional challenges, as the reduced representational precision of quantized weights can disrupt standard initialization strategies, impacting task performance. Recent works Li et al. (2023a) ; Yao et al. (2023) ; Liao et al. (2024) ; Guo et al. (2024b) proposed minimizing quantization error through strategic initialization of the low-rank components in LoRA, to align with the original weight states of the model. This strategy has demonstrated success in fine-tuning lower-bit quantized LLMs. Contributions. In this paper, we introduce CLoQ ( C alibrated Lo RA for Q uantized LLMs), an efficient layer-wise, data-driven initialization strategy specifically designed for quantized LLMs by leveraging a small calibration dataset. CLoQ consists of two main steps: a post-training quantization phase to obtain quantized weights and a generalized low-rank approximation phase under a linear transformation to compute the corresponding optimal adapters. We derive a novel closed-form solution to the low-rank approximation problem, which can be efficiently computed using just two singular value decompositions (SVDs). Our CLoQ method requires no back-propagation, making it a highly efficient LoRA initialization scheme for fine-tuning quantized models. We demonstrate the effectiveness of CLoQ through extensive validation on multiple benchmark datasets. Our results show that CLoQ consistently outperforms existing LoRA methods for quantized LLMs, particularly at ultra-low bit-widths. For instance, the fine-tuning accuracy of INT2 CLoQ on the Llama2-13B model Touvron et al. (2023) surpasses that of INT4 QLoRA Dettmers et al. (2023) in the arithmetic reasoning tasks, as shown in Table 3 . Figure 1: Fine-tuning results of Llama2-7B and Llama2-13B across various tasks. Left: the perplexity measured on WikiText-2. Middle: the accuracy achieved on GSM8K. Right: the average accuracy across multiple arithmetic reasoning tasks. Figure 2: The discrepancy \|\boldsymbol{X}(\boldsymbol{Q}+\boldsymbol{A}\boldsymbol{B}^{T}-\boldsymbol{W})\| between the LoRA initialization and the original pre-trained weight matrix, computed using the spectral norm and the Frobenius norm, respectively. The layer shown in the figures above is randomly selected from the Llama2-7B model. The initialization is derived using both CLoQ and LoftQ under INT2 quantization. Notably, CLoQ significantly reduces this discrepancy, demonstrating its effectiveness. Notations. We clarify the mathematical notations that will be used throughout this paper: we denote vectors by bold small letters and matrices by bold capital ones. For any matrix \boldsymbol{X}\in\mathbb{R}^{m\times n} , \boldsymbol{X}^{\top}\in\mathbb{R}^{n\times m} is the transpose of \boldsymbol{X} , and \mathrm{Tr}(\boldsymbol{X}):=\sum_{i=1}^{m}X_{i,i} denotes the trace of \boldsymbol{X} when m=n . For any two matrices \boldsymbol{X},\boldsymbol{Y}\in\mathbb{R}^{m\times n} , \langle\boldsymbol{X},\boldsymbol{Y}\rangle:=\mathrm{Tr}(\boldsymbol{X}^{\top}\boldsymbol{Y})=\sum_{i=1}^{m}\sum_{j=1}^{n}X_{i,j}Y_{i,j} is the inner product. We denote the Frobenius norm of \boldsymbol{X} by \|\boldsymbol{X}\|_{\mathrm{F}}:=\sqrt{\mathrm{Tr}(\boldsymbol{X}^{\top}\boldsymbol{X})}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}X_{i,j}^{2}} . In addition, for any diagonal matrix \boldsymbol{\Sigma}=\mathrm{diag}(\sigma_{1},\dots,\sigma_{n}) , we denote its square root by \boldsymbol{\Sigma}^{\frac{1}{2}}:=\mathrm{diag}(\sigma_{1}^{\frac{1}{2}},\dots,\sigma_{n}^{\frac{1}{2}}) . 2 Background Integer Quantizer. Given a set of m weights \boldsymbol{w}\in\mathbb{R}^{m} , the widely-used b -bit uniform integer (INT) quantizer Choi et al. (2018) determines the float scaling factor \delta=\frac{\max(\boldsymbol{w})-\min(\boldsymbol{w})}{2^{b}-1} and zero-point z=-\left\lfloor\frac{\min(\boldsymbol{w})}{\delta}\right\rceil , where \left\lfloor\cdot\right\rceil is nearest-to-round operation. The quantizer projects \boldsymbol{w} onto the equally spaced grids \mathbb{Q}=\{z\cdot\delta,(z+1)\cdot\delta,\ldots,\left(z+(2^{b}-1)\right)\cdot\delta\}^{m} to obtain the quantized weights \boldsymbol{q}=\delta\cdot\left(\text{clip}\left(\left\lfloor\frac{{\boldsymbol{w}}}{\delta}\right\rceil+z,0,2^{b}-1\right)-z\right). For channel-wise (or group-wise) quantization, the scaling factor \delta is shared by the quantized weights within the same channel (or group, respecitively). Post-Training Quantization. Post-training quantization (PTQ) has been a cornerstone technique for compressing LLMs. PTQ methods directly identify low-precision representations of a model without requiring retraining, making them particularly well-suited for extremely large-scale AI models including LLMs. The simplest approach in this category is data-free quantization, commonly referred to as RTN, which involves rounding the pre-trained model weights to their nearest quantized states. More advanced PTQ algorithms, such as OPTQ Frantar et al. (2022b) ,