QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models
Hyesung Jeon, Seojune Lee, Beomseok Kang, Yulhwa Kim, Jae-Joon Kim · 2025
Read QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models by Hyesung Jeon, Seojune Lee, Beomseok Kang, Yulhwa Kim, Jae-Joon Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17428.txt.
Introduction
Hyesung Jeon, Seojune Lee, Beomseok Kang, Yulhwa Kim, and Jae-Joon Kim shrink language models for cheap fine-tuning. Their 2025 arXiv paper joins two trends: quantization, which cuts inference cost by storing weights in fewer bits, and parameter-efficient fine-tuning, which cuts training cost. In a quantized model, trimming the quantization error before fine-tuning is the key to accuracy.
Plain low-rank adapters lack the expressive range for that job. Fourier-style adapters are stronger, but bolting them onto quantized models can waste compute and miss the error. Their method, QWHA, uses the Walsh-Hadamard transform as its kernel and adds a careful adapter start, with adaptive parameter choice and value refinement. Tests show the method cuts quantization error while letting fine-tuning proceed. The result is a route to accurate quantized models that stay cheap to adapt.
Catalog record
| CREATOR | Hyesung Jeon, Seojune Lee, Beomseok Kang, Yulhwa Kim, Jae-Joon Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17428
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models Hyesung Jeon, Seojune Lee, Beomseok Kang, Yulhwa Kim, Jae-Joon Kim Abstract The demand for efficient deployment of large language models (LLMs) has driven interest in quantization, which reduces inference cost, and parameter-efficient fine-tuning (PEFT), which lowers training overhead. This motivated the development of quantization-aware PEFT to produce accurate yet efficient quantized models. In this setting, reducing quantization error prior to fine-tuning is crucial for achieving high model accuracy. However, existing methods that rely on low-rank adaptation suffer from limited representational capacity. Recent Fourier-related transform (FT)-based adapters offer greater representational power than low-rank adapters, but their direct integration into quantized models often results in ineffective error reduction and increased computational overhead. To overcome these limitations, we propose QWHA, a method that integrates FT-based adapters into quantized models by employing the Walsh-Hadamard Transform (WHT) as the transform kernel, together with a novel adapter initialization scheme incorporating adaptive parameter selection and value refinement. We demonstrate that QWHA effectively mitigates quantization errors while facilitating fine-tuning, and that its design substantially reduces computational cost. Experimental results show that QWHA consistently outperforms baselines in low-bit quantization accuracy and achieves significant training speedups over existing FT-based adapters. The code is available at https://github.com/vantaa89/qwha . 1 Introduction Fine-tuning enables large language models (LLMs) to generalize beyond their pre-training, allowing adaptation to various domains ( Wei et al. 2022 ; Liu et al. 2023 ; Qin et al. 2024 ; DeepSeek-AI et al. 2025 ) . While full fine-tuning yields superior accuracy, it often incurs significant overhead due to the extensive computations required to update all the trainable model parameters ( Loshchilov & Hutter 2017 ; Zhu et al. 2025 ) . Parameter-efficient fine-tuning (PEFT) addresses this issue by optimizing only a small subset of the parameters while leaving most of them frozen ( Li & Liang 2021 ; Liu et al. 2022 ; Hu et al. 2022 ; Liu et al. 2024 ; Kopiczko et al. 2024 ) . Beyond reducing training overhead, recent studies have shown that combining PEFT with model compression techniques can enhance inference efficiency at the same time ( Dettmers et al. 2023 ) . Among these techniques, quantization, which lowers the bit precision of model parameters, has gained particular attention due to its robustness against accuracy degradation under high compression ratios ( Frantar et al. 2023 ; Lin et al. 2024 ; Dettmers et al. 2024 ; Kim et al. 2024b ; Shao et al. 2024 ; Ashkboos et al. 2024 ; Zhang et al. 2024 ; Liu et al. 2025 ) . Consequently, quantization-aware PEFT (QA-PEFT) has been widely explored as a promising approach for efficient adaptation and inference in LLMs. Prior works on QA-PEFT typically relied on low-rank adaptation (LoRA) ( Li et al. 2024 ; Guo et al. 2024 ; Kim et al. 2024a ; Liao et al. 2024 ; Deng et al. 2025 ) . In contrast, for standard PEFT, several alternatives to LoRA have recently been proposed to address the representational limitations of low-rank structures. In particular, Fourier-related transform (FT)-based adapters have emerged as strong alternatives. They train a sparse set of coefficients to represent weight updates in the transform domain, offering superior representational capacity ( Gao et al. 2024b ; Du et al. 2025 ; Shen et al. 2025 ) . However, our observations show that directly applying FT-based adapters to quantized models often yields worse performance than LoRA-based methods specifically designed for QA-PEFT. This highlights the importance of explicit consideration for quantization effects when fine-tuning quantized models. LoRA-based methods adopt quantization-aware initialization strategies that compensate for the errors between full- and low-precision weights using low-rank approximation with the adapters prior to fine-tuning. However, applying such initialization in FT-based adapters is non-trivial, as identifying the optimal sparse set of parameters and their values to approximate a given matrix is an NP-hard problem ( Natarajan 1995 ) . Moreover, the choice of transform type becomes an additional design consideration. This raises a research question: how to effectively exploit FT-based adapters in QA-PEFT. To the best of our knowledge, neither FT-based adapters nor their initialization techniques have been explored in the context of QA-PEFT. Figure 1: Overview of Quantization-aware Walsh-Hadamard Adaptation ( QWHA ). The weight update from QWHA is formulated as \Delta{\bm{W}}={\bm{F}}{\bm{H}}^{-1} , where {\bm{H}} is a predefined Walsh-Hadamard transform (WHT) matrix and {\bm{F}} is a trainable sparse coefficient matrix consisting of values {\bm{c}} and their indices {\bm{E}} . The multiplication {\bm{F}}{\bm{H}}^{-1} indicates the expansion of learned coefficients ( i.e. , {\bm{c}} ), over the transform basis ( i.e. , columns of {\bm{H}}^{-1} ). Note that, the coefficients {\bm{c}} are the only trainable parameters, and {\bm{H}} remains constant. Our key contributions are in the adoption of WHT into the adapter ( WHA ) and their initialization, particularly {\bm{E}} ( AdaAlloc ) and {\bm{c}} ( Refinement ). Table 1: Comparison of adapter types and parameter selection strategies. Adapter types include low-rank adapters (LoRA), recent FT-based adapters (DCA and DHA), and our proposed adapter ( WHA ). Strategies to determine parameter location {\bm{E}} in {\bm{F}} include magnitude-based selection, random uniform selection, training via reparameterization, and our proposed method ( AdaAlloc ). Adapter Type Parameter Selection Strategy Ability Factors LoRA DCA DHA WHA Magnitude Random Trainable AdaAlloc Fine‐tuning Quantization Error Reduction In this paper, we present QWHA, a novel QA-PEFT method that introduces a FT-based adapter together with a quantization-aware initialization scheme, as illustrated in Figure 1 and Table 1 . We adopt WHT in our adapter design ( WHA ), inspired by its high-fidelity reconstruction ability in the spectral domain, to effectively compensate for quantization errors ( Hedayat 1978 ) . In addition, the WHT kernel consists solely of \pm 1 elements, enabling efficient computations using only additions and subtractions, thereby eliminating matrix multiplications ( Dao-AILab 2024 ) . We further reduce computation by applying a single transform in the adapter, unlike conventional FT-based adapters that apply two transforms. For quantization-aware adapter initialization, we develop a tractable solution that first selects parameter locations {\bm{E}} and then assigns their values {\bm{c}} . We introduce a channel-wise parameter allocation scheme that guarantees a lower bound on the number of parameters per channel to facilitate fine-tuning while allocating more parameters to channels with larger quantization errors, and then select the highest-magnitude coefficients within each channel to effectively reduce quantization error ( AdaAlloc ). Finally, we refine the selected parameter values, thereby enabling substantial reduction of quantization error ( Refinement ). We theoretically analyze the superior representation capacity of our proposed adapter and empirically validate the benefits of our adapter design and initialization method across diverse datasets and models. 2 Background 2.1 LLM Quantization LLM quantization is a key technique for improving inference efficiency by reducing the memory bottleneck caused by model weights through lowering their bit precision ( Frantar et al. 2023 ) , typically expressed by the following equation: \tilde{{\bm{W}}_{Q}}=\text{clamp}\left(\text{round}\left(\frac{{\bm{W}}_{0}}{s}\right)-z,0,2^{n}-1\right)\quad{\bm{W}}_{Q}=(\tilde{{\bm{W}}_{Q}}+z)\times s (1) Here, {\bm{W}}_{0} denotes the pre-trained weight matrix, while \tilde{{\bm{W}}}_{Q} and {\bm{W}}_{Q} represent the quantized integer weights and the corresponding dequantized weights, respectively. s and z are quantization scales and integer zero-points. Clamping is applied to the rounded and shifted value within the range 0 to 2^{n}-1 , where n