Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis

Hong Huang, Dapeng Wu · 2025

Read Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis by Hong Huang, Dapeng Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14742.txt.

Introduction

Huang and Wu address a hard trade-off in their 2025 paper: shrinking large language models for home devices without losing skill. Cutting number precision saves memory, but stray large values, called outliers, spoil the results. The pair note a pattern they name the outlier spatial stability hypothesis. Outlier channels tend to stay in the same spots during training.

Their framework, Quaff, builds on that pattern. It calms outliers only in the steady channels, using light steps and no full-precision weight store. Across ten test sets the method held up well. On one reasoning test it cut wait time sharply and saved about a third of memory. The authors say such gains let modest home graphics cards fine-tune big models.

Catalog record

CREATORHong Huang, Dapeng Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis Hong Huang, Dapeng Wu Abstract Large language models (LLMs) have made exciting achievements across various domains, yet their deployment on resource-constrained personal devices remains hindered by the prohibitive computational and memory demands of task-specific fine-tuning. While quantization offers a pathway to efficiency, existing methods struggle to balance performance and overhead, either incurring high computational/memory costs or failing to address activation outliers, a critical bottleneck in quantized fine-tuning. To address these challenges, we propose the Outlier Spatial Stability Hypothesis ( OSSH ): During fine-tuning, certain activation outlier channels retain stable spatial positions across training iterations. Building on OSSH, we propose Quaff , a Quantized parameter-efficient fine-tuning framework for LLMs, optimizing low-precision activation representations through targeted momentum scaling. Quaff dynamically suppresses outliers exclusively in invariant channels using lightweight operations, eliminating full-precision weight storage and global rescaling while reducing quantization errors. Extensive experiments across ten benchmarks validate OSSH and demonstrate Quaff’s efficacy. Specifically, on the GPQA reasoning benchmark, Quaff achieves a 1.73 × 1.73\times latency reduction and 30 % 30\% memory savings over full-precision fine-tuning while improving accuracy by 0.6 % 0.6\% on the Phi-3 model, reconciling the triple trade-off between efficiency, performance, and deployability. By enabling consumer-grade GPU fine-tuning ( e.g., RTX 2080 Super) without sacrificing model utility, Quaff democratizes personalized LLM deployment. The code is available at https://github.com/Little0o0/Quaff.git . 1 Introduction Large language models (LLMs) ( Wu et al., 2023a ; Floridi and Chiriatti, 2020 ; Zhang et al., 2022 ) exhibit remarkable achievement in various domains, from creative writing to conversational chatbots. Growing demands for privacy-preserving, personalized LLMs ( e.g., a virtual companion chatbot) deployed on local devices clash with the prohibitive computational and memory costs of fine-tuning, building a critical barrier for individuals and small enterprises. While parameter-efficient fine-tuning (PEFT) reduces trainable parameters, it still imposes unsustainable overhead when scaling to billion-parameter models. Figure 1: Comparison of accuracy, average latency per step of WAQ baselines with Phi-3 on the GPQA benchmark using LoRA fine-tuning. The size of the marker represents the GPU memory footprints. Figure 2: (a) Spatial stability of outlier channels: activation magnitude distribution during fine-tuning, demonstrating stable spatial indices for outlier channels across iterations. (b) Activation distribution shift: the magnitudes of outliers change significantly during fine-tuning. (c) Quaff Efficacy: Failure of static scaling due to outlier fluctuations, contrasted with Quaff’s targeted momentum scaling on stable outlier channels, successfully suppressing outliers. Quantization ( Dettmers et al., 2021 ; Dettmers et al., 2022 ; Lin et al., 2023 ; Frantar et al., 2022 ) offers a pathway to efficiency, but most of existing work focuses narrowly on weight-only quantization (WOQ) in fine-tuning Kwon et al. (2022) ; Dettmers et al. (2024) ; Xu et al. (2023) ; Li et al. (2023) ; Liu et al. (2023) ; Guo et al. (2023) ; Kim et al. (2024) ; He et al. (2023) ; Lee et al. (2024a) ; Frantar et al. (2022) ; Lin et al. (2023) , which compresses frozen weights to low precision ( e.g., 4-bit). However, WOQ introduces computational bottlenecks via hardware-unfriendly mixed-precision operations between quantized weights and full-precision activations. Weight-Activation Quantization (WAQ) Zhou et al. (2016) ; Dettmers et al. (2021) ; Hubara et al. (2018) addresses this by quantizing both weights and activations into hardware-friendly formats ( e.g., INT8), but faces a fundamental challenge in LLMs, emergent channel-wise outliers Wei et al. (2022) ; Wu et al. (2023b) ; Yao et al. (2022) ; Dettmers et al. (2022) ; Xiao et al. (2023) ; Huang et al. (2024b) , where large activations amplify quantization errors and degrade model performance. Prior methods suppress outliers in inference via channel-wise scaling between activations and weights Dettmers et al. (2022) ; Wei et al. (2022) ; Xiao et al. (2023) ; Wei et al. (2023) But these methods fail to adapt to fine-tuning due to activation distribution shifts (Fig. 2 (b)): 1. Static scaling Xiao et al. (2023) predefines factors on calibration data, but mismatched scaling amplifies quantization errors as distributions evolve. While some methods Lin et al. (2025) ; Huang et al. (2024b) ; Ashkboos et al. (2024) attempt to mitigate this by replacing scaling with rotation, they introduce architectural rigidity and computational Inefficiency. 2. Dynamic scaling Dettmers et al. (2022) ; Xiao et al. (2023) adapts factors in real time but requires storing and rescaling of full-precision weights, incurring prohibitive memory/compute costs. We find the key problem in previous approaches is coupling bottleneck : scaled weight quantization depends on real-time activation statistics, preventing hardware-friendly deployment. To address this, we first introduce the Outlier Spatial Stability Hypothesis ( OSSH ): During fine-tuning, certain activation outlier channels retain stable spatial position across training iterations. Building on OSSH, we propose Quaff , a Qua ntized parameter-e f ficient f ine-tuning framework for LLMs that decouples weight and activation quantization via targeted momentum scaling. Quaff dynamically computing scaling factors exclusively for invariant outlier channels, eliminating full-precision weight storage and global rescaling, enabling low quantization error and high efficiency. We evaluate Quaff across ten benchmarks, including reasoning (MMLU-Pro Wang et al. (2024b) , GPQA Rein et al. (2023) , instruction-tuning(Alpaca-Finance Bharti (2023) , Self-instruct Wang et al. (2022) ) and long text task (LongForm Köksal et al. (2023) , LAMBADA Paperno et al. (2016) ), using LLaMA-2 Touvron et al. (2023) , Phi-3 Abdin et al. (2024) , and OPT Zhang et al. (2022) models with four general PEFT methods (LoRA Hu et al. (2021) , Prompt tuning Lester et al. (2021) , P-tuning Liu et al. (2021) and IA3 Liu et al. (2022) ). Our experiment also evaluates the deployability of Quaff on a consumer-grade laptop with RTX 2080 super GPU (8GB). Extensive experimental results suggest that Quaff achieves the best trade-offs: it achieves 1.73\times faster fine-tuning and 30\% memory reduction versus full-precision baselines while improving accuracy by 0.6\% on GPQA. Against state-of-the-art (SOTA) WAQ methods, Quaff delivers a 2.1\% accuracy gain under identical constraints (Fig. 1 ), validating its ability to harmonize efficiency (latency/memory), accuracy, and deployability. To our knowledge, Quaff is the first work to systematically resolve the triple trade-off in WAQ LLM fine-tuning, enabling local-device fine-tuning as effortlessly as a toast. 2 Background and Challenge This section introduces foundational concepts of neural network quantization, analyzes the unique challenges of weight-activation quantization (WAQ) in LLMs, and identifies fundamental limitations in existing approaches for fine-tuning. 2.1 Quantization Fundamentals Quantization reduces numerical precision in neural networks by converting weights and activations from high-bit formats ( e.g., FP32) to low-bit representations ( e.g., INT8), optimizing memory usage and computational efficiency. The standard symmetric round-to-nearest quantization Jacob et al. (2018) maps a floating-point matrix \mathbf{X} into an N -bit integer matrix \mathbf{X}_{int} : \mathbf{X}_{int}=Q(\mathbf{X})=\left[\frac{\mathbf{X}}{\Delta_{\mathbf{X}}}\right],\quad\Delta_{\mathbf{X}}=\frac{\mathrm{max}(|\mathbf{X}|)}{2^{N-1}-1}, (1) where [\cdot] denotes rounding function, and \Delta_{\mathbf{X}} is quantization step size. The granularity of \Delta_{\mathbf{X}} (scalar, vector, or matrix) determines how finely quantization is applied (see Appendix F ). 2.2 Weight-Activation Quantization in LLMs Weight-activation quantization (WAQ) compresses both weights \mathbf{W}\in\mathbb{R}^{c_{in}\times c_{out}} and activations \mathbf{X}\in\mathbb{R}^{t\times c_{in}} , where t is the number of tokens, c_{in} is the number of input channels, and c_{out} is the number of output channels, to accelerate matrix multiplication \mathbf{Y}=\mathbf{X}\mathbf{W} : \mathbf{Y}\approx\Delta_{\mathbf{X}}\cdot(\mathbf{X}_{int}\mathbf{W}_{int})\cdot\Delta_{\mathbf{W}}, (2) where INT8

Read the complete text