QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition

Yuxuan Hu, Xiaodong Chen, Cuiping Li, Hong Chen, Jing Zhang · 2025

Read QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition by Yuxuan Hu, Xiaodong Chen, Cuiping Li, Hong Chen, Jing Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19353.txt.

Introduction

Yuxuan Hu and four co-authors present QUAD, a 2025 method for shrinking large language models to 4-bit precision without the usual accuracy loss. Quantization cuts compute cost, but activation outliers wreck mid-sized models like Llama-3-8B. QUAD decomposes activations with singular value decomposition, building a fixed transform that shifts the outlier parts into extra dimensions kept at full precision. The rest quantizes to 4 bits.

The spared outlier weights double as a fine-tuning surface, letting cheap adaptation close much of the remaining gap. On Llama-3 and Qwen-2.5 models, QUAD kept 94 to 96 percent of full accuracy at the tightest setting, and about 98 percent with tuning. Code is public. The authors pitch the method for running strong models on lean hardware.

Catalog record

CREATORYuxuan Hu, Xiaodong Chen, Cuiping Li, Hong Chen, Jing Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition Yuxuan Hu, Xiaodong Chen, Cuiping Li, Hong Chen, Jing Zhang Abstract Large Language Models (LLMs) excel in diverse applications but suffer inefficiency due to massive scale. While quantization reduces computational costs, existing methods degrade accuracy in medium-sized LLMs (e.g., Llama-3-8B) due to activation outliers. To address this, we propose QUAD (Quantization with Activation Decomposition), a framework leveraging Singular Value Decomposition (SVD) to suppress activation outliers for effective 4-bit quantization. QUAD estimates activation singular vectors offline using calibration data to construct an orthogonal transformation matrix P, shifting outliers to additional dimensions in full precision while quantizing rest components to 4-bit. Additionally, QUAD enables parameter-efficient fine-tuning via adaptable full-precision outlier weights, narrowing the accuracy gap between quantized and full-precision models. Experiments demonstrate that QUAD achieves 94–96% accuracy under W4A4 quantization and 98% accuracy with W4A4/A8 and parameter-efficient fine-tuning for Llama-3 and Qwen-2.5 models. Our code is available at repository . 1 Introduction Large Language Models (LLMs) [ 1 ; 2 ; 3 ] have demonstrated remarkable performance across numerous fields and have been widely adopted in various applications, such as chat assistants, coding copilots [ 4 ; 5 ; 6 ] , and beyond. However, the scaling law [ 7 ] has led to increasingly deeper LLMs with hundreds of billions of parameters, rendering them inefficient for text-processing tasks. Concurrently, existing work [ 8 ] highlights that for throughput-oriented serving systems, many workloads are predominantly compute-bound. Consequently, there is a critical need for effective methods to compress LLMs and enhance the efficiency of General Matrix Multiplication (GEMM) operations, given that GEMM dominates computational tasks within LLMs. Weight and activation quantization aims to address this issue by representing both weights and activations in lower precision, thereby reducing storage overhead and leveraging more efficient hardware, such as INT4 tensor cores, to accelerate GEMM computations. While existing approaches [ 9 ; 10 ; 11 ] have successfully quantized LLM weights to 4 bits or even lower with nearly no loss in accuracy, quantizing both weights and activations remains challenging. This difficulty arises due to the higher prevalence of outliers in activations, making them harder to quantify than weights. Existing methods like QuaRot [ 12 ] , SpinQuant [ 13 ] , and DuQuant [ 14 ] attempt to suppress outliers in both weights and activations by rotating the matrix and quantizing both to 4 bits. These techniques perform well for large-scale LLMs, such as Llama-2-70B, yet still result in significant performance degradation for medium-sized LLMs, such as Llama-3-8B. We attribute this to the fact that for medium-sized LLMs, the rotation matrix is insufficient in suppressing outliers, making it difficult to quantize activations to 4 bits. Thus, more effective methods are required to eliminate outliers from activations. Singular Value Decomposition (SVD) is an effective tool used to decompose a matrix’s high and low-frequency components, thus enabling the removal of outliers from the matrix [ 15 ] . Leveraging the impressive performance of SVD in eliminating outliers, we apply it to activations and propose our method, named QUAD ( Qu antization with A ctivation D ecomposition). However, migrating SVD to activations presents several challenges. Firstly , weights remain static, whereas activations dynamically change with varying inputs during serving. Secondly , while the SVD of the weight matrix can be computed offline, the SVD of activations cannot be performed online. Thirdly , the introduced SVD must be compatible with existing rotation methods. To address these challenges, QUAD estimates the singular vectors of activations [ 16 ; 17 ] offline using a small amount of calibration data and constructs the transformation matrix {P}\in\mathbb{R}^{(C+r)\times C} based on these singular vectors. The matrix P has two key properties: (1) it shifts outliers in activations to an additional r dimension, thereby eliminating outliers in the original activations; (2) it satisfies P^{\top}P=I . Consequently, matrix P enables equivalent transformations of the LLM and is compatible with existing methods. After transformation, for GEMM computations, most weights and activations are quantized to 4 bits, while the r outlier dimensions and their corresponding weights are retained in full precision. Beyond removing activation outliers, QUAD can also be applied to parameter-efficient fine-tuning of quantized models [ 18 ; 19 ; 20 ] . Specifically, we retain the weights corresponding to the outlier dimensions in full precision, meaning these portions of weights can serve as parameter-efficient adapters to fine-tune the model. Additionally, we demonstrate that the adapter initialized by QUAD provides a sub-optimal solution for approximating full fine-tuning with unchanged input distributions. Based on the full-precision portion of the model, we further reduce the gap between the quantized and full-precision models through parameter-efficient fine-tuning. To evaluate the effectiveness of QUAD, we conducted extensive experiments on diverse LLMs and datasets. The contributions of this work are summarized as follows: • We introduce QUAD, which suppresses outliers in activations via SVD decomposition and integrates with existing rotation methods to enhance the performance of quantized models. • Beyond quantization, QUAD can also be used for parameter-efficient fine-tuning, and we propose fine-tuning the full-precision portion of the quantized model to further bridge the gap between the quantized and full-precision models. • Based on these improvements, QUAD maintains 94% to 96% of the full-accuracy model’s performance under W4A4 quantization and achieves 98% of the full-accuracy model’s performance when combined with W4A4/A8 quantization and parameter-efficient fine-tuning. 2 Related Work Quantization is a commonly used approach for LLM deployment to compress storage space and improve inference speed by representing weights and activations in lower bits. Existing quantization methods can be broadly categorized into two categories: Quantization-aware training (QAT) and Post-training quantization (PTQ). QAT combines quantization with training and fine-tuning to represent the model with lower precision while maintaining model performance. Representative QAT methods include LLM-QAT [ 21 ] , BitDistiller [ 11 ] , EfficientQAT [ 22 ] , and the BitNet series [ 23 ; 24 ] . Since the QAT method has large resources and time overhead, more work focuses on PTQ, which can achieve quantization with only a small amount of calibration data. The main challenge of PTQ methods comes from the outliers in the parameters and activations of LLM, which bring large quantization errors. Therefore, existing methods have proposed the following methods to overcome the impact of outliers, including model equivalent transformation and weight compensation. Model equivalent transformation, including shifting, scaling, and rotation. SmoothQuant [ 25 ] and AWQ [ 10 ] propose employing scaling operations, while OS+ [ 26 ; 27 ] proposes shifting operations to suppress outliers. In addition to scaling and shifting, QUIP [ 28 ] , QuaRot [ 12 ] , DuQuant [ 14 ] , and SpinQuant [ 13 ] further utilize rotation operations to suppress outliers. These transformations are also used in subsequent work, such as QUIK [ 29 ] , QmniQuant [ 30 ] , AffineQuant [ 31 ] , QServe [ 32 ] , and OSTQuant [ 33 ] . The weight compensation technique, which improves quantization by adjusting the weights during the quantization process, was first proposed by OBS [ 34 ] and subsequently widely applied to LLM by GPTQ [ 9 ] . In addition to transformations and compensation, existing work

Read the complete text