Taming Sensitive Weights: Noise Perturbation Fine-tuning for Robust LLM Quantization

Dongwei Wang, Huanrui Yang · 2024

Read Taming Sensitive Weights: Noise Perturbation Fine-tuning for Robust LLM Quantization by Dongwei Wang, Huanrui Yang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06858.txt.

Introduction

Dongwei Wang and Huanrui Yang present Noise Perturbation Fine-tuning, a 2024 fix for fragile weights in model compression. Quantization shrinks language models for cheap serving. A few outlier weights react badly to that squeeze. Keeping them at full precision breaks hardware efficiency.

Their method finds the outliers and shakes them with random noise during light fine-tuning. The shaking flattens the loss surface around those weights. Tamed outliers then survive quantization with less damage. No special mixed-precision handling is needed afterward on the serving hardware. The trick aims at smooth deployment on plain hardware.

Catalog record

CREATORDongwei Wang, Huanrui Yang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Taming Sensitive Weights: Noise Perturbation Fine-tuning for Robust LLM Quantization Dongwei Wang, Huanrui Yang Abstract Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights’ impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose N oise P erturbation F ine- t uning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark. 1 Introduction Large Language Models (LLMs), usually with billions of parameters, have demonstrated impressive problem-solving abilities across diverse tasks [ 3 , 10 , 21 , 24 ] . The enhanced performance, largely driven by the scaling of both training data and model parameters [ 14 ] , has made it challenging to deploy LLMs on edge computing devices. For example, a model like GPT-3 [ 11 ] , with 175 billion parameters, requires 350 GB storage space in FP16 and powerful GPUs such as A100 for quick inference, which makes deployment on devices like laptops or mobile phones infeasible without significant model compression. As a promising approach, low-bit weight quantization can help address this issue by enabling efficient inference and reducing storage requirements. Pioneering works such as GPTQ [ 12 ] and Squeeze LLM [ 15 ] can compress LLaMA [ 22 ] model weights to 3-4 bits with a nearly lossless performance, achieving 2-3 \times speed up on GPUs. The most straightforward way to perform weight quantization is via linear uniform quantization, which quantizes the entire model to the same bit-width using simple quantizers like Round-to-Nearest (RTN). However, it is widely acknowledged that weights are not equally important in a neural network . There is a small fraction of weights that are very sensitive to quantization and lead to significant performance degradation if quantized. This is because these sensitive weights (also referred to as outliers ) have a larger impact on the outputs of their respective layers, which in turn affects the final loss of the model. As shown in Fig. 1 , the performance of two OPT [ 26 ] models dropped significantly after applying RTN quantization. However, preserving 0.5% of the outliers as FP16 can greatly recover the performance of the quantized model. A 4-bit quantized model with 0.5% full-precision outliers can achieve performance comparable to that of a full-precision model. In light of this, existing work typically focuses on reducing the quantization error of outliers by preserving them. For example, SqueezeLLM [ 15 ] proposed to store outliers as a sparse FP16 matrix and design a non-uniform sensitivity-based quantizer for other weights. In SpQR [ 6 ] , outliers are extracted in the form of channel groups and quantized to a higher bit-width. AWQ [ 16 ] proposed to select outliers based on activation distribution and reduce their quantization error by per-channel scaling. In general, existing works tend to preserve outliers at a higher precision than normal weights. However, even though this may not significantly impact the overall size of the quantized model, it results in the model being in a mixed-precision state. According to [ 19 ] , most GPUs prefer operations with a uniform data type, as mixed precision requires different storage methods in memory, hindering the full utilization of GPU memory bandwidth. Additionally, mixed precision disrupts the SIMD processing model, leading to inefficient use of GPU cores. In this paper, we aim to mitigate the special treatment on outlier weights by enabling the model to maintain good performance even after quantizing these outliers to the same bit-width. Drawing inspiration from previous work on analyzing quantization error impacts, like HAWQ [ 8 , 7 ] and HERO [ 25 ] , we pinpoint the quantization error of the outliers is contributed by both their quantization errors and the loss function’s Hessian trace with respect to them. This observation leads to the proposal of an efficient fine-tuning method named N oise P erturbation F ine- t uning (NPFT), which helps reduce the Hessian trace of outlier weights with a short parameter-efficient fine-tuning (PEFT) process. Specifically, we estimate the Hessian trace with the expected model loss under random weight perturbations. We further fine-tune the model to reduce the outlier Hessian trace by performing PEFT optimization with random weight perturbation being added to the outlier weight locations in the base model. The NPFT process avoids the costly higher-order gradient computation needed to optimize Hessian or Fisher matrix directly, while also enjoys a smoother convergence compared to full QAT as all weights are still kept as floating-points in the training process. As shown in Fig. 1 , NPFT fine-tuned model shows better PPL than quantized baseline even with the simple RTN quantizer. Figure 1: (Left) RTN suffers from the degradation caused by quantizing outliers. Preserving 0.5% of the outliers in FP16 can greatly recover the performance of the quantized model. (Right) NPFT brings significant improvement to the performance of single bit-width models without preserving any outliers in FP16 . When applied to OPT 1.3B/2.7B models, our method outperforms RTN baseline by a large PPL margin of over 2.9 / 0.9 on the C4 benchmark. In summary, we make the following contribution in this paper: • We propose NPFT, a PEFT method that efficiently reduces the Hessian trace of outlier weights in LLMs; • The reduced Hessian trace enables outlier weights to be quantized to the same precision as other weights, without special treatment, while still preserving good quantized model performance; • NPFT works with different post-training quantizers, enhancing the performance of quantized models across various bit-width settings. NPFT enables stable perplexity improvements on various uniform and non-uniform quantizers. For example, NPFT helps RTN achieve a 3.69 perplexity improvement on OPT-1.3B-4bits and achieve performance comparable to GPTQ on LLaMA2-7B-4bits. Additionally, by eliminating the need of preserving any outlier weights in FP16, NPFT achieves a 10% reduction in inference latency on the 4090 GPU. 2 Related Work LLM Quantization. LLM quantization can be categorized into two branches: quantization-aware training (QAT) methods [ 2 , 17 , 9 , 5 ] and post-training quantization (PTQ) methods [ 16 , 15 , 6 , 12 , 23 ] . QAT typically requires extensive retraining to recover accuracy after quantization, whereas PTQ does not involve retraining. Although QAT can enhance the performance of quantized models, it is not easily scalable to LLMs due to the significant computational resources required for retraining and the difficulties in convergence. Therefore, most works on LLM quantization

Read the complete text