FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference

Coleman Hooper 1,2 , Charbel Sakr 1 , Ben Keller 1 , Rangharajan Venkatesan 1 , Kurt Keutzer 2 , Sophia Shao 2 , Brucek Khailany 1 · 2025

Read FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference by Coleman Hooper 1,2 , Charbel Sakr 1 , Ben Keller 1 , Rangharajan Venkatesan 1 , Kurt Keutzer 2 , Sophia Shao 2 , Brucek Khailany 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14152.txt.

Introduction

Hooper and co-authors propose in 2025 a fine grained mixed precision method to shrink large language models for fast hardware inference. Cutting numeric precision saves energy and memory, but crude cuts harm accuracy. Their method keeps most values at low precision and spares the blocks that matter most.

A policy weighs each block's disturbance by its effect on model loss to pick which blocks stay at high precision. A weighted clipping step helps blocks that do drop in precision keep accuracy. Hardware changes support the mixed scheme at block level. The co design aims to give hardware level speed gains without the usual accuracy loss.

Catalog record

CREATORColeman Hooper 1,2 , Charbel Sakr 1 , Ben Keller 1 , Rangharajan Venkatesan 1 , Kurt Keutzer 2 , Sophia Shao 2 , Brucek Khailany 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14152

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference Coleman Hooper 1,2 , Charbel Sakr 1 , Ben Keller 1 , Rangharajan Venkatesan 1 , Kurt Keutzer 2 , Sophia Shao 2 , Brucek Khailany 1 Abstract . Quantization is a powerful tool to improve large language model (LLM) inference efficiency by utilizing more energy-efficient low-precision datapaths and reducing memory footprint. However, accurately quantizing LLM weights and activations to low precision is challenging without degrading model accuracy. We propose fine-grained mixed precision (FGMP) quantization, a post-training mixed-precision quantization methodology that maintains accuracy while quantizing the majority of weights and activations to reduced precision. We co-design hardware which exploits the mixed-precision data representation to achieve greater energy efficiency and which quantizes activations to mixed precision on the fly. Our work makes the following contributions: 1) We develop a policy that uses the perturbation in each value, weighted by the Fisher information, to select which weight and activation blocks to keep in higher precision. This approach preserves accuracy by identifying which weight and activation blocks need to be retained in higher precision to minimize the perturbation in the model loss. 2) We also propose a sensitivity-weighted clipping approach for fine-grained quantization which helps retain accuracy for blocks that are quantized to low precision. 3) We then propose hardware augmentations to leverage the efficiency benefits of FGMP quantization. Our hardware implementation encompasses i) datapath support for FGMP at block granularity, and ii) a mixed-precision activation quantization unit to assign activation blocks to high or low precision on the fly with minimal runtime and energy overhead. Our design, prototyped using NVFP4 (an FP4 format with microscaling) as the low-precision datatype and FP8 as the high-precision datatype, facilitates efficient FGMP quantization, attaining < < 1% perplexity degradation on Wikitext-103 for the Llama-2-7B model relative to an all-FP8 baseline design while consuming 14% less energy during inference and requiring 30% less weight memory. 1. Introduction Figure 1. Perplexity degradation (less is better) versus compression rate (higher is better), evaluated using Llama-2-7B on Wikitext-103 with 4-bit weight-activation quantization. Compression rate is computed as 16 divided by the average bit width for weights and activations (assuming 4K context length). We group prior work into algorithm-only approaches that use integer quantization (“Algo.”), prior work on microscaling quantization (“ \mu scale’’), and prior methods for mixed-precision quantization (‘‘MP’’). Our method, FGMP, (shown here with 70%, 80%, and 90% of blocks in FP4) attains reduced perplexity degradation relative to existing post-training quantization methods. 2 2 2 Note that 6-bit configurations are reported for ( Shao et al., 2023 ; Xiao et al., 2023 ) since the corresponding W4A4 configurations have greater than 1 PPL loss, and ( Shao et al., 2023 ; Zhao et al., 2024 ) quantize the KV cache activations in addition to the linear layers. Figure 2. a) Diagram outlining our approach for FGMP quantization at block granularity. We quantize both weights and activations to mixed precision, preserving sensitive blocks in higher precision (FP8) and quantizing the rest to low precision (NVFP4). b) Visualization of a slice of the activation and weight precision assignments for Layer 7 Fully Connected 1 (“FC1”) in the Llama-2-7B model using our policy with 10% of blocks in FP8 and with our sensitivity-weighted clipping method applied. The NVFP4 and FP8 blocks are interleaved in an unstructured manner. Large language models (LLMs) are scaling rapidly in both model size and computational requirements, which makes it challenging to efficiently perform inference using these networks. One approach for reducing both computational and memory requirements for LLM inference is quantization of model weights and activations to low-precision number formats. However, aggressive quantization of all weights and activations is challenging due to the presence of data particularly sensitive to perturbations, such as large-magnitude outliers in weights and activations. Quantization-aware training (QAT) may be employed to mitigate these perturbations ( Jacob et al., 2018 ; Nagel et al., 2021 ) ; however, the costs of re-training and unavailability of training data with recent LLMs can make QAT difficult and motivates post-training quantization (PTQ) approaches ( Frantar et al., 2022 ; Lin et al., 2024 ) . In this paper, we explore techniques to improve PTQ inference accuracy. Using low precision (e.g. 4-bit) quantization for weights and activations for LLMs is desirable since it allows the use of energy-efficient low-precision compute datapaths (in addition to memory savings). However, accurate post-training quantization with weights and activations in 4-bit precision is challenging; prior works still have substantial accuracy degradation when quantizing to low precision for both weights and activations ( Xiao et al., 2023 ; Liu et al., 2024b ; Shao et al., 2023 ; Ashkboos et al., 2024 ) . The reason that these methods struggle to retain accuracy in aggressive regimes like 4-bit quantization is due to challenges with numerical outliers which skew the quantization range ( Kim et al., 2023 ; Dettmers et al., 2022 ; Dettmers et al., 2023 ) , as well as disproportionately sensitive values ( Kim et al., 2023 ) . To address this challenge, previous works have applied mixed-precision quantization, where a mixture of high-precision and low-precision quantization is used to preserve accuracy by retaining particularly sensitive or hard to quantize parts of the network in higher precision ( Zhao et al., 2024 ; Ashkboos et al., 2023 ) . However, existing methods typically perform coarse-grained mixed precision (e.g., at the granularity of rows of a matrix ( Zhao et al., 2024 ; Ashkboos et al., 2023 ) ). As oulined in Figure 2 , the distribution of sensitive and hard-to-quantize values is unstructured in both LLM weights and activations; existing coarse-grained mixed-precision quantization methods cannot fully adapt to these unstructured distributions. Our work addresses the challenges with existing mixed-precision quantization methods by leveraging fine-grained mixed-precision (FGMP) quantization, in which we identify small, contiguous regions of hard-to-quantize weights and activations to be retained in a higher-precision data format while the majority of weights and activations are quantized to a lower-precision format. We pursue mixed-precision quantization at much finer block granularity. We define a block as a small ( < 100 elements) 1-dimensional vector along the dot product dimension of each tensor (as in ( Huang et al., 2024 ) ). We perform FGMP quantization by independently assigning each block to either a higher-precision or lower-precision data format (see Figure 2 ). This provides improved accuracy since it allows for more flexible precision assignment, adapting to different distributions of sensitive values (both inter-layer and intra-layer). However, although FGMP is a promising approach for maintaining model accuracy with the majority of values in reduced precision, it is challenging to leverage the potential efficiency benefits due to the difficulty of performing computation using different precisions for adjacent blocks. There are also distinct challenges with applying FGMP quantization to activations, as we need to be able to efficiently identify hard-to-quantize activation blocks online during inference to quantize the output activations of each layer. Even if we leveraged low-precision datapaths for the majority of computations, software implementations for dynamically selecting precisions when quantizing blocks of activations may be impractical. Our work aims to address these challenges through hardware-software co-design in order to enable fine-grained mixed-precision quantization while

Read the complete text