Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations

Patrick Blumenberg, Thomas Graave, Tim Fingscheidt, Insitute for Communications Technology, Technische Universität Braunschweig, Germany · 2025

Read Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations by Patrick Blumenberg, Thomas Graave, Tim Fingscheidt, Insitute for Communications Technology, Technische Universität Braunschweig, Germany (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06653.txt.

Introduction

Blumenberg, Graave, and Fingscheidt present a 2025 paper on shrinking large language models so they use less memory. A common trick stores model weights in blocks at four bits each, with set patterns such as NF4 and AF4. The authors show those set patterns leave larger errors than needed. They set out to find the best pattern for each block instead.

Their family of quantizers, called BOF4, cuts the error below both older methods. They give both a math-based fix and a data-driven one and prove the two match in practice. A variant tweaks how each block is scaled and cuts error more. Tests also weigh how zero and large weights should be handled.

Catalog record

CREATORPatrick Blumenberg, Thomas Graave, Tim Fingscheidt, Insitute for Communications Technology, Technische Universität Braunschweig, Germany
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations Patrick Blumenberg, Thomas Graave, Tim Fingscheidt, Insitute for Communications Technology, Technische Universität Braunschweig, Germany Abstract Large language models (LLMs) demand extensive memory capacity during both fine-tuning and inference. To enable memory-efficient fine-tuning, existing methods apply block-wise quantization techniques, such as NF4 and AF4, to the network weights. We show that these quantization techniques incur suboptimal quantization errors. Therefore, as a first novelty, we propose an optimization approach for block-wise quantization. Using this method, we design a family of quantizers named 4-bit block-wise optimal float (BOF4), which consistently reduces the quantization error compared to both baseline methods. We provide both a theoretical and a data-driven solution for the optimization process and prove their practical equivalence. Secondly, we propose a modification to the employed normalization method based on the signed absolute block maximum (BOF4-S), enabling further reduction of the quantization error and empirically achieving less degradation in language modeling performance. Thirdly, we explore additional variations of block-wise quantization methods applied to LLMs through an experimental study on the importance of accurately representing zero and large-amplitude weights on the one hand, and optimization towards various error metrics on the other hand. Lastly, we introduce a mixed-precision quantization strategy dubbed outlier-preserving quantization (OPQ) to address the distributional mismatch induced by outlier weights in block-wise quantization. By storing outlier weights in 16-bit precision (OPQ) while applying BOF4-S, we achieve top performance among 4-bit block-wise quantization techniques w.r.t. perplexity. 1 Introduction Driven by scaling the transformer architecture to billions of parameters, large language models (LLMs) have achieved remarkable performance in language modeling tasks. However, their size poses significant challenges for deployment, particularly in memory-constrained settings. Numerous post-training quantization (PTQ) techniques have been developed to mitigate this, reducing the memory footprint of the weights and, in many cases, improving inference speed [ 1 , 2 , 3 , 4 ] . Fine-tuning imposes even greater memory demands, making it challenging to adapt LLMs on consumer-grade GPU hardware. To address this, Dettmers et al. [ 5 ] introduced QLoRA, a memory-efficient fine-tuning method that combines 4-bit quantization of pre-trained weights with low-rank adaptation (LoRA) [ 6 ] . For quantization, Dettmers et al. [ 5 ] propose 4-bit NormalFloat (NF4), a quantization method with a fixed codebook. This method normalizes blocks of network weights by their absolute maximum (block-wise absmax normalization). Unlike other, more accurate PTQ methods, NF4 quantizes weights without computing network activations based on calibration data, making the quantization process itself more efficient in terms of both time and memory, while maintaining acceptable accuracy degradation at 4 bits per weight. Dettmers et al. [ 5 ] claim that the NF4 codebook is information-theoretically optimal due to its equal utilization of the 16 reconstruction levels. However, Yoshida demonstrates that this claim is incorrect [ 7 ] . We add that equal utilization of reconstruction levels is not a theoretically justified criterion for the optimality of a quantizer. Yoshida [ 7 ] also proposes an alternative codebook (AF4) designed to address the shortcomings of NF4. In this work, we show that neither NF4 nor AF4 minimizes the quantization error of the network weights. For the first time, we provide a rigorous mathematical analysis of block-wise absmax quantization and explore multiple design variations through an experimental study. As a first contribution, we derive an expectation-maximization (EM) algorithm inspired by Lloyd’s algorithm [ 8 ] that computes the correct, information-theoretically optimal codebook for block-wise absmax quantization w.r.t. the mean absolute error (MAE) or mean squared error (MSE) criterion. Additionally, we propose an alternative normalization technique: Instead of normalizing blocks by their absolute maximum value, we normalize by the signed absolute maximum. This simple modification results in a significant reduction of the quantization error. Using our EM algorithm, we compute a family of optimal quantization codebooks which we refer to as 4-bit block-wise optimal float (BOF4), or BOF4-S when signed normalization is used. Furthermore, we identify that block-wise absmax quantization is sensitive to outlier weights affecting the distribution of the normalized weights. We address this by introducing an outlier-preserving quantization (OPQ) that stores outliers in 16-bit precision. When combined with BOF4-S, OPQ substantially improves perplexity over NF4 and AF4. The paper is structured as follows: Section 2 reviews related work on block-wise quantization. Section 3 outlines our mathematical analysis and novel quantization methods. Section 4 details the experimental setup, and Section 5 presents and discusses the results. We conclude in Section 7 . 2 Related Work Block-wise quantization based on blocks of input values normalized by their absolute maximum was introduced by Dettmers et al. [ 9 ] as a method for quantizing optimizer states during neural network training. Subsequent works [ 5 , 7 , 10 ] applied this technique to LLM network weights for memory-efficient fine-tuning. We refer to this quantization method as block-wise absmax quantization . 2.1 Block-Wise Absmax Quantization In block-wise absmax quantization network weights w_{b,i}\in\mathbb{R} are first grouped into blocks, with block indices b\in\mathcal{B}=\{1,\ldots,B\} , and indices of weights within a block i\in\mathcal{I}=\{1,\ldots,I\} , where B\in\mathbb{N} is the number of blocks, and I\in\mathbb{N} the block size. Then, the weights are normalized by the absolute maximum weight in their respective block: w^{\mathrm{max}}_{b}=\max_{i\in\mathcal{I}}|w_{b,i}|,\quad b\in\mathcal{B} (1) x_{b,i}=\frac{w_{b,i}}{w^{\mathrm{max}}_{b}}\in[-1,1],\quad i\in\mathcal{I},b\in\mathcal{B} (2) Next, each normalized weight x_{b,i} is quantized independently using scalar quantization. The absolute block maxima w^{\mathrm{max}}_{b} , commonly referred to as quantization constants , are stored in addition to the quantized weights for later decoding. Overall the block- dependent quantization function Q_{b}() for weights w_{b,i} is defined as Q_{b}(w_{b,i})=w^{\mathrm{max}}_{b}\cdot\tilde{Q}(\frac{w_{b,i}}{w^{\mathrm{max}}_{b}})=w^{\mathrm{max}}_{b}\cdot\tilde{Q}(x_{b,i}), (3) where \tilde{Q}() is a block- independent quantization function. 2.2 4-bit Block-Wise Quantization for LLMs In this section, we discuss the previous block-wise absmax quantization that our work builds upon. 4-bit NormalFloat (NF4) : NF4 [ 5 ] is a 4-bit scalar quanitzer for block-wise absmax quantization. The L=2^{4}=16 reconstruction levels \hat{x}(\ell),\;\ell\in\mathcal{L}=\{1,\ldots,L\} are computed based on quantiles of the assumed Gaussian network weight distribution p_{W}=\mathcal{N}(0,\sigma^{2}) . Dettmers et al. [ 5 ] claim that their construction leads to equal utilization of the 16 reconstruction levels. However, this was already shown to be incorrect by Yoshida [ 7 ] . Furthermore, an equal probability for all codebook points is not a general criterion for the optimality of a quantizer. Instead, quantization aims at rate-distortion optimality [ 11 ] . Accordingly, a codebook assigning equal probability to each codebook point is only optimal for uniformly distributed input data. This has been well-known for decades, most prominently through the necessary conditions for optimality that underpin Lloyd’s algorithm [ 8 ] . 4-Bit AbnormalFloat (AF4): Yoshida [ 7 ] analyzes the distribution of normalized network weights and performs direct minimization of the mean absolute error (MAE) to obtain a block-wise absmax quantization codebook for normally distributed network weights, named AF4. This quantizer aims to correct an oversight in the design of NF4 [ 5 ] , which does not account for the dependence of the distribution of normalized weights on the block size. However, Yoshida’s optimization method targets the minimum MAE of normalized weights \mathrm{MAE}(x_{b,i},\tilde{Q}(x_{b,i})) , instead of minimizing the end-to-end quantization error of the network weights \mathrm{MAE}(w_{b,i},Q_{b}(w_{b,i})) . Both NF4 and AF4 contain reconstruction levels at -1, 0, and 1, such

Read the complete text