RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm
Yongyi Yang, Jianyang Gao, Wei Hu · 2025
Read RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm by Yongyi Yang, Jianyang Gao, Wei Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03717.txt.
Introduction
Yongyi Yang, Jianyang Gao, and Wei Hu present a fast quantization method that squeezes large language models with little data in this 2025 paper. Shrinking a trained model cuts its running cost sharply. Older shrinking methods demand heavy calibration data. They also lock users into fixed bit widths. Their framework, RaanA, has two new parts. One is a randomized quantizer variant tuned for speed and accuracy.
The other assigns bit widths layer by layer by sensitivity. Sensitive layers keep more bits; sturdy ones get fewer. The mix keeps quality while saving the most space. Speed is a headline claim against rival methods. Data efficiency matters where calibration sets are private or scarce. The paper is an AI efficiency methods study.
Catalog record
| CREATOR | Yongyi Yang, Jianyang Gao, Wei Hu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03717
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm Yongyi Yang, Jianyang Gao, Wei Hu Abstract Post-training Quantization (PTQ) has become a widely used technique for improving inference efficiency of large language models (LLMs). However, existing PTQ methods generally suffer from crucial limitations such as heavy calibration data requirements and inflexible choice of target number of bits. In this paper, we propose RaanA, a unified PTQ framework that overcomes these challenges by introducing two novel components: 1) RaBitQ-H, a variant of a randomized vector quantization method RaBitQ, designed for fast, accurate, and highly efficient quantization; and 2) AllocateBits, an algorithm that optimally allocates bit-widths across layers based on their quantization sensitivity. RaanA achieves competitive performance with state-of-the-art quantization methods while being extremely fast, requiring minimal calibration data, and enabling flexible bit allocation. Extensive experiments demonstrate RaanA’s efficacy in balancing efficiency and accuracy. The code is publicly available at https://github.com/FFTYYY/RaanA . 1 Introduction The rapid advancement in deep learning has demonstrated that increasing the size of neural networks (NNs) often leads to significant improvements in performance across various tasks ( Kaplan et al., 2020 ; Betker et al., 2023 ; Achiam et al., 2023 ; Grattafiori et al., 2024 ) . Large language models (LLMs), such as GPT ( Brown et al., 2020 ; Achiam et al., 2023 ) and LLaMA ( Touvron et al., 2023 ; Grattafiori et al., 2024 ) , have exhibited remarkable capabilities, but their deployment remains a challenge due to substantial computational and memory requirements. Among the various strategies developed to improve LLM inference, Post-Training Quantization (PTQ) has emerged as a widely adopted approach for optimizing the memory usage and inference speed of LLMs, balancing between accuracy and efficiency ( Frantar & Alistarh, 2022 ; Frantar et al., 2023 ; Lin et al., 2024 ; Guan et al., 2024 ; Lee et al., 2024 ; Chee et al., 2023 ) . The essential idea of PTQ is to reduce the numerical precision of the parameters of a trained model, replacing high precision floating-point representations with lower precision alternatives, which in turn reduces memory consumption and alleviates memory bandwidth constraints. Optimal Brain Quantizer (OBQ) is one of the earlier influential methods in PTQ. It formulates the quantization problem as an optimization task, where the core objective is to minimize the layer-wise quantization error \widehat{\boldsymbol{W}}^{(\ell)*}=\arg\min_{\widehat{\boldsymbol{W}}\in\mathcal{C}}\left\|{\boldsymbol{X}}^{(\ell)}{\boldsymbol{W}}^{(\ell)}-{\boldsymbol{X}}^{(\ell)}\widehat{\boldsymbol{W}}\right\|_{\mathcal{F}}^{2} , where {\boldsymbol{X}}^{(\ell)} , {\boldsymbol{W}}^{(\ell)} and \widehat{\boldsymbol{W}} are the input feature, original weight matrix and quantized weight matrix of layer \ell (a linear transformation layer) respectively. This approach has led to significant improvements in quantization accuracy by optimizing the quantization parameters (rescale, zero-point etc.) according to layer-wise behavior approximation. OBQ has become a foundation of PTQ research, with many recent methods building upon its framework ( Frantar et al., 2023 ; Lee et al., 2024 ; Lin et al., 2024 ; Chee et al., 2023 ) . Despite their effectiveness, existing approaches based on the OBQ framework typically suffer from notable limitations: • First, the OBQ framework only controls the error of the output of each layer and does not account for the overall error in the model output; more importantly, it treats all layers identically, whereas in practice, their importance can vary: the error in the first layer propagates through all subsequent layers, while that in the last layer affects only itself. Omitting this hierarchy in error sensitivity can lead to suboptimal quantization strategies. • Second, the optimal solution of the OBQ framework highly depends on the input {\boldsymbol{X}}^{(\ell)} which, however, is not constant across batches, and as a result, all the quantization methods based on this framework require a lot of calibration data to accurately estimate the input features (more precisely, the so-called layer-wise Hessian {\boldsymbol{X}}^{(\ell)\top}{\boldsymbol{X}}^{(\ell)} ) ( Hassibi et al., 1993 ) . This high requirement of data not only limits the efficiency of quantization, but also the generalization performance of the method, as it can perform worse if the data distribution is far from its calibration data distribution. • Moreover, the OBQ framework directly replaces each weight matrix with a quantized weight matrix. Since we are only concerned with the output of a layer rather than its internal structure, there is potential for greater flexibility by allowing additional operations within a layer. The above-mentioned limitations are not exclusive to OBQ-based methods, but also shared by many other PTQ methods. For example, Norm-Tweaking ( Li et al., 2024 ) , although only a plugin for adjusting LayerNorm parameters, also highly relies on calibration data, and EasyQuant ( Tang et al., 2024 ) , although it does not require calibration, only controls layer-wise error and is limited to entry-wise float-point rounding. In this paper, we propose a novel PTQ framework that overcomes the above issue, based on the following ideas. RaBitQ-H: Adopting Efficient Vector Quantization Methods to LLM Quantization If we don’t stick to preserving the structure of the original matrix multiplication and embrace a broader class of operations, it is possible to bring in much stronger tools from vector quantization. Specifically, in this paper we adopt RaBitQ ( Gao & Long, 2024 ; Gao et al., 2024 ) , a state-of-the-art vector quantization algorithm that enjoys very fast computation and well-controlled error-bound. However, RaBitQ is originally designed to solve approximate nearest neighbor search (ANNS) and can not be directly adopted to LLM quantization (see the detailed explanation in Section 5 ). To address this issue, in this paper we propose RaBitQ-H, a variant of RaBitQ, to address the challenges of adopting RaBitQ to LLM quantization. AllocateBits: Allocating Bits Unevenly Across Layers As mentioned before, it is suboptimal to uniformly assign computational resources across all layers, as some layers can be more important than others. There has been a research direction called Mixed Precision Quantization (MPQ), which allows different numbers of bits used in each layer ( Pandey et al., 2023 ; Guan et al., 2024 ; Behtash et al., 2025 ) . However, existing MPQ methods are mostly based on heuristics and are typically limited to binary choices of bit-widths (e.g. 4-bits v.s. 8-bits), making them not sufficiently flexible. In this paper, we propose a simple and effective approach to estimate the importance of each layer and formulate the bit allocation task as an integer programming problem that allows arbitrary bit-width choices. By solving this integer program, we obtain an optimal bit allocation strategy across layers. In this paper, we propose RaanA : a novel PTQ framework combining Ra BitQ-H an d A llocateBits . RaanA enjoys the following superiority: 1. Extremely Fast and Device-Independent : Unlike other state-of-the-art PTQ methods that generally require hours or even days to perform on large models, RaanA generally only requires tens of minutes on 70b models. Additionally, RaanA is largely device-independent, with most of its computation can be efficiently performed on CPUs, reducing its reliance on GPU devices. 2. Minimal Calibration Required : RaanA requires only a few or even zero samples for calibration, as opposed to most existing methods that typically require a huge amount of data. 3. Strong Performance : RaanA performs comparably to state-of-the-art PTQ methods. Notably, it remains effective even at extremely low bit-widths (e.g., <3 average bits), a regime where many other methods struggle. 4. High Flexibility :