FP4 All the Way: Fully Quantized Training of LLMs
Brian Chmiel Maxim Fishman †, Ron Banner Daniel Soudry ∘ † Intel, Israel · 2025
Read FP4 All the Way: Fully Quantized Training of LLMs by Brian Chmiel Maxim Fishman †, Ron Banner Daniel Soudry ∘ † Intel, Israel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19115.txt.
Introduction
Brian Chmiel and co-authors showed this 2025 feat: training a large language model almost entirely in four-bit numbers. Training normally uses far wider numbers, and the cost is enormous. Four-bit floating point slashes memory and energy, if accuracy survives. The team tested design choices like block size, scaling style, and rounding rules on data sets up to 200 billion tokens.
One format, where blocks of sixteen tiny numbers share a scale, worked best. Random rounding on the backward pass and plain rounding on the forward pass kept training stable. They also found a noise threshold below which tiny gradients stop teaching. A seven-billion-parameter model trained on 256 accelerators matched the standard baseline on downstream tasks. Code accompanies the paper.
Catalog record
| CREATOR | Brian Chmiel Maxim Fishman †, Ron Banner Daniel Soudry ∘ † Intel, Israel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19115
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FP4 All the Way: Fully Quantized Training of LLMs Brian Chmiel Maxim Fishman †, Ron Banner Daniel Soudry ∘ † Intel, Israel Abstract We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately 3 \sqrt{3} times the quantization noise, quantized training becomes less effective. Leveraging these insights, we successfully train a 7-billion-parameter model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to a standard BF16 baseline, confirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in https://github.com/Anonymous1252022/fp4-all-the-way . 1 Introduction The rapid advancement of Large Language Models (LLMs) has led to unprecedented breakthroughs in natural language understanding and generation. State-of-the-art models now scale to hundreds of billions of parameters, enabling remarkable capabilities across diverse applications. However, this progress comes at a significant cost, with training and inference demanding immense computational power and memory bandwidth. As model sizes grow, hardware constraints become a major bottleneck, necessitating innovations in numerical precision and memory-efficient architectures. Until recently, the dominant numerical format for pretraining Large Language Models (LLMs) was BF16, which provided a balance between precision and efficiency. However, as model sizes and dataset scales have grown, researchers have explored lower-precision alternatives to improve computational efficiency and reduce memory requirements. A few pioneering studies have demonstrated that full training in FP8 is not only feasible but also effective. [ 13 ] showcased the potential of FP8 training on small-scale datasets (100B), [ 8 ] extended it to trillions of tokens dataset with a 7B parameters model, and [ 6 ] demonstrated FP8’s viability at an even larger scale, successfully training a massive 671B parameter Mixture-of-Experts (MoE) model on a vast dataset, achieving state-of-the-art results. FP8 is quickly emerging as the new standard for large-scale LLM training. As research continues to push precision boundaries further, the next logical step is exploring FP4, which promises even greater efficiency while maintaining training stability and model accuracy. Recently, NVIDIA introduced its new GPU architecture, Blackwell [ 1 ] , marking a major milestone as the first GPU to support FP4 matrix multiplication in hardware. This advancement enables nearly 2× acceleration in matrix multiplication throughput compared to FP8, significantly boosting performance and energy efficiency for large-scale deep learning workloads. Blackwell supports two distinct FP4 formats—MXFP4 and NVFP4—each with different design choices in block size and scale encoding. In this paper, we aim to systematically investigate the full range of block sizes and scaling formats for FP4 training, analyzing their impact on accuracy and stability and highlighting the advantages of the NVFP4 format. Beyond the choice of numerical format and block size, another critical aspect of low-precision quantization is the rounding mode used during quantization. The two most commonly used rounding modes are round-to-nearest (RtN), which deterministically maps each value to its closest representable value, and stochastic rounding (SR), which probabilistically rounds based on the distance to neighboring values to reduce quantization bias. While prior empirical work [ 19 , 4 ] has highlighted the importance of SR, particularly in the backward pass for stabilizing gradient updates, its benefits have largely been observed heuristically. In this work, we provide an analysis of noisy gradients training and identify the point where they stop being effective and show the importance of SR in low-precision training. Recent studies have explored the use of FP4 formats for training large language models, yet none achieved full quantization across all key components—weights, activations, and gradients. In contrast, our approach introduces, for the first time, a fully quantized FP4 training framework that addresses all these components simultaneously, and on significantly larger datasets. Table 2 highlights these important distinction. This end-to-end FP4 training setup enables us to evaluate the practical viability of low-precision methods at scale. Moreover, we evaluate our FP4 models on various downstream tasks—showing on-par results with the BF16 baseline—demonstrating that efficiency gains do not come at the cost of quality. We make several key contributions: 1. FP4 Format Optimization. We conduct comprehensive experiments on block size and scaling formats for FP4 training, revealing that block sizes below 16 elements offer diminishing returns on accuracy. Our comparison of different exponent-mantissa configurations (E1M6 through E8M0) found the best performance was achieved by E4M3—the format used in NVFP4, validating NVIDIA’s hardware design choices. 2. Split Rounding Strategy. We establish that combining stochastic rounding in the backward pass with round-to-nearest in the forward pass significantly improves training stability and final model accuracy in FP4 training compared to using either method alone throughout the training process. 3. Precision Transition Analysis. We provide a theoretical framework that identifies the critical point at which FP4 precision becomes less effective for continued training progress. Specifically, when the full-precision gradient standard deviation falls below \sqrt{3} times the quantization noise standard deviation. We suggest, at the end of the training, to apply a higher precision Quantization Aware Finetuning (QAF) to increase the signal-to-noise ratio higher above this threshold and quickly converge to the baseline. 4. End-to-End Large-Scale FP4 Training. We successfully train a 7B-parameter LLM entirely in FP4 precision on a trillion tokens, using 256 Intel Gaudi2 accelerators. While a slight gap in final training loss is observed compared to the BF16 baseline, it is fully closed through the brief QAF phase. This leads to downstream task performance on par with BF16, confirming the practical viability of FP4 for real-world large-scale applications. 2 Related work Quantization is a key area of research in the effort to compress neural networks for more efficient deployment and reduced resource consumption. The two predominant approaches in this space are Post-Training Quantization (PTQ) [ 22 , 11 , 9 ] and Quantization-Aware Training (QAT) [ 7 , 3 , 12 ] . PTQ focuses on converting pre-trained models to low-bit representations without additional training, making it attractive for rapid deployment, especially in inference scenarios. In contrast, QAT incorporates quantization effects during model training or fine-tuning, enabling the network to adapt and maintain accuracy under low-precision constraints. Both methods have seen significant advancements, with recent work demonstrating competitive performance at 4-bit precision [ 9 ] and below [ 20 ] . Fully Quantized Training (FQT) is a more challenging task than PTQ or QAT, as it requires training from scratch with low-precision weights, activations, and gradients to accelerate all matrix multiplications. Until recently, applying FQT beyond 16-bit precision was considered difficult due to instability and convergence issues. However, recent works have demonstrated its feasibility. [ 13 ] presented the first FQT of a large language model in FP8 on a dataset of up to 100 billion tokens. [ 8 ] extended this to 2