Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
Meng Xin, Sweta Priyadarshi, Jingyu Xin, Bilal Kartal, Aditya Vavre, Asma Kuriparambil Thekkumpate, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Ido Shahaf, Akhiad Bercovich, Kinjal Patel, Suguna Varshini Velury, Chenjie Luo, Zhiyu Cheng, Jenny Chen, Chen-Han Yu, Wei Ping, Oleg Rybakov, Nima Tajbakhsh, · 2026
Read Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery by Meng Xin, Sweta Priyadarshi, Jingyu Xin, Bilal Kartal, Aditya Vavre, Asma Kuriparambil Thekkumpate, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Ido Shahaf, Akhiad Bercovich, Kinjal Patel, Suguna Varshini Velury, Chenjie Luo, Zhiyu Cheng, Jenny Chen, Chen-Han Yu, Wei Ping, Oleg Rybakov, Nima Tajbakhsh, (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20088.txt.
Introduction
Meng Xin and colleagues distil large models into aggressively compressed versions. Their technical report presents quantization-aware distillation for models squeezed to the NVFP4 format, a four-bit number format. Quantization alone usually costs accuracy. Distillation trains the compressed student to imitate a full-precision teacher, recovering much of the loss.
The report covers both language models and vision-language models. It shares best practices the team found for the recovery process. The distillation loss is a KL divergence between teacher and student outputs. Four-bit inference slashes memory and energy costs for serving models. A reliable recipe for recovering accuracy makes such deployment realistic. The report is written for practitioners facing the same squeeze.
Catalog record
| CREATOR | Meng Xin, Sweta Priyadarshi, Jingyu Xin, Bilal Kartal, Aditya Vavre, Asma Kuriparambil Thekkumpate, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Ido Shahaf, Akhiad Bercovich, Kinjal Patel, Suguna Varshini Velury, Chenjie Luo, Zhiyu Cheng, Jenny Chen, Chen-Han Yu, Wei Ping, Oleg Rybakov, Nima Tajbakhsh, |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20088
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery Meng Xin, Sweta Priyadarshi, Jingyu Xin, Bilal Kartal, Aditya Vavre, Asma Kuriparambil Thekkumpate, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Ido Shahaf, Akhiad Bercovich, Kinjal Patel, Suguna Varshini Velury, Chenjie Luo, Zhiyu Cheng, Jenny Chen, Chen-Han Yu, Wei Ping, Oleg Rybakov, Nima Tajbakhsh, Oluwatobi Olabiyi, Dusan Stosic, Di Wu, Song Han, Eric Chung, Sharath Turuvekere Sreenivas, Bryan Catanzaro, Yoshi Suhara, Tijmen Blankevoort, Huizi Mao Abstract Abstract. This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-precision teacher model into a quantized student model using a KL divergence loss. While applying distillation to quantized models is not a new idea, we observe key advantages of QAD for today’s LLMs: 1. It shows remarkable effectiveness and stability for models trained through multi-stage post-training pipelines, including supervised fine-tuning (SFT), reinforcement learning (RL), and model merging, where traditional quantization-aware training (QAT) suffers from engineering complexity and training instability; 2. It is robust to data quality and coverage, enabling accuracy recovery without full training data. We evaluate QAD across multiple post-trained models including AceReason Nemotron, Nemotron 3 Nano, Nemotron Nano V2, Nemotron Nano V2 VL (VLM), and Llama Nemotron Super v1, showing consistent recovery to near-BF16 accuracy. NVFP4 checkpoints Nemotron 3 Nano 30B-A3B , Nemotron Nano 9B V2 , Nemotron Nano 12B V2 VL , Llama 3.1 Nemotron Nano VL 8B QAD codes : Megatron-LM version , NeMo version , HuggingFace Transformers version 1 Introduction The rapid expansion of large language models (LLMs) has increased the demand for more efficient numerical formats to lower computational cost, memory demand, and energy consumption during training and inference. 8-bit floating point formats (FP8 and MXFP8) have emerged as popular data types for accelerated training of LLMs ( 36 ; 9 ) . Recent advances in narrow-precision hardware ( 42 ) have positioned 4-bit floating point (FP4) as the next logical step ( 7 ; 6 ; 26 ; 50 ) , delivering a two- to three-fold boost in arithmetic performance and reducing memory usage by half compared to FP8. NVFP4, which features a smaller block size (16 vs. 32 for MXFP4), FP8 scale factors (E4M3) for fine-grained scaling, and second-level FP32 scaling for a larger dynamic range, has demonstrated superior accuracy to INT4 and MXFP4 on many models ( 12 ) . For very large LLMs, NVFP4 with post-training quantization (PTQ) shows decent accuracy on different benchmarks. However, for small LLMs, the accuracy drop from PTQ is often non-negligible. There are many efforts to incorporate quantization into the training process. NVFP4 quantized training ( 46 ) has demonstrated promising convergence on pretraining tasks. With the primary goal of training speedup, models trained with NVFP4 are still evaluated in BF16. Quantization-aware training (QAT) ( 16 ) is an effective training-based method for inference accuracy recovery, which has shown great results since the CNN era. Many QAT methods reuse the same pipelines and training objectives as the original high-precision models. However, this approach faces significant challenges for modern LLMs in real practices, such as: 1. Complex training pipelines : Modern LLMs undergo multi-stage post-training ( 3 ; 4 ) , such as SFT, RL, model merging, making it difficult to replicate the original training procedure. 2. Data availability and quality : The original training data of open models may not be available, and public datasets are often of worse quality. This technical report evaluates quantization-aware distillation (QAD) for NVFP4 inference accuracy recovery on post-trained LLMs. QAD uses the original full-precision model as a teacher and trains the quantized model as a student using KL divergence loss rather than task-specific objectives. We demonstrate the following key findings: 1. QAD aligns the quantized model to the high-precision model better than QAT. 2. QAD works effectively for models trained through multi-stage post-training pipelines including SFT & RL, demonstrating remarkable stability. 3. QAD is robust to incomplete data coverage and can recover accuracy even with partial domain data, enabling cross-domain knowledge transfer. The report is organized as follows: Section 2 provides a background on NVFP4 and quantization methods; Section 3 describes the QAD method and training setup and demonstrates key properties through comprehensive results across LLMs and VLMs; Section 4 provides detailed analysis of design choices and the impact of training data. 2 Background and Related Work 2.1 NVFP4 Format and Post-Training Quantization NVFP4 Format . NVFP4 is a 4-bit floating-point format designed for efficient training and inference on modern GPU architectures ( 46 ; 1 ) . Compared to FP8, NVFP4 offers 2-3× higher arithmetic throughput and approximately 1.8× memory reduction. NVFP4 extends the MXFP4 format ( 50 ) with a smaller block size (from 32 to 16) and two-level scaling (per-block E4M3 scales plus per-tensor FP32 scale). The smaller block size enables more localized adaptation to data distributions, while E4M3 scales provide non-power-of-two scaling factors for lower quantization error, and the second-level FP32 scale extends the overall dynamic range ( 1 ) . Post-Training Quantization (PTQ) . PTQ is a simple and cost-effective method that requires no training ( 41 ; 62 ; 61 ) . It involves a process called calibration that determines quantization parameters (e.g., scale factors and zero points) using a small representative dataset (calibration set). With PTQ, training and inference are completely decoupled, making it attractive for practitioners without access to original training pipelines or computational resources for fine-tuning. The simplest PTQ method is max calibration, which sets the scale factor such that the maximum absolute value in the calibration data maps to the maximum representable value in the quantized format. Despite its simplicity, max calibration works surprisingly well in many cases. More sophisticated calibration methods have been proposed, including MSE-based calibration ( 16 ) , KL divergence minimization ( 37 ) , and learnable clipping thresholds ( 8 ) . More sophisticated PTQ methods have been proposed to minimize quantization error through advanced techniques. For example, AdaRound ( 40 ) and BRECQ ( 21 ) optimize weight rounding and block-wise reconstruction, respectively. For LLMs specifically, ZeroQuant ( 64 ) and GPTQ ( 13 ) apply layer-wise calibration with efficient approximations. Other approaches focus on preserving sensitive information (AWQ ( 22 ) ) or optimizing quantization parameters via gradient descent (OmniQuant ( 51 ) ). A prominent direction is transformation-based PTQ, which applies reversible transformations (e.g., rotations, affine transformations) to suppress outliers and flatten distributions; examples include SmoothQuant ( 63 ) , QuaRot ( 2 ) , QuIP# ( 60 ) , SpinQuant ( 29 ) , and FlatQuant ( 55 ) . Additionally, low-rank approximation methods like SVDQuant ( 54 ) and EoRA ( 25 ) have been proposed to mitigate quantization error by absorbing outliers or compensating for information loss. These advanced methods typically achieve better accuracy than simple max calibration, especially for lower bit-widths. PTQ works quite well for FP8 on most LLMs. For NVFP4, PTQ also works well for very large models (See Appendix C ). However, PTQ often struggles with small models and sensitive tasks. Recent work has highlighted that common PTQ algorithms often fail to improve over baseline NVFP4 performance because the small block size neutralizes traditional outlier mitigation