Metis: Training LLMs with FP4 Quantization
Hengjie Cao Mengyi Chen Yifeng Yang Ruijun Huang Fang Dong, Jixian Zhou Anrui Chen Mingzhi Dong Yujiang Wang Jinlong Hou, Yuan Cheng Fan Wu Fan Yang Tun Lu Ning Gu Li Shang, Shanghai Innovation Institute Huawei · 2025
Read Metis: Training LLMs with FP4 Quantization by Hengjie Cao Mengyi Chen Yifeng Yang Ruijun Huang Fang Dong, Jixian Zhou Anrui Chen Mingzhi Dong Yujiang Wang Jinlong Hou, Yuan Cheng Fan Wu Fan Yang Tun Lu Ning Gu Li Shang, Shanghai Innovation Institute Huawei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00404.txt.
Introduction
Hengjie Cao leads a team of sixteen authors who present Metis in 2025, a framework for training large language models with four-bit floating point quantization. Quantization rounds numbers to a coarse grid to save compute. The barrier to low-bit training is anisotropy in singular value spectra, where a few huge values stretch the numeric range and distort the signal.
Metis works in the spectral domain, splitting such spectra into narrower sub-distributions that are quantized separately. Two traits of the dominant spectral subspace, its preservation under sparse random sampling and under random projection, keep decomposition cost negligible. On an eight-billion-parameter LLaMA-3 model trained on one hundred billion tokens, the method holds up at four-bit precision.
Catalog record
| CREATOR | Hengjie Cao Mengyi Chen Yifeng Yang Ruijun Huang Fang Dong, Jixian Zhou Anrui Chen Mingzhi Dong Yujiang Wang Jinlong Hou, Yuan Cheng Fan Wu Fan Yang Tun Lu Ning Gu Li Shang, Shanghai Innovation Institute Huawei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-00404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Metis: Training LLMs with FP4 Quantization Hengjie Cao Mengyi Chen Yifeng Yang Ruijun Huang Fang Dong, Jixian Zhou Anrui Chen Mingzhi Dong Yujiang Wang Jinlong Hou, Yuan Cheng Fan Wu Fan Yang Tun Lu Ning Gu Li Shang, Shanghai Innovation Institute Huawei Abstract This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLMs). These spectra are dominated by a small fraction of large singular values, inducing wide numerical ranges that cause quantization bias and severe spectral distortion, ultimately degrading training performance. This work presents Metis , a spectral-domain quantization framework that partitions anisotropic spectra into narrower sub-distributions for independent quantization, thereby reducing errors and preserving spectral structure. To minimize overhead, Metis leverages two key properties of the dominant spectral subspace: preservation via sparsely random sampling and preservation via random projection, reducing decomposition cost to a negligible level. On LLaMA-3 8B trained with 100B tokens, Metis enables robust W4A4G4 training with FP4 quantization of weights, activations, and gradients, yielding only a 0.4% training loss gap and a 0.1% degradation in downstream accuracy relative to BF16. Beyond matching BF16 fidelity, Metis also surpasses our implementation of Nvidia’s recently announced (yet to be publicly released) FP4 recipe, consistently achieving lower loss and higher downstream accuracy while incurring significantly lower computational overhead. The code implementation for Metis is available at: https://anonymous.4open.science/r/Metis-quantization-644B . 1 Introduction Training large language models (LLMs) with low-bit quantization of parameters, activations, and gradients offers substantial gains in efficiency, cost, and scalability. In recent years, progress has advanced from FP32 to BF16 and, more recently, to FP8 training ( Micikevicius et al. 2022 ; Peng et al. 2023 ; Perez et al. 2023 ) . Looking ahead, Nvidia’s Blackwell technical report shows that the NVFP4 format reduces memory consumption by 1.8× and accelerates General Matrix Multiplications (GeMM) by 7× compared to FP8, underscoring the efficiency potential of FP4 training ( Alvarez et al. ; Devleker & Ghodsian ) . However, pushing the training frontier further down to FP4 is not a straightforward continuation: FP4 imposes exponentially tighter constraints on precision and dynamic range, which conflict with the inherently wide distributions of parameters, activations, and gradients. This study investigates the origins of these wide distributions and analyzes their impact on training stability and effectiveness under FP4 quantization. The key findings are outlined below and visualized in Fig. 1 . Anisotropy is universal in modern LLMs. In weight, activation, and gradient matrices, a small fraction of singular values dominate, yielding a highly imbalanced spectrum. This phenomenon is consistently observed across model architectures and parameter scales up to 671B. Anisotropy induces wide numerical distributions. Wide distributions of weights, activations, and gradients arise from the superposition of singular components: larger components contribute to the large-value region, whereas smaller ones concentrate near zero. This spread originates from variability in singular values, which projects aligned components into entries of corresponding magnitudes. Quantization bias induces spectral distortion. Commonly used block-level low-bit quantization introduces bias that disproportionately favors large values, thereby reducing the effective resolution for small values. Under anisotropy, this effect leads to severe distortion in spectral space: smaller singular components incur substantially larger value errors and direction perturbations than their larger counterparts. Figure 1: Overview of anisotropy and its impact on quantization, illustrated using a gradient matrix from LLaMA-3 8B. (A) Singular value spectrum exhibits strong anisotropy, with a few singular values dominating the spectrum. (B) The wide matrix distribution arises from the superposition of singular components: large components (e.g., i=0) drive the high-value region, while small components concentrate near zero. (C) Quantization bias disproportionately rounds many small values to zero. (D–E) In spectral space, smaller singular components incur substantially larger relative quantization errors in singular values and more severe perturbations in singular directions. Details corresponding to (A) in Section 2.1 , (B) in Section 2.2 , and (C–E) in Section 2.3 . Inspired by these findings, we propose Metis , a quantization framework that preserves the spectral structure under FP4 formats and allows robust FP4 training. Metis operates in the spectral domain, applying decomposition to weight, activation, and gradient matrices to disentangle dominant from long-tailed singular components, thereby allowing quantization over substantially narrower distributions. A central challenge is the computational complexity of spectral decomposition. To address this, we leverage two key structural properties revealed in our empirical study: (i) Subspace Preservation via Sparsely Random Sampling , where the dominant subspace estimated from a sparsely random sampled subset is reliably generalized to the whole batch; (ii) Subspace Preservation via Random Projection , where the dominant subspace can be faithfully captured within a reduced hidden dimension via random projection. Building on these insights, Metis renders the decomposition cost negligible by employing sparse random sampling over sequences and random projections on the hidden dimension, each reducing complexity by approximately two orders of magnitude. Metis enables robust W4A4G4 training by quantizing all GeMM matrices to 4-bit floating point. On an 8B LLaMA-3 model ( et al. 2024 ) trained with 100B tokens from the DCLM dataset ( Li et al. 2025 ) , Metis narrows the gap to BF16 to only a 0.4% increase in training loss and a 0.1% degradation in downstream accuracy. Compared to our implementation of NVIDIA’s recently announced (but not yet publicly released) FP4 recipe ( Devleker & Ghodsian ) , Metis consistently achieves lower training loss and higher downstream accuracy while incurring significantly less computational overhead. 2 Analysis Anisotropy emerges as a key structural factor underlying the wide distributions observed in weights, activations, and gradients. This section analyzes these anisotropic matrices and examines how they misalign with low-bit quantization schemes. Unless otherwise specified, all experiments are conducted on the LLaMA-3 8B model trained on 100B tokens from the DCLM dataset. 2.1 Anisotropy: A Universal Property of Modern LLMs For a matrix \mathbf{M}\in\mathbb{R}^{m\times n} , we perform Singular Value Decomposition (SVD) to obtain singular values \{\sigma_{i}\}_{i=1}^{\min(m,n)} , left singular vectors \{\mathbf{u}_{i}\}\in\mathbb{R}^{m} , and right singular vectors \{\mathbf{v}_{i}\}\in\mathbb{R}^{n} , such that \mathbf{M}=\sum_{i=1}^{\min(m,n)}\sigma_{i}\mathbf{u}_{i}\mathbf{v}_{i}^{\top}. We assume singular values are sorted in descending order, i.e., \sigma_{1}\geq\sigma_{2}\geq\dots\geq\sigma_{r}>0 with r=\min(m,n) . Anisotropy is characterized by a spectrum where a few leading singular values dominate, yielding a highly imbalanced distribution across directions. Previous studies have reported anisotropy in activation matrices ( Ethayarajh 2019 ; Mu et al. 2017 ; Puccetti et al. 2022 ; Rudman & Eickhoff 2023 ; Yu et al. 2021 ) , and our analysis further demonstrates that it is universal across weights, activations, and gradients. As shown in Figure 2 (A), their singular value spectra exhibit pronounced anisotropy, with only 0.63%, 3.15%, and 2.91% of components (identified by the elbow point of maximum curvature) dominating the spectra of weights, activations, and gradients, respectively. We further validate this pattern on publicly released models, including the Qwen family ( Bai et al. 2023 ) and DeepSeek-R1 ( Liu et al. 2024 ) , where weight matrices from 7B to 671B parameters consistently show fewer than 3% of singular values dominating the spectrum, confirming anisotropy as a universal property across architectures and scales. Figure 2: Analysis of weight, activation, and gradient matrices (layer 32, FeedForward(FFN)). (A) The singular value spectra exhibit strong anisotropy, with only 0.63%,