MoR: Mixture of Representations for Mixed-Precision Training
Bor-Yiing Su, Peter Dykas, Mike Chrzanowski, Jatin Chhugani · 2025
Read MoR: Mixture of Representations for Mixed-Precision Training by Bor-Yiing Su, Peter Dykas, Mike Chrzanowski, Jatin Chhugani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22804.txt.
Introduction
Bor-Yiing Su, Peter Dykas, Mike Chrzanowski, and Jatin Chhugani present early work on training neural networks in mixed numerical precision without losing accuracy. Low-precision formats speed training and save memory, but picking which tensors can safely use them is tricky. Their framework, Mixture of Representations, inspects each tensor, and even parts of tensors, and picks a number format based on the values it actually contains.
Concrete algorithms switch dynamically between the compact FP8 format and the wider BF16 format, at both whole-tensor and sub-tensor levels. In their experiments the approach keeps model quality on par with established methods while putting more than ninety-eight percent of tensors into FP8. Because it needs no fine-grained partitioning schemes, the authors argue it can pair with other low-precision techniques and make training robust across datasets and partition choices.
Catalog record
| CREATOR | Bor-Yiing Su, Peter Dykas, Mike Chrzanowski, Jatin Chhugani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoR: Mixture of Representations for Mixed-Precision Training Bor-Yiing Su, Peter Dykas, Mike Chrzanowski, Jatin Chhugani Abstract Mixed-precision training is a crucial technique for scaling deep learning models, but successful mixed-precision training requires identifying and applying the right combination of training methods. This paper presents our preliminary study on Mixture-of-Representations (MoR), a novel, per-tensor and sub-tensor level quantization framework that dynamically analyzes a tensor’s numerical properties to select between a variety of different representations. Based on the framework, we have proposed and experimented concrete algorithms that choose dynamically between FP8 and BF16 representations for both per-tensor and sub-tensor level granularities. Our universal approach is designed to preserve model quality across various quantization partition strategies and datasets. Our initial findings show that this approach can achieve state-of-the-art results with 98.38% of tensors quantized to the FP8 format. This work highlights the potential of dynamic, property-aware quantization while preserving model quality. We believe this approach can generally improve the robustness of low precision training, as demonstrated by achieving FP8 accuracies that are on par with existing approaches without the need for fine-grain partitioning, or can be used in combination with other training methods to improve the leverage of even lower precision number formats such as NVFP4. 1 Introduction The scaling law [ 10 ] suggests that model performance increases with the number of parameters and the volume of training data, fueling a race to train ever-larger foundation models to achieve state-of-the-art performance on industry benchmarks. For instance, the Llama 3.1 405B model was trained on 24,576 H100 GPUs for months [ 8 ] , the Megatron-Turing NLG 530B model on 4,480 GPUs for months [ 21 ] , the GPT-4 model was estimated to have been trained on over 20,000 GPUs for three months [ 17 ] , and the DeepSeek-V3 671B model is trained with 2.788M GPU hours [ 5 ] . Consequently, training large foundation models is time-consuming, power-hungry, and expensive. This has led to significant investment in optimizing the training workflow to accelerate performance and reduce resource consumption. Among the most effective optimizations is the use of low-precision numerical representations. Using low-precision representations offers numerous benefits: (a) From a compute perspective, modern hardware offers significant speedups for lower-precision numerical operations. For example, an NVIDIA H100 GPU delivers 2x the FLOPS for FP8 GEMM operations compared to BF16 [ 2 ] , while the newer GB200 and GU300 GPUs provide 4x and 6x FLOPS for FP4 GEMM, respectively [ 16 , 14 ] . (b) From a bandwidth perspective, loading a vector of FP8 values requires only half the memory bandwidth of loading a BF16 vector of the same length. Therefore, low-precision representations directly reduce the latency of both computation and memory access, significantly accelerating training. (c) Moreover, applying low-precision formats during training enables their direct use for inference, creating consistency between the two phases and obviating the need for post-training quantization (PTQ) or quantization-aware training (QAT). The formal specification of the FP8 format can be found in [ 11 ] . It defines two FP8 formats: E4M3 and E5M2. The E4M3 format has 4 bits for exponent, and 3 bits for mantissa; while the E5M2 format has 5 bits for exponent, and 2 bits for mantissa. But FP8 training is not only about the FP8 formats. In order to increase the representation accuracy, improve the training stability, and also retain the model quality, many prior arts define comprehensive recipes for FP8 training. [ 7 ] and [ 11 ] suggested current and delayed per-tensor scaling. Essentially, defining a scaling factor for the full tensor to shift the numerical range of the tensor to the representable range by the format type. Delayed scaling computes this scaling factor based on historical absolute maximum (amax) values of the tensor; while current scaling computes this scaling factor based on the amax of the current tensor. [ 24 ] suggested partitioning the tensor into blocks, and compute one scaling factor per block. Llama3 [ 8 ] uses per-channel scaling that computes one scaling factor per row or column based on the dot product direction. DeepSeekV3 [ 5 ] combines both sub-channel scaling and per-block scaling into the recipe, and proposes using 1\times 128 sub-channel scaling for the activation tensor, and 128\times 128 per-block scaling for the weight tensor to achieve a balance between fine grained control and computational overhead. The micro-scaling format [ 18 , 19 , 12 ] suggested using a fine-grained 1\times 32 sub-channel block to compute the scaling factor. The NVFP4 format [ 6 ] suggested using a fine-grained 1\times 16 sub-channel block to compute the scaling factor. Another common strategy is to selectively apply FP8 quantization. [ 22 , 11 ] used E4M3 format in the forward pass, and uses E5M2 format in the backward pass. [ 23 ] used per-group scaling for the optimizer states, and per-tensor scaling for the activation tensors. [ 4 ] suggested using fp4 format for the majority of the training, but fall back to higher precision (FP32/BF16) at the late stage of training. [ 13 ] applied nvfp4 for most of the layers, but left the first two and last eight layers in BF16, it also studied how reducing the number of BF16 layers affect model quality. All of these prior arts focus on selecting the right methods and determining their appropriate application point (location and time) to create a low-precision training recipe that preserves model quality. What we are trying to accomplish, however, is to start gaining insights into why a specific recipe can achieve model quality on par with the baseline. Therefore, we developed the MoR framework, which attempts to capture invariance through tensor analysis. In this paper, we also showed that relative error can serve as a reasonable invariance for FP8 training. While works like DeepSeekV3 [ 5 ] and MXFP8 [ 12 ] achieve good results because their fine-grained partition ensures that the quantization error is small for all tensors, we illustrated that using relative error as an invariance allows us to obtain on-par model quality with much coarser-grained partitions. The relative error invariance we use is conservative and serves as a sufficient, but not a necessary, condition. We plan to continue working in this direction to explore invariance metrics that are more efficient and can be used for more aggressive quantization formats (such as NVFP4). Our primary contributions are as follows: 1. We propose a new strategy for computing the scaling factor that ensures consistency across the entire tensor, regardless of the chosen scaling granularity. 2. We introduce a novel dynamic quantization framework that makes real-time decisions based on numerical properties at runtime, allowing the model to adapt precision throughout the training process. 3. Using this framework, we develop and evaluate tensor-level and sub-tensor-level Mixture-of-Representations (MoR) recipes that dynamically select FP8 and BF16 types, achieving substantial efficiency without compromising quality. 2 Group Amax Mantissa Scaling A major challenge for low-precision quantization is that it requires mapping a tensor’s wide dynamic range to the limited representable range of the target format, such as FP8. For example, the E4M3 FP8 format can only represent positive values between 2^{-9} and 448; and the E5M2 FP8 format can only represent positive values between 2^{-16}