Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
Taowen Liu, Marta Andronic, Deniz Gündüz, George A. Constantinides · 2025
Read Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding by Taowen Liu, Marta Andronic, Deniz Gündüz, George A. Constantinides (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00874.txt.
Introduction
Taowen Liu, Marta Andronic, Deniz Gunduz, and George Constantinides study how to train large language models in low precision without losing accuracy. Rounding numbers to fewer bits saves memory and compute, but the rounding noise can slow convergence and hurt the final model.
They focus on stochastic rounding, which rounds up or down at random so errors average out, and on how it interacts with batch size. Their theory and experiments show larger batches can offset the noise from cheaper back propagation at low precision. They also find that quantizing weights and quantizing activations affect gradient variance in different ways. The results point to practical settings for training capable models on limited edge hardware.
Catalog record
| CREATOR | Taowen Liu, Marta Andronic, Deniz Gündüz, George A. Constantinides |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding Taowen Liu, Marta Andronic, Deniz Gündüz, George A. Constantinides Abstract LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors—especially batch size—remains underexplored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during backpropagation . Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights. 1 Introduction Training large language models (LLMs) demands significant computational and memory resources. Mixed-precision training, using lower-precision formats, offers a crucial path to efficiency Micikevicius et al. 2017 ; Das et al. 2018 ; Wu et al. 2018 ; Lee et al. 2023 , enabling training on edge devices Kandala and Varshney 2024 . However, reduced precision introduces quantization noise, potentially hindering convergence. Stochastic rounding (SR) Croci et al. 2022 provides an attractive, unbiased alternative to deterministic rounding methods for managing this noise. As shown in Figure 1 , SR can maintain training stability and performance where round-to-nearest (RTN) fails, especially under aggressive quantization. Despite SR’s advantages, its practical interaction with mini-batch stochastic gradient descent (SGD), particularly the role of batch size in mitigating SR-induced noise, remains underexplored. Key questions persist: Can larger batch sizes—a common variance reduction tool—effectively counteract SR noise? How does this apply distinctly to quantizing shared model weights versus per-sample activations and gradients? And crucially, how does SR’s variance impact SGD convergence guarantees? Addressing these is vital for establishing theoretically grounded guidelines for optimal precision, rounding, and batch size selection, especially to unlock aggressive quantization for training LLMs on edge devices. Although prior work has explored mixed-precision training empirically Micikevicius et al. 2017 ; Blake et al. 2023 ; Peng et al. 2023 and theoretically analyzed aspects like gradient quantization for efficient communication ( Xia et al. 2021 ; Xia et al. 2022 ; Xia et al. 2025 ; Xia et al. 2024 ) or weight quantization Li et al. 2017 , a rigorous understanding of low-precision arithmetic within the backpropagation process itself, combined with the specific variance characteristics introduced by SR in a mini-batch context, remains limited. Chen et al. 2020 analyzed the impact of quantization error but did not explicitly incorporate batch size as a variable interacting with different sources of SR noise. This paper addresses this gap through a theoretical and empirical analysis of SR within mini-batch SGD. We explicitly model and differentiate the statistical properties of noise from quantizing shared weights versus per-sample activations and gradients during backpropagation. Our core theoretical finding, supported by empirical validation, is that increased batch sizes can effectively compensate for reduced precision during backpropagation by mitigating SR-induced variance from per-sample operations. Specifically, we demonstrate that reducing precision in activation and gradient quantization can be offset by a quantifiable increase in batch size—for instance, a 1-bit reduction may be balanced by at most a fourfold batch increase to maintain convergence quality, with practical increases often being milder. Our main contributions are: • We develop a comprehensive theoretical analysis that explicitly models the impact of mini-batch size on SGD convergence when employing SR for quantizing different components in the training pipeline. This framework distinguishes between noise originating from weight quantization and noise from per-sample activation and gradient quantization. • Leveraging our framework, we prove that the variance introduced by stochastically rounding per-sample activations and their gradients during backpropagation (specifically, for computing weight gradients like \widehat{A}_{in}^{T}\widehat{\nabla A}_{out} and input activation gradients like \widehat{\nabla A}_{out}\widehat{W}^{T} ) decays inversely with the mini-batch size ( 1/b ). This provides a theoretical underpinning for using larger batches to counteract reduced precision in these operations. • We conduct experiments on both image classification and LLM fine-tuning to validate our theoretical predictions. These experiments demonstrate SR’s superiority over RTN and confirm the predicted batch size scaling effect. From these results, we derive practical guidelines quantifying the trade-off, e.g., showing that a 1-bit precision reduction in activation/gradient quantization can be compensated by approximately doubling the batch size in practice to maintain similar convergence behavior. 2 Related Works 2.1 Mixed-Precision Training Mixed-precision training has emerged as a crucial approach for efficient LLMs training by quantizing weights, activations, and gradients. Typically, general matrix-matrix multiplication (GEMM) operations are performed in mixed-precision during gradient computation. Early research demonstrated the viability of training with reduced precision using FP16 Gupta et al. 2015 ; Micikevicius et al. 2017 and INT16 Das et al. 2018 . The field progressed with the development of FP8 -based methods Wang et al. 2018b ; Banner et al. 2018 ; Yang et al. 2020 and even 4-bit methods Sun et al. 2020 ; Chmiel et al. 2021 . Various techniques have been proposed to address the challenges of mixed-precision training, including loss scaling Micikevicius et al. 2017 , precision-aware parameter initialization Blake et al. 2023 , and blockwise quantization Peng et al. 2023 . 2.2 Quantization-Aware Training Quantization-aware training (QAT) represents another significant direction, focusing on learning quantized weights during training Hubara et al. 2018 . Researchers have explored diverse approaches, including uniform and non-uniform quantization methods Zhou et al. 2017 , mixed-precision quantization Dong et al. 2019 , and learned quantization strategies Esser et al. 2019 . Figure 1: Stochastic rounding (SR) achieves higher accuracy with larger batch sizes, while round-to-nearest (RTN) fails to converge at the same precision. 2 2 2 Validation accuracy of WideResNet-16 on CIFAR-10. 2.2.1 Stochastic Rounding Stochastic rounding (SR) Croci et al. 2022 ; Xia et al. 2021 ; Xia et al. 2022 ; Xia et al. 2025 ; Xia et al. 2024 is a probabilistic technique used to mitigate the errors introduced by quantization, especially in low-precision arithmetic by providing unbiased estimates of quantized values. Most theoretical analyses of mixed-precision training have focused on gradient quantization for communication efficiency or storage efficiency Xia et al. 2021 ; Xia et al. 2022 ; Xia et al. 2025 ; Xia et al. 2024 . Chen et al. 2020 proposed a framework for analyzing the impact of gradient quantization on convergence properties. SR randomly rounds the value to one of the two nearest levels, with probabilities proportional to the distance of the value from each level. SR can be efficiently implemented in hardware using a pseudo-random number generator such as a linear feedback shift register (LFSR). SR has been adopted in modern hardware implementations such as Graphcore processors Knowles 2021 . To define SR formally, we first define the threshold quantization function, which can express both round-to-nearest and stochastic rounding. Definition 1 ( Threshold Quantization Function ) . Let x be the number to be quantized, let \Delta be the precision level, and let \epsilon be the threshold. The threshold quantization function is defined as: \begin{split}\mathbb{Q}_{\Delta}(x,\epsilon)=\Delta\cdot\begin{cases}\lfloor\frac{x}{\Delta}\rfloor,\text{when }\frac{x}{\Delta}-\lfloor\frac{x}{\Delta}\rfloor<\epsilon\\ \lfloor\frac{x}{\Delta}+1\rfloor,\text{otherwise}\end{cases}\end{split} (1) Note: Specifically, when \Delta=0 , for any \epsilon , define \mathbb{Q}_{0}(x,\epsilon)=x . Definition 2 ( Round-to-nearest (RTN) ) . Given a real number x and a quantization step size \Delta , the RTN