Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets

Kristi Topollai, Anna Choromanska · 2026

Read Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets by Kristi Topollai, Anna Choromanska (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16731.txt.

Introduction

Topollai and Choromanska explain why quantizing optimizer states stalls LLM training. Low-precision moving averages round many updates back to the stored value. The state then sits stale, and adaptation lags the nominal decay. Their model predicts one-step stalling odds after initialization. Stalling builds over time in a trackable way. The picture explains why resetting the state helps. A reset briefly restores responsiveness in a stale state. Their theory picks good reset periods, and tests confirm the gains.

Catalog record

CREATORKristi Topollai, Anna Choromanska
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65530
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets Kristi Topollai, Anna Choromanska Abstract Quantizing optimizer states is becoming an important ingredient of memory-efficient large-scale pre-training, but the resulting optimizer dynamics remain only partially understood. We study low-precision exponential moving average (EMA) optimizer states and show how quantization can cause many nominal updates to round back to the same stored value, making the state effectively stale and slowing adaptation beyond what the nominal decay would suggest. We then develop a simple predictive model of stalling that estimates one-step stalling probabilities and characterizes how stalling builds up over time after the initialization. This perspective provides a mechanistic explanation for why optimizer-state resets help in low precision: once a quantized EMA becomes effectively stale, resetting it can temporarily restore responsiveness. Motivated by this picture, we derive a simple theory-guided method for choosing useful reset periods, showing that in low precision the key question is not only whether resets help, but when they should be applied. Experiments in controlled simulations and LLM pre-training show that suitable reset schedules recover the performance lost to low-precision state storage while substantially reducing optimizer-state memory. 1 Introduction Modern large-scale pre-training relies heavily on optimizers such as AdamW 19 ; 23 , Lion 4 , Shampoo 14 , SOAP 35 , and Muon 22 , which maintain running statistics, typically exponential moving averages (EMAs), to smooth gradients, adapt step sizes, or construct preconditioners. At the same time, modern training pipelines increasingly store these states in reduced precision to lower memory cost and improve throughput 7 ; 28 ; 40 ; 39 . This makes optimizer-state quantization practically important, but its effect on the actual dynamics of the optimizer remains insufficiently understood. A related empirical observation is that resetting optimizer states can improve pre-training 18 ; 17 ; 9 . These resets are usually motivated through spike mitigation or generic training stability. However, in our experiments their benefit is strongly precision-dependent: resets help substantially when EMA states are stored in BF16 or lower precision, but are often much less useful in full precision. This suggests that state quantization and state resets should not be viewed separately: understanding why quantized states become ineffective is also the key to understanding when resets help. In this paper, we study that connection through the dynamics of quantized EMAs. Consider a generic recursion s_{t}=\beta s_{t-1}+(1-\beta)\Delta_{t}, where s_{t} is an optimizer state, \beta is a decay coefficient close to 1 , and \Delta_{t} denotes the new update information. This form captures the state evolution of many widely used optimizers. In low precision, each step first forms a higher-precision update and then maps it back to the storage format. When the increment is too small relative to the local spacing of the floating-point grid, the update rounds back to the same stored value, so the state does not change. We refer to this phenomenon as state-update stalling , also known as signal swamping and unchanged-state dynamics in low-bit EMA updates 15 ; 37 ; 42 . In high-decay recursions, repeated stalling slows adaptation, effectively lengthens the memory of the EMA beyond what the nominal decay suggests, and makes quantized states substantially less responsive than their full-precision counterparts. This viewpoint also helps explain several recurring features of low-precision training. In particular, it clarifies why large \beta_{2} values become more problematic as precision decreases, why stochastic rounding 13 ; 26 ; 6 helps but does not fully remove the problem when stalling is severe, and why the first and second moments of Adam-like methods respond differently to quantization. Most importantly, it explains why resets can help and why their timing matters: once a quantized EMA has become effectively stale, resetting it can restore responsiveness, but resetting too early discards useful averaging. The key question is therefore not only whether resets help, but when they should be applied. We formalize these effects through a simple predictive model of quantized EMA dynamics. The model combines local floating-point grid geometry with a short-horizon approximation to estimate one-step stalling probabilities and to characterize the finite post-initialization period during which a quantized EMA remains meaningfully responsive. This responsive startup window provides a natural basis both for understanding when resets should help and for choosing useful reset periods as a function of decay and precision. We validate the resulting predictions in controlled simulations and LLM pre-training, showing that suitable reset schedules recover the performance lost to low-precision state storage while substantially reducing optimizer-state memory. 2 Related Work Optimizers for large-scale pre-training. Modern LLM pre-training increasingly follows compute-optimal scaling and open training recipes 16 ; 33 ; 34 ; 11 , most of which rely on optimizers with persistent state. Beyond momentum SGD and Adam/AdamW 19 ; 23 , this includes memory-reduced adaptive methods such as Adafactor 30 , SM3 1 and Adam-mini 44 , structured preconditioners such as Shampoo 14 and SOAP 35 , and more recent alternatives such as Lion 4 and Muon 22 . Low-precision and quantized pre-training. Recent low-precision pre-training builds on FP8 formats 24 , with systems such as FP8-LM 28 and COAT 40 demonstrating end-to-end FP8 LLM training, and more recent work extending to FP4 through mixed-precision or native low-bit stacks 38 ; 5 ; 3 . Closest to our setting is direct quantization of optimizer states: 8-bit and 4-bit states for Adam-like methods 7 ; 20 , 4-bit Shampoo preconditioners 39 , and 8-bit quantized Muon states 12 . Our contribution is complementary: rather than proposing a new quantization or scaling scheme, we analyze a specific failure mode of quantized EMA states, state-update stalling under requantization, and use it to explain when stochastic rounding and periodic resets help. Optimizer state resetting. Restarting or state resetting have a long history in optimization 27 and were later adapted to neural network training 36 . More recent work shows that clearing optimizer states can help in non-stationary settings such as reinforcement learning 2 . In LLM training, methods such as SPAM 18 ; 17 and 9 use momentum resets mainly for spike suppression and stabilization, while 32 show that implicit resets via adapting the momentum coefficient can also improve performance. In contrast, we show that in low precision the benefit of resets is instead tied to quantized-state stalling: resets help by temporarily restoring responsiveness once EMA states become effectively stale. 3 State update stalling under quantized EMA states Preliminaries. We consider exponential moving averages (EMAs) of the form x_{t}=\beta x_{t-1}+(1-\beta)s_{t}, (1) where s_{t} is a stochastic signal (e.g., g_{t} or g_{t}^{2} ) and \beta\in(0,1) is the decay coefficient. In the quantized setting, the EMA state is stored in low precision but updated in high. Let x_{t}^{q} denote the stored state, let D(\cdot) denote dequantization (equivalently, an exact cast to FP32), and let Q(\cdot) denote quantization back to the storage format. One update therefore reads x_{t-1}=D(x_{t-1}^{q}) , forms the FP32 update x_{t}=\beta x_{t-1}+(1-\beta)s_{t} , and writes back x_{t}^{q}=Q(x_{t}) . Since D(\cdot) is just a cast, no new quantization error is introduced during the read phase; all new errors enter on the write phase through Q(\cdot) . Definition 3.1 (State update stalling) . We say that state update stalling occurs at step t if x_{t}^{q}=x_{t-1}^{q}.

Read the complete text