Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers
Nikhil Nayak, Julia White, Urchade Zaratiana, Kelton Zhang · 2026
Read Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers by Nikhil Nayak, Julia White, Urchade Zaratiana, Kelton Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20756.txt.
Introduction
Nikhil Nayak and nine colleagues correct random update bias in preconditioned language-model optimizers. Such optimizers are central to training, but their random rules are usually read as direct approximations to population descent with scaling. The authors show two finite-sample biases: gradient-scaler coupling from shared minibatches, and bias from inverting or rooting a noisy scaler estimate. Their single-batch framework uses cross-fitted scaling and variance-corrected inversion to address both.
Catalog record
| CREATOR | Nikhil Nayak, Julia White, Urchade Zaratiana, Kelton Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94414 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers Nikhil Nayak, Julia White, Urchade Zaratiana, Kelton Zhang Abstract Preconditioned optimizers are central to language model training, but their stochastic update rules are usually treated as direct approximations to population preconditioned descent. We show that this view misses two finite-sample biases. First, the gradient and preconditioner are typically estimated from the same minibatch, introducing gradient–preconditioner coupling bias. Second, even when the preconditioner estimate is unbiased, its inverse or inverse-root is generally biased because inversion is nonlinear. We propose a single-batch bias-correction framework that addresses both effects: cross-fitted preconditioning estimates the numerator and preconditioner from independent microbatch groups, while variance-corrected inversion uses microbatch variability to subtract the leading delta-method bias term. The framework applies to diagonal moment, diagonal curvature, and matrix preconditioning methods, instantiated in AdamW, Sophia, and Shampoo. Bias correction reduces held-out pretraining loss on Qwen2.5-0.5B by 0.15 0.15 , 0.07 0.07 , and 0.11 0.11 nats, respectively; the effects on mixed-quality pretraining and downstream instruction tuning are consistently neutral-to-positive. Together, these results establish bias correction as a practical mechanism for reducing finite-sample update bias and improving the performance of preconditioned optimizers. 1 Introduction Large language models are usually trained with first-order stochastic optimizers such as SGD, Adam, AdamW, and their matrix- or curvature-aware variants. These methods work well in practice, but their update rules are usually treated as deterministic transformations of stochastic quantities. In particular, adaptive optimizers estimate both a descent direction and a geometry term from minibatches. The geometry term may be a scalar learning-rate estimate, a diagonal second-moment estimate, a curvature proxy, or a structured matrix preconditioner. This paper focuses on a statistical question that is usually implicit in optimizer design. For the population objective f(\theta)=\mathbb{E}_{z\sim\mathcal{D}}[\ell(\theta;z)] (1) and a population preconditioner P(\theta) , the ideal preconditioned update at parameter value \theta is u^{\star}(\theta)=P(\theta)^{-1}\nabla f(\theta), (2) whereas in practice we only observe minibatch estimates \hat{g} and \hat{P} and apply the standard adaptive update \hat{u}=\hat{P}^{-1}\hat{g}. (3) Even when \hat{g} is an unbiased gradient estimator and \hat{P} is an unbiased preconditioner estimator, the product in Equation ( 3 ) is not generally an unbiased estimator of Equation ( 2 ). There are two distinct sources of mismatch. First, if \hat{P} and \hat{g} are estimated from the same random minibatch, the two terms are statistically coupled. Second, even if \hat{P} is unbiased for P , the inverse \hat{P}^{-1} is generally biased for P^{-1} because inversion is nonlinear. These effects are not implementation mistakes. They arise from using finite stochastic estimates inside nonlinear adaptive update rules. The goal of this paper is to make these two biases explicit and propose a practical correction framework. The proposed method is single-step and single-batch: it does not require revisiting the same parameter value, which is unrealistic in neural network training. At a fixed \theta_{t} , the current training batch is split into microbatch groups. One group estimates the gradient, another estimates the preconditioner, and microbatch variation is used to correct the inverse-preconditioner estimate before the update is applied. We evaluate the framework on three optimizer families. AdamW represents diagonal moment-based adaptive optimization. Sophia represents diagonal curvature-based or lightweight second-order optimization. Shampoo represents matrix-preconditioned optimization. These three cases test whether the proposed bias correction is useful across increasingly structured preconditioners: diagonal moments, diagonal curvature, and matrix geometry. Contributions. This paper makes three contributions. 1. We give a statistical decomposition of the stochastic preconditioned update into two distinct finite-sample biases: gradient–preconditioner coupling, and the nonlinearity of inversion. 2. We propose a single-batch correction framework that combines cross-fitted preconditioning with a delta-method inverse correction estimated from microbatch variability, and we analyze its bias-reduction and convergence properties. 3. We instantiate the framework across diagonal moment, diagonal curvature, and matrix preconditioning (AdamW, Sophia, Shampoo) and evaluate it on pretraining and instruction tuning; the strongest empirical results are 0.1489 -, 0.0701 -, and 0.1103 -nat reductions in held-out pretraining cross-entropy for AdamW, Sophia, and Shampoo. 2 Related work Momentum, acceleration, and convergence guarantees. The study of faster first-order optimization predates modern adaptive methods. Robbins and Monro introduced stochastic approximation as a framework for iterative optimization from noisy observations ( 21 ) . Polyak’s heavy-ball method showed how momentum can accelerate deterministic iterative methods ( 18 ) , while Nesterov’s accelerated method achieved the optimal O(1/k^{2}) rate for smooth convex optimization with first-order information ( 16 ) . Polyak–Ruppert averaging later showed that averaging stochastic approximation iterates can recover optimal asymptotic behavior under classical conditions ( 17 ) . For nonconvex stochastic optimization, guarantees typically shift from convergence to a global minimum to convergence toward approximate stationarity; for example, 9 give complexity guarantees for stochastic first-order methods in smooth nonconvex problems. In deep learning, momentum remains central: 23 showed that carefully scheduled momentum and initialization allow SGD with momentum to train deep and recurrent networks effectively, and Lookahead wraps an inner optimizer with slow/fast weights to improve stability and reduce variance ( 27 ) . These works improve convergence rates, stability, or asymptotic behavior of the optimization trajectory. Our work addresses a different but compatible question: whether the stochastic preconditioned update used by such optimizers is itself a biased estimator of the corresponding population preconditioned direction. Adaptive diagonal optimizers. AdaGrad introduced coordinate-wise adaptive scaling from accumulated squared gradients, with both diagonal and full-matrix viewpoints ( 6 ) . Adam made this form of adaptivity the default in deep learning by combining exponential moving averages of first and second moments with initialization bias corrections for those moving averages ( 12 ) . AdamW decoupled weight decay from the adaptive gradient update, which is now standard in language model training ( 14 ) . Subsequent work identified convergence pathologies in Adam-like exponential moving average schemes and proposed AMSGrad-style long-memory fixes ( 20 ) . Other widely used variants target optimizer memory or large-batch stability: Adafactor factorizes second-moment accumulators to reduce memory from matrix size to row/column size ( 22 ) , while LAMB adds layerwise trust ratios for large-batch BERT training ( 26 ) . These methods design the form of the preconditioner, its memory footprint, or its large-batch scaling. Our work is orthogonal: given a stochastic preconditioner estimate, we study the bias introduced by multiplying it with a stochastic gradient and by applying a nonlinear inverse. This is also distinct from Adam’s standard bias correction, which corrects the initialization bias of exponential moving averages, not the finite-sample bias of the inverse preconditioner. Curvature and matrix preconditioning. Natural gradient methods use the Fisher geometry as the local metric for optimization ( 2 ) . K-FAC makes this idea practical for neural networks by approximating layerwise Fisher blocks with Kronecker factors ( 15 ) . Shampoo instead maintains structured second-moment matrices for tensor dimensions and applies inverse-root preconditioners ( 10 ) ; scalable implementations add numerical and systems refinements that make Shampoo practical for deep networks and language models ( 3 ) . SOAP further combines Shampoo’s eigenspace with Adam-like second-moment tracking to reduce the degradation from infrequent eigendecompositions ( 25 ) . In the diagonal-curvature direction, Sophia estimates a diagonal Hessian or Gauss–Newton–Bartlett-style curvature proxy and clips the resulting curvature-scaled update for language model pretraining ( 13 ) . These methods estimate richer geometry than