On the Optimizer Dependence of Neural Scaling Laws
Vansh Ramani*, Shourya Vir Jain* · 2026
Read On the Optimizer Dependence of Neural Scaling Laws by Vansh Ramani*, Shourya Vir Jain* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29387.txt.
Introduction
Vansh Ramani and Shourya Jain show that neural scaling powers depend on the optimizer. It is usually treated as fixed by design and data. In careful random-feature tests, they measure it across five optimizer variants and six spectral settings. Preconditioned optimizers yield steeper scaling, and at s near one, typical of natural language, the full natural gradient hits a much larger value than gradient descent. Whether this shift transfers to large-scale training remains open, and outlooks should account for the choice.
Catalog record
| CREATOR | Vansh Ramani*, Shourya Vir Jain* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40782 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29387
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Optimizer Dependence of Neural Scaling Laws Vansh Ramani*, Shourya Vir Jain* Abstract The scaling exponent α \alpha in neural scaling laws L ( N ) ∝ N − α L(N)\propto N^{-\alpha} is commonly treated as a fixed constant set by architecture and data. We present evidence that α \alpha depends systematically on the optimizer. In controlled random-feature regression experiments—the canonical theoretical framework for neural scaling—we measure α \alpha across five optimizer variants and six spectral conditions. Preconditioned optimizers consistently yield steeper scaling (larger α \alpha ), with the α \alpha -shift increasing across most of the tested spectral range, peaking near s = 1.5 s=1.5 , and remaining large at s = 2.0 s=2.0 . At s ≈ 1.0 s\approx 1.0 (characteristic of natural language), the full natural gradient achieves α ≈ 0.31 \alpha\approx 0.31 versus α ≈ 0.12 \alpha\approx 0.12 for gradient descent—a 2.6 × 2.6\times larger fitted exponent that, within the random-feature model, compounds with each model-size doubling. Whether and how this exponent shift transfers to large-scale LLM training—where recent evidence suggests the advantage may attenuate with scale—remains an important open question. Our results imply that scaling-law forecasts should account for optimizer choice, and we provide a spectral diagnostic predicting when advanced optimizers will pay off. \hldauthor Department of Computer Science and Engineering, Indian Institute of Technology Delhi and Department of Electrical Engineering, Indian Institute of Technology Delhi *Equal contribution Accepted at the 4th Workshop on High-dimensional Learning Dynamics (HiLD), ICML 2026. 1 Introduction Neural scaling laws—the observation that test loss decreases as a power law L(N)\propto N^{-\alpha} with model size N —govern how frontier laboratories allocate compute ( 11 ; 8 ) . The exponent \alpha determines returns to scaling: larger \alpha means each doubling of parameters yields greater loss reduction. Current practice treats \alpha as a fixed constant fit from a few training runs, then extrapolated to set budgets for orders-of-magnitude larger models. Yet recent results suggest \alpha may not be fixed. 5 showed that second-order optimizers—Muon, SOAP ( 15 ) , and Shampoo ( 7 ) —can deliver \sim\!1.4\times compute speedups over AdamW when their hyperparameters are scaled carefully. 12 confirmed this at frontier scale: Muon matches AdamW at 52% of training FLOPs. However, the picture is not uniformly positive: 16 report that the Muon/SOAP speedup over AdamW decays with model size, dropping to {\sim}1.1\times at 1.2B parameters—suggesting the effect may attenuate at scale. These conflicting findings motivate a more precise question: does Muon improve the constant factor in the scaling law, or does it change the exponent itself? And if the exponent shifts, does the shift persist, shrink, or vanish at scale? This distinction matters enormously. A constant-factor improvement saves a fixed proportion of compute. An exponent improvement, if it persists, compounds : the advantage grows with every doubling. If preconditioned optimizers change \alpha —even partially—then forecasts calibrated under AdamW may underestimate returns to scaling under Muon. We investigate this question in the random-feature regression framework of 13 and 4 , where data spectrum, optimizer preconditioner, and model capacity can be cleanly isolated. This framework—rooted in the natural gradient theory of 1 and its modern extensions ( 14 ) —lets us study how preconditioning interacts with spectral structure to determine scaling exponents. Contributions. We make four contributions: (1) the first systematic measurement of \alpha as a joint function of optimizer type and data spectral decay, across five preconditioner variants and six spectral conditions ( Section 3 ); (2) a mechanistic explanation via an informal proposition: preconditioned optimizers equalize per-mode learning rates, increasing well-learned modes at given N ( Section 2 ); (3) a comparison to published optimizer-scaling results, treated as qualitative external context rather than a parameter-by-parameter fit ( Section 4 ); and (4) a falsifiable prediction: the \alpha -shift should generally grow with spectral decay before finite-size or finite-budget saturation; on flat-spectrum data ( s<0.3 ), preconditioned optimizers should provide negligible scaling improvement ( Section 5 ). 2 Setup: Random-Feature Regression with Optimizer Preconditioning 2.1 The Random-Feature Scaling Model We adopt the random-feature regression framework of 13 , which provides a solvable model for neural scaling laws. Data distribution. Inputs \mathbf{x}\in\mathbb{R}^{D} are drawn from \mathcal{N}(0,\Sigma) with eigenvalues \lambda_{i}=i^{-(1+s)} , i=1,\ldots,D , where s>0 is the spectral exponent . Natural language embeddings typically exhibit s\approx 0.8\text{--}1.2 based on Zipfian frequency distributions ( 11 ; 2 ) . Teacher function. f^{*}(\mathbf{x})=\sum_{k=1}^{K^{*}}v_{k}\,\sigma(\mathbf{w}^{*}_{k}\cdot\mathbf{x}) , with ReLU activation \sigma and teacher coefficients v_{k}^{2}\propto k^{-b} (source exponent b ). Student model. \hat{y}(\mathbf{x})=\sum_{j=1}^{N}a_{j}\,\sigma(\mathbf{w}_{j}\cdot\mathbf{x}) , where \{\mathbf{w}_{j}\} are fixed random weights from \mathcal{N}(0,I_{D}/D) and trainable parameters are \mathbf{a}\in\mathbb{R}^{N} . The student minimizes L=\mathbb{E}[(\hat{y}(\mathbf{x})-f^{*}(\mathbf{x}))^{2}] . This produces power-law scaling L(N)\propto N^{-\alpha} where \alpha depends on s and b ( 13 ) . Crucially, the top-layer problem is a linear regression, allowing different optimizers to be implemented as preconditioners with no confounding from feature learning. 2.2 Optimizer Variants as Preconditioners We implement five variants, each preconditioning the gradient \nabla_{\mathbf{a}}L : • GD (baseline): \mathbf{P}=I . • Diagonal (AdamW proxy): \mathbf{P}=\diag(\mathbf{F}^{\top}\mathbf{F})^{-1/2} . • Full NG (Shampoo/K-FAC proxy): \mathbf{P}=(\mathbf{F}^{\top}\mathbf{F})^{-1} ; equivalent to solving the normal equations ( 1 ; 14 ) . • Sign-GD : \mathbf{a}\leftarrow\mathbf{a}-\eta\,\sign(\nabla L) . • Matrix-Sign (Muon proxy): \mathbf{P}=(\mathbf{F}^{\top}\mathbf{F})^{-1/2} ; equalizes singular values, analogous to Muon’s Newton-Schulz orthogonalization ( 9 ) . Here \mathbf{F}\in\mathbb{R}^{n\times N} is the random-feature matrix on training data. 2.3 Spectral Intuition: Why Preconditioning Shifts \alpha Under GD with power-law data, learning proceeds mode-by-mode from highest variance downward. At model size N , GD effectively resolves O(N^{\gamma}) spectral modes for \gamma<1 set by s and b . A preconditioner equalizes convergence rates, rescuing “underserved” low-variance modes. We state this as a spectral argument (not a formal theorem; a fully rigorous proof would require resolvent-trace analysis in the style of 4 ): Proposition 1 (Spectral Heuristic—Informal) . Consider random-feature regression with covariance \lambda_{i}\propto i^{-(1+s)} and preconditioner \mathbf{P} . Let \alpha_{\textup{GD}}(s) and \alpha_{\mathbf{P}}(s) denote the scaling exponents under GD and preconditioned GD respectively. Heuristically: (i) \alpha_{\mathbf{P}}(s)\geq\alpha_{\textup{GD}}(s) for any positive-definite \mathbf{P} that downweights high-variance modes relative to low-variance modes; (ii) the gap \Delta\alpha(s)=\alpha_{\mathbf{P}}(s)-\alpha_{\textup{GD}}(s) is expected to grow with s until finite-size or finite-budget effects become important; and (iii) \Delta\alpha(s)\to 0 as s\to 0 (flat spectrum). Supporting argument. When \mathbf{P} fully inverts the covariance (full NG), every mode converges at the same rate, so all N capacity units are used uniformly. Under GD, the effective capacity per mode decays with the eigenvalue, so the bottom modes remain under-learned. The fraction of “wasted” capacity grows with spectral steepness s , making the \alpha -gap larger. When the spectrum is flat ( s\to 0 ), GD already treats modes equally, so preconditioning adds nothing. Appendix D provides a detailed per-mode convergence-rate derivation supporting all three claims, including explicit convergence factors under GD, full NG, and Matrix-Sign ( Equations 2 and 3 ). 3 Simulation Results 3.1 Experimental Protocol We fix dimension D\!=\!1000 , teacher complexity K^{*}\!=\!100 , source exponent b\!=\!1.0 , and sweep six spectral conditions ( s from 0.25 to 2.0 ) and eight model sizes ( N from 25 to 5{,}000 ) across all five optimizers. Each configuration uses 10 random seeds, T\!=\!2000 gradient steps, and optimizer-specific step-size rules ( Appendix B ). Training set size scales with N