On the Nonlinearity of Learning Rate Scaling for LLM Training
Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang · 2026
Read On the Nonlinearity of Learning Rate Scaling for LLM Training by Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29158.txt.
Introduction
Yang, Zhang, Xu and Zhang examine a rule of thumb for cutting training costs for large language models: the best learning rate obeys a log-linear scaling law in data and model size. If true, the law would let practitioners extrapolate the right rate from small runs instead of full-scale sweeps. Their study trains GPT-2-style models, from twenty-two to over seven hundred million parameters, on up to a hundred billion tokens.
The optimal learning rate bends upward at larger scales, so log-linear extrapolation goes wrong. The bend largely disappears when raw rates are replaced by the effective rate, the step size in normalised weight space. It also fades when extrapolation runs over data rather than model size. They explain the bend through weight norms, which reach equilibrium slowly when the optimal rate is small. Tests with an optimiser controlling the effective rate support this.
Catalog record
| CREATOR | Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Nonlinearity of Learning Rate Scaling for LLM Training Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang Abstract Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2–style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data D D extrapolation is used instead of model size N N extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation. 1 Introduction Efficiently training large language models requires carefully tuned hyperparameters, yet sweeping these hyperparameters directly at the target scale is often prohibitively expensive. Hyperparameter transfer —predicting effective hyperparameters at large scale from inexpensive smaller-scale runs—has therefore become a central tool in modern model development [ 44 , 22 ] . The learning rate is a particularly consequential hyperparameter, as it directly affects both training efficiency and stability. Accordingly, many prior works have studied the transfer of learning rates across training settings. One line of work is \mu P [ 44 ] : under this parameterization, the optimal learning rate remains stable across width scaling. Subsequent works have extended this framework to other settings such as depth scaling [ 6 , 5 , 9 ] . These approaches mostly focus on learning-rate transfer along the model-size scaling axis. Some recent work shows that \mu P style analysis, though very elegant, can be inaccurate when weight norm changes during training [ 20 , 48 ] . As a complement to \mu P-style analyses, and to enable transfer along the data-size axis, another line of work directly models the optimal learning rate \eta^{\ast} as a function of the data scale D and model size N . The most common assmuption is that \eta^{\ast} follows a log-linear, or power-law, relationship in D and N [ 17 , 4 , 22 ] : \log\eta^{\ast}(D,N)=a\log D+b\log N+c. Existing studies often fit the scaling law under a fixed search protocol and assess whether it extrapolates to larger model or data scales. We examine this paradigm from two perspectives. First, obtaining an accurate scaling-law fit can itself require substantial compute, but this cost is often omitted when comparing transfer strategies. To enable a fair comparison, we propose measuring distinct power-law relationships under the same compute budget. Second, rather than jointly fitting the dependence on the data scale D , the model scale N , and the learning rate, we further quantify the extrapolation efficiency of different transfer strategies on two choices of the transfer axis, the data scale D versus the model scale N , and two parameterizations of the learning rate, the learning rate versus the effective learning rate. Here the effective learning rate is defined as [ 42 ] \eta_{\mathrm{eff}}(t)=\|\hat{w}_{t+1}-\hat{w}_{t}\|_{2}, where \hat{w}=w/\|w\|_{2} denotes the normalized weight direction. Motivated by the above two perspectives, we conduct a systematic empirical study of learning-rate transfer across model and data scales. Specifically, we train GPT-2–style models [ 34 ] ranging from 22M to 707M parameters on 5B to 100B tokens, and hold out the largest 707M model as the test set for evaluating optimal learning-rate prediction. We release a comprehensive dataset 1 1 1 https://huggingface.co/datasets/neurips2026lrtransfer/LR-Transfer-Trajectory spanning model sizes, data scales, and learning rates, with dense coverage of loss trajectories and optimization statistics. Our main findings from these experiments are as follows. • We find that the optimal learning rate \eta^{\ast} at large data scale D and model size N is higher than predicted by log-linear extrapolation from smaller D and N . In contrast, the effective learning rate \eta_{\mathrm{eff}} exhibits substantially more linear scaling behavior ( Section 5 ). • Under controlled compute budgets, we quantitatively compare the extrapolation performance along the data-size and model-size axes. We find that fitting learning-rate scaling law along the data-size axis can yield much more accurate lr predictions: it incurs only 2\% additional training compute relative to using the true optimal learning rate, and requires less than 20\% of the compute of hyperparameter sweeping at the target scale (details in Table 1 ). • By analyzing the implicit dynamics of AdamW and the mapping between \eta and \eta_{\mathrm{eff}} , we explain the upward curvature observed in \eta^{\ast} . We further verify the linear scaling behavior of \eta_{\mathrm{eff}} by explicitly controlling it during training with the AdamH optimizer [ 42 ] ( Section 6 ). 2 Related Works Transferring learning rate across training scales. Hyperparameter tuning is crucial for the efficient training of large language models. In this work, we focus on the transferring the learning rate. One common approach is to fit parametric scaling laws, which model the optimal learning rate as a power-law function of the training scale, such as model size [ 17 , 33 ] , training loss [ 40 ] , or compute budget [ 7 ] . Later work refines this formulation by making the optimal learning rate depend on both model size and data size [ 4 , 48 ] , and 22 further study the joint tuning of learning rate and batch size. However, the extrapolation performance of these power-law relations, as well as the range of training scales over which they hold, remains insufficiently evaluated. We aim to address this gap. Another line of work for transferring learning rates across model scales is the theoretically motivated Maximal Update Parameterization ( \mu P) [ 44 ] . Later works extend the original width scaling to other training settings, such as depth scaling, different optimizers, and low-precision training [ 8 , 6 , 45 , 5 , 9 ] . However, \mu P focuses on model scaling and does not directly cover transfer to larger data scales. Effective learning rate. Many weight matrices in mordern architectures are scale-invariant due to normalization layers [ 15 , 2 ] : rescaling such weights by a positive scalar leaves the network output, and hence the loss, unchanged [ 37 , 12 ] . As a result, the relevant training dynamics is governed primarily by changes in weight direction rather than weight scale. Previous work shows that the effective learning rate is jointly determined by the learning rate and weight decay [ 23 , 19 ] . This perspective has further motivated new optimizer designs that explicitly control the weight norm and update norm [ 25 , 11 , 43 ] . In this work, we further show that the effective learning rate, rather than the learning rate, is the right quantity to consider for hyperparameter transfer. We discuss more related works on hyperparameter tuning and transfer in Appendix A . 3 Power Laws for Optimal Learning Rates Our experiments