Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu · 2026
Read Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay by Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06797.txt.
Description
Binghui Li and five co-authors work out the best way to schedule a learning rate when training follows functional scaling laws. Those laws split progress into learning signal and forgetting noise. In a kernel regression setting, task difficulty and model capacity are each captured by a single exponent. For a fixed training horizon, the paper characterizes the schedule that minimizes final loss under a stability constraint.
A sharp transition emerges. Easy tasks are best served by decaying the learning rate from the start. Hard tasks favor holding the largest stable rate for most of training before a final decay, the warmup-stable-decay pattern. In both cases the decay exponent depends only on model capacity, so difficulty decides when to decay and capacity decides how. The authors also show that exact decay shapes often matter little, though overly slow final decay can stall progress.
Catalog record
| CREATOR | Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06797