Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu · 2026

Read Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay by Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06797.txt.

Description

Binghui Li and five co-authors work out the best way to schedule a learning rate when training follows functional scaling laws. Those laws split progress into learning signal and forgetting noise. In a kernel regression setting, task difficulty and model capacity are each captured by a single exponent. For a fixed training horizon, the paper characterizes the schedule that minimizes final loss under a stability constraint.

A sharp transition emerges. Easy tasks are best served by decaying the learning rate from the start. Hard tasks favor holding the largest stable rate for most of training before a final decay, the warmup-stable-decay pattern. In both cases the decay exponent depends only on model capacity, so difficulty decides when to decay and capacity decides how. The authors also show that exact decay shapes often matter little, though overly slow final decay can stall progress.

Catalog record

CREATORBinghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06797

Browse the full library · About the Tower of Babel