Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model

Blake Bordelon, Francesco Mori · 2026

Read Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model by Blake Bordelon, Francesco Mori (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04774.txt.

Description

Choosing a learning rate is one of the most fiddly parts of training a neural network, usually settled by trial and error. Blake Bordelon and Francesco Mori build a theory of optimal schedules in a 2026 study of a solvable random feature model trained by stochastic gradient descent.

Their optimal schedule comes from both numerical optimization and optimal control theory. Two regimes emerge. In the easy phase, the rate decays as a polynomial. In the hard phase, it follows a warmup, stable, then decay pattern. Adding momentum and adjusting batch size each helped in specific phases. Tests on image classification and a language model matched the predicted phase behavior.

Catalog record

CREATORBlake Bordelon, Francesco Mori
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04774

Browse the full library · About the Tower of Babel