Deriving Hyperparameter Scaling Laws via Modern Optimization Theory

Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang, Niccolò Ajroldi, Bernhard Schölkopf, Antonio Orvieto · 2026

Read Deriving Hyperparameter Scaling Laws via Modern Optimization Theory by Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang, Niccolò Ajroldi, Bernhard Schölkopf, Antonio Orvieto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15958.txt.

Description

Shulgin and co-authors derive tuning rules for modern optimizers from fresh theory. Hyperparameter transfer across batch size and horizon leans on folklore rules. Their lens is the Linear Minimization Oracle family of methods. That family covers normalized SGD, signSGD, and Muon. Recent convergence bounds serve as the object to minimize. Closed power-law schedules drop out for key knobs. Learning rate, momentum, and batch size each get a budget law. Momentum and batch size interact in ways worth care, they note.

Catalog record

CREATOREgor Shulgin, Dimitri von Rütte, Tianyue H. Zhang, Niccolò Ajroldi, Bernhard Schölkopf, Antonio Orvieto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74084
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15958

Browse the full library · About the Tower of Babel