On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling

Moritz Haas, Sebastian Bordt, Ulrike von Luxburg, Leena Chennuru Vankadara . · 2025

Read On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling by Moritz Haas, Sebastian Bordt, Ulrike von Luxburg, Leena Chennuru Vankadara . (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22491.txt.

Introduction

Moritz Haas, Sebastian Bordt, Ulrike von Luxburg, and Leena Chennuru Vankadara explain why very large learning rates can train wide networks well. Their 2025 paper confronts a theory-practice gap: standard theory predicts instability at big rates and no feature learning at stable ones. Real networks, though, train stably at large rates and learn real features even when very wide.

The authors split the supposedly unstable regime in two. In one part, training truly blows up. In the other, outputs grow without bound while gradients and activations stay calm, a controlled divergence. At the edge of that calmer regime, a clean infinite-width limit exists under the standard loss. The finer map reconciles practice with theory and guides rate choices for large models.

Catalog record

CREATORMoritz Haas, Sebastian Bordt, Ulrike von Luxburg, Leena Chennuru Vankadara .
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE210 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22491

Browse the full library · About the Tower of Babel