A Proof of Learning Rate Transfer under μ \mu P

Soufiane Hayou · 2025

Read A Proof of Learning Rate Transfer under μ \mu P by Soufiane Hayou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01734.txt.

Introduction

Soufiane Hayou proves a claim practitioners of deep learning rely on but had not seen justified. Under the maximal update parameterization, known as muP, the best learning rate for a neural network stays put as the network gets wider. MuP is a way of scaling weights and learning rates with width designed to keep feature learning alive in the infinite-width limit.

For a linear multi-layer perceptron, Hayou shows the optimal learning rate converges to a nonzero constant as width grows. That is exactly the transfer property trainers exploit. They tune on small models and deploy on large ones. The property fails under standard parameterization and under the neural tangent parameterization, both analyzed for contrast. Intuitive proofs are backed by extensive experiments.

Catalog record

CREATORSoufiane Hayou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Proof of Learning Rate Transfer under μ \mu P Soufiane Hayou Abstract We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with μ \mu P, a neural network parameterization designed to “maximize” feature learning in the infinite-width limit. We show that under μ ​ P \mu P , the optimal learning rate converges to a non-zero constant as width goes to infinity, providing a theoretical explanation to learning rate transfer. In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parametrization (SP) and Neural Tangent Parametrization (NTP). We provide intuitive proofs and support the theoretical findings with extensive empirical results. \eta^{*} \mu P: good LR transfer Learning rate \eta Train loss \eta^{\star}\to 0 SP: optimum shifts Learning rate \eta Train loss Figure 1: Conceptual illustration of learning rate transfer. Left: Under \mu P, loss curves across widths share (approximately) the same optimal learning rate \eta^{*} . Right: Under SP, the optimal learning rate \eta^{\star}_{n} shifts toward 0 as width grows. Curves illustrating different widths (darker \Rightarrow wider). 1 Introduction The recent successes in AI are mostly fueled by scale: large neural networks trained on large corpuses of data. Given a fixed training dataset, the size of a neural network can be scaled by increasing the width (hidden dimension) and/or depth (number of layers). As we scale these dimensions, several hyperparameters (HPs) must be adjusted with scale to avoid numerical overflows. Motivated by this empirical observation, several works have explored the large-width limit of neural networks and its impact on optimal HPs. He et al. 2016 introduced the “ 1/\textrm{fan-in} ” initialization which normalizes the weights to achieve order one activations as width grows (Note that Neal 1996 was the first to introduce the “ 1/\textrm{fan-in} ” initialization in the context of Bayesian neural networks). The Neural Tangent Kernel (NTK, [ 21 ] ) was one of the first attempts to understand training dynamics of large-width neural networks. The authors showed that under the neural tangent parametrization, training dynamics converge to a kernel regime in the infinite-width limit, a phenomenon known as lazy training [ 6 ] . In this regime, neural features are almost identical to their values at initialization and training dynamics can be linearized around initialization. It quickly became clear that NTK regime does not represent practical training of neural network, which exhibit significant feature learning. Yang and Hu 2021 reverse-engineered this problem by investigating neural parametrizations that result in feature learning in the infinite-width limit and introduced the Maximal Update Parametrization ( \mu P) which sets precise scaling exponents for the initialization and learning rate. A nice by-product of \mu P is HP transfer, or where optimal HPs seem to converge as width increases, a very useful property since it allows tuning HPs on relatively small models and using them for larger models with no additional tuning cost (see Fig. 1 for a conceptual illustration). The authors conjectured that HP transfer resulted from the fact that \mu P achieves “maximal” feature learning, and therefore the limiting dynamics are “optimal” in the sense that no other limit (corresponding to other parametrizations) is better in terms of training loss, thus leading to the convergence of the optimal HPs as width grows. While this intuition is valid to some extent, to the best of our knowledge, no rigorous proof of HP transfer exists in the literature. Perhaps the most important hyperparameter is the learning rate, which generally requires some tuning in practice. Motivated by this, we focus on learning rate transfer in this work and present the first proof for this phenomenon in deep linear networks parametrized with \mu P. Specifically, we consider a linear Multi-Layer Perceptron (MLP) and show that at training steps t , the optimal learning rate converges to a non-zero constant as width goes to infinity, providing a theoretical proof for learning rate transfer observed in practice. Our proof is based on the observation that with linear MLPs, the loss function at any training step can be expressed as a polynomial function of the learning rate. We study convergence dynamics of these polynomials and their roots and conclude on the convergence of the optimal learning rate as width goes to infinity. We further show that other parametrizations such as Standard Parametrization (SP) (and Neural Tangent Parametrization (NTP)) lead to significant shift in optimal learning rate as width grows, thus requiring expensive tuning. The paper is structured as follows. In Section 2 , we introduce notation and definitions. In Section 3 , we provide a full characterization of LR transfer after one step and study the convergence rate of the optimal LR. In Section 4 , we provide a proof for LR transfer for general step t . In both Section 3 and Section 4 , extensive simulations are provided to support the theoretical results. In Section 5 , we provide additional empirical results with varying setups: activation function, optimizer, depth, training time. 1.1 Related work Infinite-width. There is a rich literature on the theory of infinite-width neural networks. The first works on infinite-width theory are related to approximation results showing that neural networks are universal approximators when the width to infinity (see e.g. [ 20 , 10 ] ). Perhaps the first methodological work on infinite-width neural networks was a study of priors in large-width Bayesian neural network by Neal 1996 , where the author studied how Gaussian prior should be scaled as network width increases, and showed that single-layer Bayesian networks converge to a Gaussian process in the infinite-width limit, a result that was later used in [ 36 ] to compute infinite-width posteriors, and was later generalized to multi-layer networks in [ 25 , 12 ] . Subsequent research has examined the impact of initialization [ 33 , 16 , 26 , 11 ] , the activation functions [ 16 ] , learning rate [ 38 ] , batch size [ 40 ] , etc. Others works studied how these HPs should scale with depth (assuming large-width) [ 17 , 39 , 5 ] . There is also a rich literature on training dynamics of infinite-width neural networks, including the literature on the neural tangent kernel [ 21 , 18 , 3 , 6 , 2 ] , and the literature on mean-field neural networks [ 34 , 28 , 29 , 8 ] . Hyperparameter transfer. Yang and Hu 2021 introduced \mu P, a neural network parametrization that specifies how initialization and learning rate should scale with model width n . The authors derived this parametrization by searching for HPs that yield feature learning in the infinite-width limit, in contrast to neural tangent parametrization which leads to a kernel regime in the limit [ 21 ] . In particular, the authors observed that \mu P leads to an interesting phenomenon: HP transfer with width, where optimal HPs tend to stabilize as width increases. It was conjectured that feature learning properties of the infinite-width limit under \mu P is the main factor behind HP transfer. In [ 38 ] , the authors showed that \mu P yields HP transfer in Large Language Models (LLMs) of GPT-3 scale. However,

Read the complete text