Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario
Florentin Coeurdoux, Grégoire Ferré, Jean-Philippe Bouchaud, [0.7em] Capital Fund Management, Paris, France · 2026
Read Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario by Florentin Coeurdoux, Grégoire Ferré, Jean-Philippe Bouchaud, [0.7em] Capital Fund Management, Paris, France (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18450.txt.
Introduction
Coeurdoux, Ferré, and Bouchaud model early stopping in gradient flow with random matrix theory. Trained models often show a window where signal is detectable before overfitting dominates. In a linear teacher-student model, learning happens as a lone eigenvalue pulls away from the noisy bulk. Anisotropy in input covariance creates fast and slow learning directions, yielding a transient Baik-Ben Arous-Péché transition: the teacher spike may never emerge, persist, or flash briefly before reabsorption. Phase diagrams map these regimes, validated against finite-size simulations.
Catalog record
| CREATOR | Florentin Coeurdoux, Grégoire Ferré, Jean-Philippe Bouchaud, [0.7em] Capital Fund Management, Paris, France |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42796 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18450
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario Florentin Coeurdoux, Grégoire Ferré, Jean-Philippe Bouchaud, [0.7em] Capital Fund Management, Paris, France Abstract Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher–student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a 2 × 2 2\times 2 Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik–Ben Arous–Péché (BBP) transition [ 6 ] : depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise. 1 Introduction Understanding when gradient-based learning extracts signal from data, and when it instead amplifies noise and overfits, is a central problem in modern learning theory. A recurring empirical observation is that the spectrum of trained weight matrices often passes through a transient regime in which an isolated eigenvalue separates from a noisy bulk before the spectrum becomes more complicated at later stages of training [ 13 , 18 , 17 , 20 ] . Although this phenomenon is frequently associated with the practical success of early stopping, its underlying mechanism remains poorly understood. In this paper we isolate a minimal, analytically tractable mechanism for such a transient spectral separation. We study gradient flow in a linear teacher–student model and show that anisotropy in the input covariance can produce a transient BBP transition : an outlier may emerge from the bulk at intermediate times (with a non-zero overlap with the teacher direction) and later be reabsorbed as training continues, in which case the teacher direction cannot be reconstructed [ 6 ] . The effect is driven by the coexistence of fast and slow directions in the data covariance. Fast directions learn the teacher early, whereas slow directions continue to inject noise and can eventually hide the signal again. Our model is deliberately simple. It is not intended to reproduce the full heavy-tailed phenomenology observed in deep nonlinear networks at convergence; rather, it is designed to capture and explain the bulk-plus-spike regime in a setting where the full dynamics can be analyzed explicitly. The key solvable case is a two-block covariance model in which the singular values of the input matrix take only two values. In that simplified yet realistic setting, the bulk spectrum of the symmetrized weight matrix is described by a 2\times 2 Dyson equation, while the rank-one teacher induces a finite-rank perturbation whose outlier condition can be reduced to a rank-two determinant. Three regimes emerge from our model, depending on signal strength and input covariance anisotropy: (i) a weak-signal regime in which no outlier ever emerges, so the teacher is never spectrally detectable; (ii) a strong-signal regime in which an outlier emerges and remains separated from the bulk at all times; and (iii) an “early-stopping” regime in which an outlier exists only during a finite time window, providing a precise random-matrix formulation of early stopping. Contributions In the solvable setting above, we (a) derive a 2\times 2 Dyson equation characterizing the full time-dependent bulk spectrum of the symmetrized weight matrix; (b) establish an explicit rank-two BBP-type outlier condition valid at every finite time t ; (c) classify the possible learning regimes and give closed-form phase diagrams in the (\theta,t) and (\theta,\lambda_{-}) planes; and (d) identify numerically an optimal early-stopping time, defined through the teacher–eigenvector overlap, that lies strictly inside the transient BBP window. Related works The BBP phase transition was introduced by Baik, Ben Arous and Péché [ 6 ] and has since become a standard tool in spiked random matrix models and finite-rank perturbation theory [ 8 ] . Exact learning dynamics in linear and deep-linear models have been studied extensively, including solvable analyses of gradient-flow learning and early stopping [ 16 , 1 , 4 ] . The connection between spectral properties of weight matrices and generalisation has been explored empirically by Martin and Mahoney [ 13 ] , Thamm et al. [ 18 ] , Staats et al. [ 17 ] , Yunis et al. [ 20 ] , Olsen et al. [ 14 ] , and others. An inverse BBP transition has been observed in the dynamical Hessian during gradient descent for the problem of phase retrieval [ 9 ] . More broadly, random matrix methods have become increasingly useful in the study of high-dimensional learning dynamics and implicit regularisation [ 7 , 2 , 3 , 12 , 5 , 10 ] . Our contribution is to provide an arguably oversimplified but explicit random-matrix mechanism for a transient spectral spike during learning, i.e. a BBP transition followed by an inverse BBP transition, and to tie this transient spectral geometry to a computable optimal stopping time. 2 Model and Setup 2.1 Gradient flow on a linear model We consider inputs X\in\mathbb{R}^{N\times M} and targets Y\in\mathbb{R}^{N\times M} in the high-dimensional regime N,M\to\infty,\qquad\frac{M}{N}\to\gamma\geq 1. We train a linear model with weight matrix A\in\mathbb{R}^{N\times N} by minimizing A_{\mathrm{LS}}=\underset{A}{\arg\min}\,J(A),\qquad J(A)=\frac{1}{2}\|AX-Y\|_{F}^{2}. (1) Although the minimizer is explicit and reads A_{\mathrm{LS}}\equiv YX^{\top}(XX^{\top})^{-1} , our interest is not the optimizer itself but the spectral evolution of the gradient-flow trajectory \frac{dA_{t}}{dt}=-\nabla J(A_{t}),\qquad A_{0}=A_{\mathrm{init}}. Since \nabla J(A)=(AX-Y)X^{\top}=AXX^{\top}-YX^{\top}, the flow solves the linear matrix ODE \dot{A}_{t}=-A_{t}XX^{\top}+YX^{\top}. Its explicit solution is A_{t}=A_{\mathrm{init}}e^{-tXX^{\top}}+YX^{\top}(XX^{\top})^{-1}\bigl(I-e^{-tXX^{\top}}\bigr). (2) This is a standard semigroup evolution, and in the present least-squares setting it provides a natural continuous-time framework for early stopping [ 4 ] . At long times it converges to the least-squares estimator A_{\infty}\equiv A_{\mathrm{LS}} . The key point for us is that the learning speed is anisotropic: the eigenvalues of XX^{\top} determine how fast the corresponding directions are learned or damped. This observation motivates the two-block covariance model introduced below. 2.2 Random matrix assumptions To model a teacher–student setting, we assume that the targets are generated by a rank-one teacher Y=A_{\mathrm{teach}}^{\theta}X+Z,\qquad A_{\mathrm{teach}}^{\theta}=\theta\,vv^{\top}, (3) where v\in\mathbb{R}^{N} is a fixed unit vector and \theta\geq 0 is the signal amplitude. The noise matrix Z\in\mathbb{R}^{N\times M} has i.i.d. \mathcal{N}(0,1/M) entries, independent of everything else. We take A_{\mathrm{init}} to be a GOE matrix at the usual 1/\sqrt{N} scale, independent of X and Z . Substituting ( 3 ) into ( 2 ) yields the decomposition A_{t}^{\theta}=\underbrace{A_{\mathrm{init}}e^{-tXX^{\top}}+ZX^{\top}(XX^{\top})^{-1}\bigl(I-e^{-tXX^{\top}}\bigr)}_{\text{noise part}}+\underbrace{\theta\,vv^{\top}\bigl(I-e^{-tXX^{\top}}\bigr)}_{\text{teacher part}}. (4) We consider the additive symmetrization S_{t}^{\theta}:=A_{t}^{\theta}+(A_{t}^{\theta})^{\top}, because it is the simplest construction that preserves the bulk-plus-spike structure of the problem while remaining fully solvable at finite time: the noise contribution becomes a Wigner-type matrix with a piecewise-constant variance profile, and the teacher contribution becomes a finite-rank symmetric perturbation. The more standard