Saddle-to-saddle dynamics in deep ReLU networks: Low-rank bias in the first saddle escape

Ioannis Bantzis, James B. Simon, Arthur Jacot · 2025

Read Saddle-to-saddle dynamics in deep ReLU networks: Low-rank bias in the first saddle escape by Ioannis Bantzis, James B. Simon, Arthur Jacot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21722.txt.

Introduction

Ioannis Bantzis, James Simon, and Arthur Jacot wrote this machine learning theory paper, published in two thousand twenty-five, on how deep networks first move during training. A network with tiny starting weights sits at a saddle point, a flat spot it must escape before real learning begins.

The authors study the escape directions, the paths along which gradient descent first leaves that saddle. The best escape path has a low-rank shape in its deeper layers: one dominant direction, far larger than the rest. The gap grows with depth in a precise way they prove. Related results pin down more of the escape geometry. The team links this to a broader picture where training hops from saddle to saddle, each stop raising the network's effective rank. The work helps explain why deep nets build their skills in stages.

Catalog record

CREATORIoannis Bantzis, James B. Simon, Arthur Jacot
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Saddle-to-saddle dynamics in deep ReLU networks: Low-rank bias in the first saddle escape Ioannis Bantzis, James B. Simon, Arthur Jacot Abstract When a deep ReLU network is initialized with small weights, gradient descent (GD) is at first dominated by the saddle at the origin in parameter space. We study the so-called escape directions along which GD leaves the origin, which play a similar role as the eigenvectors of the Hessian for strict saddles. We show that the optimal escape direction features a low-rank bias in its deeper layers: the first singular value of the ℓ \ell -th layer weight matrix is at least ℓ 1 4 \ell^{\frac{1}{4}} larger than any other singular value. We also prove a number of related results about these escape directions. We suggest that deep ReLU networks exhibit saddle-to-saddle dynamics, with GD visiting a sequence of saddles with increasing bottleneck rank ( Jacot, 2023b ) . 1 Introduction In spite of the groundbreaking success of Deep Neural Networks (DNNs) , the training dynamics of GD in these models remain ill-understood, especially when the number of hidden layers is large. A significant step in our understanding is the (relatively recent) realization that there exist multiple regimes of training in large neural networks: a kernel or lazy regime, where DNNs simply implement kernel methods w.r.t. the Neural Tangent Kernel (NTK) ( Jacot et al., 2018 ; Du et al., 2019 ; Allen-Zhu et al., 2019 ) , and an active (or rich) regime characterized by the presence of feature learning ( Chizat and Bach, 2018b ; Rotskoff and Vanden-Eijnden, 2018 ; Chizat and Bach, 2018a ) and some form of sparsity such as a low-rank bias ( Li et al., 2020 ; Gunasekar et al., 2017 ; Arora et al., 2019a ) . The kernel regime is significantly simpler than the active one because the dynamics can be linearized around the initialization ( Jacot et al., 2018 ; Lee et al., 2019 ) , and the loss is approximately quadratic/convex in the region traversed by GD ( Jacot et al., 2020 ) (it also satisfies the PL inequality ( Liu et al., 2020 ) ). This makes it possible to prove strong convergence guarantees ( Du et al., 2019 ; Allen-Zhu et al., 2019 ) and apply generalization bounds from the kernel methods literature almost directly ( Arora et al., 2019b ; Bordelon et al., 2020 ) . Our understanding of the kernel regime is essentially complete, but some functions are unlearnable in the kernel regime yet learnable in the active regime ( Bach, 2017 ; Ghorbani et al., 2020 ) . There are arguably many active regimes corresponding to different ways to leave the kernel regime, including small weight initialization ( Woodworth et al., 2020 ) , large learning rate ( Lewkowycz et al., 2020 ; Damian et al., 2022 ) , large noise in training ( Smith et al., 2021 ; Pesme et al., 2021 ; Vivien et al., 2022 ; Wang and Jacot, 2024 ) , late training with the cross-entropy loss ( Ji and Telgarsky, 2018 ; Chizat and Bach, 2020 ) , and weight decay ( E et al., 2019 ; Ongie et al., 2020 ; Jacot, 2023b ) . We will focus on the effect of initialization scale, where a phase change from kernel regime to active regime occurs as the variance of the initial weights decays towards zero. Here again we can distinguish two active regimes ( Luo et al., 2021 ) : the mean-field regime which lies right at transition between regimes ( Chizat and Bach, 2018b ; Rotskoff and Vanden-Eijnden, 2018 ; Mei et al., 2018 ) , and the saddle-to-saddle regime ( Saxe et al., 2014 ; Jacot et al., 2022 ; Pesme and Flammarion, 2023 ; Boursier et al., 2022 ) for even smaller initialization. The mean-field limit was first described for shallow networks ( Chizat and Bach, 2018b ; Rotskoff and Vanden-Eijnden, 2018 ; Mei et al., 2018 ) , and has more recently been extended to the deep case ( Araújo et al., 2019 ; Bordelon and Pehlevan, 2022 ) . A limitation of these approaches is that the limiting dynamics remain complex, especially in the deep case where they are described by algorithms that are not only very costly in the worst case ( Bordelon and Pehlevan, 2022 ; Yang and Hu, 2020 ) , but also difficult to interpret and reason about. This high complexity could be explained by the fact that the mean-field limit is critical, i.e. it lies exactly at the transition between kernel and active, and therefore it must capture the complexity of both of those regimes, as well as of the whole spectrum of intermediate dynamics. 1.1 Saddle-to-Saddle dynamics This motivates the study of the saddle-to-saddle regime for even smaller initializations. As the name suggests, this regime is characterized by GD visiting a number of saddles before reaching a global minimizer. Roughly speaking, because of the small initializations, GD starts in the vicinity of the saddle which lies at the origin in parameter space and remains stuck there for a number of steps until it finds an escape direction, leading to a sudden drop in the loss. This escape direction exhibits a form of approximate sparsity (amongst other properties) that is preserved by GD. At this point, the level of sparsity can either be enough to fit the training data in which case the loss will drop to zero and training will stop, but if the network is ‘too sparse’ to fit the data, GD will approach another saddle at a lower cost (which is locally optimal given the sparsity) before escaping along a less sparse escape direction. GD can visit a sequence of saddles before reaching a final network that fits the data while being as sparse as possible. This has been described as performing a greedy algorithm ( Li et al., 2020 ) where one tries to find the best data-fit with a sparsity constraint that is gradually weakened until the training data can be fitted. Such incremental learning dynamics were first observed in diagonal linear networks ( Saxe et al., 2014 ; Saxe et al., 2019 ; Gidel et al., 2019 ) (and by extension to linear Convolutional Neural Networks , which are diagonal nets in Fourier space), before being extended to linear fully-connected networks ( Arora et al., 2019a ; Li et al., 2020 ; Jacot et al., 2022 ; Tu et al., 2024 ; Kunin et al., 2024 ) . These result in coordinate sparsity of the learned vector for diagonal networks and rank sparsity of the learned matrix for fully-connected networks. For nonlinear networks, the focus has been mainly on shallow networks, where a condensation effect is observed, wherein groups of neurons end up having the same activations (up to scaling). Roughly speaking, in the first escape direction, a first group of hidden neurons comes out first, all with the same activation (up to scaling), in the subsequent saddles, new groups can emerge or an existing group can split in two ( Chizat and Bach, 2018b ) (though sometimes they may fail to split leading to problems ( Boursier and Flammarion, 2024 )

Read the complete text