Intrinsic training dynamics of deep neural networks
Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval · 2025
Read Intrinsic training dynamics of deep neural networks by Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07370.txt.
Introduction
Sibylle Marcotte, Gabriel Peyré, and Rémi Gribonval write, in twenty twenty-five, a theory paper on how deep neural networks learn. A core puzzle is why training by gradual descent often lands on simple structures, such as sparse or low-rank ones. The authors study a stepping stone: when does the descent path of a network's weights imply a clean path for a derived quantity built from those weights? They frame a property for that implied path and tie it to conservation laws, rules that keep some quantity fixed.
They give a simple test based on the overlap of null spaces of linear maps. Applied to common networks, the test shows the property holds for most starts when the derived quantity is a path-lifting. For linear networks, they widen the set of good starts.
Catalog record
| CREATOR | Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07370
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Intrinsic training dynamics of deep neural networks Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval Abstract A fundamental challenge in the theory of deep learning is to understand whether gradient-based training in high-dimensional parameter spaces can be captured by simpler, lower-dimensional structures, leading to so-called implicit bias. As a stepping stone, we study when a gradient flow on a high-dimensional variable θ \theta implies an intrinsic gradient flow on a lower-dimensional variable z = ϕ ( θ ) z=\phi(\theta) , for an architecture-related function ϕ \phi . We express a so-called intrinsic dynamic property and show how it is related to the study of conservation laws associated with the factorization ϕ \phi . This leads to a simple criterion based on the inclusion of kernels of linear maps which yields a necessary condition for this property to hold. We then apply our theory to general ReLU networks of arbitrary depth and show that, for any initialization, it is possible to rewrite the flow as an intrinsic dynamic in lower dimension that depends only on z z and the initialization, when ϕ \phi is the so-called path-lifting. In the case of linear networks with ϕ \phi the product of weight matrices, so-called balanced initializations are also known to enable such a dimensionality reduction; we generalize this result to a broader class of relaxed balanced initializations, showing that, in certain configurations, these are the only initializations that ensure the intrinsic dynamic property. Finally, for the linear neural ODE associated to the limit of infinitely deep linear networks, with relaxed balanced initialization, we explicitly express the corresponding intrinsic dynamics. 1 Introduction A central question in deep learning theory is how the complexity of high-dimensional gradient-based training can give rise to simpler, lower-dimensional dynamics. In this work, we explore when the gradient flow on high-dimensional parameters \theta naturally induces a gradient flow on a lower-dimensional variable z=\phi(\theta) , where \phi captures structural aspects of the model. Intrinsic lower dimensional flow The study of high-dimensional optimization flows arising in the training of neural networks often benefits from the identification of lower-dimensional intrinsic dynamics. Specifically, it is of considerable interest to rewrite a high-dimensional parameter flow \theta(t) in terms of an lower-dimensional representation z(t)=\phi(\theta(t)) , using a suitable architecture-related reparametrization \phi . When the reduced flow on z(t) admits a mirror flow representation, one gains access to powerful analytical tools rooted in convex optimization theory, allowing explicit characterization of the implicit bias induced by gradient-based optimization algorithms. In particular, prior research has successfully leveraged mirror flow formulations to rigorously demonstrate implicit regularization effects, such as sparsity in scalar linear neural networks and two-layer networks with a single neuron ( Gunasekar et al. 2018 ), as well as maximum-margin classification for logistic regression problems in separable data scenarios ( Soudry et al. 2018 ; Chizat & Bach 2020 ). Recent work by Li et al. 2022 identifies sufficient conditions under which the reparametrization \phi makes the flow a mirror flow, requiring that the parametrization be commuting. However, this commuting condition is rarely satisfied in practical scenarios. A weaker condition, introduced by Marcotte et al. 2023 , demands merely that the parametrization be involutive, thus ensuring that the flow on z(t) can still be expressed as a Riemannian gradient flow (but not necessarly a mirror flow anymore) dependent solely on z and the initial parameters \theta(0) . Marcotte et al. 2023 have shown that this weaker condition applies specifically to the parametrization used in two-layer ReLU networks ( Stock & Gribonval 2022 ). Conservation laws The functions conserved during the dynamics play a crucial role in rewriting flows in lower-dimensional representations. Indeed, when a trajectory \theta(t) is constrained to remain within level sets \{\theta:\mathbf{h}(\theta)=\mathbf{h}(\theta_{0})\} where \mathbf{h} is a (vector-valued) conserved function, the dynamics are effectively restricted to a manifold of lower dimension. A particularly important class of conserved functions along these trajectories is given by the conservation laws associated with a certain architecture-dependent parametrization \phi , a concept introduced in Marcotte et al. 2023 . These laws depend exclusively on \phi , and notably, in the context of neural network training dynamics, they represent quantities preserved across trajectories irrespective of the initial conditions or the training data-set. In the specific case of linear and ReLU neural networks, these conservation laws correspond exactly to previously known “canonical” conserved functions identified in Du et al. 2018 , as demonstrated by Marcotte et al. 2023 . Furthermore, Marcotte et al. 2023 establish that if the parametrization \phi is involutive, there exist sufficiently many scalar conservation laws to fully rewrite the original trajectory \theta(t) in terms of \phi(\theta(t)) and the initial conditions alone. In the linear network case, when so-called balanced conditions (a notion introduced in Arora et al. 2019 ) are satisfied (i.e., when the initialization sets all canonical conservation laws ( Chitour et al. 2018 ) to zero), it becomes possible to rewrite the flow in terms of z=\phi(\theta) , where \phi corresponds to the product of weight matrices, as a Riemannian metric that only depend on z and \mathbf{h}(\theta_{0})=0 ( Arora et al. 2018 ; Bah et al. 2022 ). Moreover, Achour et al. 2025 extended this result to linear convolutional networks in the case of a mean squared loss, but this time for arbitrary initializations, with the Riemannian metric depending on the initialization. For linear networks and in the particular case when the loss function is the square loss, Bah et al. 2022 show that the trajectory evolves on the manifold of matrices having some fixed rank under balanced condition. Still in the square-loss setting, and in the case of two-layer linear networks, Tarmoun et al. 2021 ; Braun et al. 2022 ; Dominé et al. 2025 exploit the conservation laws to obtain an exact closed-form expression for z(t) under specific configurations, whereas Varre et al. 2023 uses the same laws to analyse an implicit bias of this dynamic. Our main contributions. We first define the notion of intrinsic dynamic property ( Definition 2.6 ), then the notion of intrinsic metric property ( Definition 2.10 ) and finally the one of intrinsic recoverability property ( Definition 3.1 ), and we show the implications ( Lemma 2.11 and Lemma 3.2 ): \text{Intrinsic Recoverability}\implies\text{Intrinsic Metric}\implies\text{Intrinsic Dynamic}. We then provide a simple criterion that characterizes the intrinsic recoverability property ( Theorem 3.3 ), and show ( Proposition 3.7 ) that this criterion is quasi equivalent to the Frobenius property ( Definition 3.6 ). We prove that the reparametrization for general ReLU networks of arbitrary depth satisfies this property ( Theorem 3.8 ), establishing that any initialization of a general ReLU network satisfies the intrinsic recoverability property ( Corollary 3.9 ), as illustrated by a characterization of the intrinsic dynamic of a 3-layer neural network ( Proposition 3.10 ). Next, by establishing a necessary condition for the intrinsic metric property to hold based on the study of kernels of linear mappings Theorem 2.14 , we show that the intrinsic metric property fails to hold for the natural reparametrizations corresponding of 2-layer linear networks (resp. of attention layers) , unless the initialization satisfies the relaxed balanced condition introduced in Definition 4.1 ( Theorem 4.4 ). We then show that relaxed balanced initializations do satisfy