Optimization Insights into Deep Diagonal Linear Networks
Hippolyte Labarrière, Cesare Molinari, Lorenzo Rosasco, Cristian Vega, Silvia Villa · 2024
Read Optimization Insights into Deep Diagonal Linear Networks by Hippolyte Labarrière, Cesare Molinari, Lorenzo Rosasco, Cristian Vega, Silvia Villa (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16765.txt.
Introduction
Hippolyte Labarrière and co-authors dissect training in deep diagonal linear networks in a 2024 paper. Gradient methods train bloated models well in practice, though their loss fields are far from convex. Diagonal linear nets offer a clean lab: layer wise products keep the net effect convex while bending the path taken.
Under mild starts, gradient flow on the layers mirrors a mirror flow on the net effect. That mirror view yields sharp rates, with loss falling at exponential speed under a set slope trait. Start scale and depth set how fast training runs. The plain case shows how odd shapes can still train kindly.
Catalog record
| CREATOR | Hippolyte Labarrière, Cesare Molinari, Lorenzo Rosasco, Cristian Vega, Silvia Villa |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16765
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimization Insights into Deep Diagonal Linear Networks Hippolyte Labarrière, Cesare Molinari, Lorenzo Rosasco, Cristian Vega, Silvia Villa Abstract Gradient-based methods successfully train highly overparameterized models in practice, even though the associated optimization problems are markedly nonconvex. Understanding the mechanisms that make such methods effective has become a central problem in modern optimization. To investigate this question in a tractable setting, we study Deep Diagonal Linear Networks. These are multilayer architectures with a reparameterization that preserves convexity in the effective parameter, while inducing a nontrivial geometry in the optimization landscape. Under mild initialization conditions, we show that gradient flow on the layer parameters induces a mirror-flow dynamic in the effective parameter space. This structural insight yields explicit convergence guarantees, including exponential decay of the loss under a Polyak–Łojasiewicz condition, and clarifies how the parametrization and initialization scale govern the training speed. Overall, our results demonstrate that deep diagonal over parameterizations, despite their apparent complexity, can endow standard gradient methods with well-behaved and interpretable optimization dynamics. 1 Introduction In recent years, the application of deep neural networks has transformed the field of machine learning, achieving outstanding results in tasks involving complex data such as images and natural language. These models are typically trained successfully using gradient-based methods, most notably stochastic gradient descent, consistently reaching zero empirical loss across various benchmarks. This empirical success is particularly striking given the highly nonconvex nature of the underlying optimization problems, where classical theory would provide little/no guarantees. Understanding why such simple optimization procedures are so effective in training overparameterized models has therefore become a central research question. A common approach to understanding optimization in overparameterized models is to analyze simplified settings where the interaction between parameterization and optimization becomes more transparent. Even in these reduced models, the dynamics often remain surprisingly intricate, yet they can shed light on how parameterization shapes convergence behavior. Within this line of work, Diagonal Linear Networks (DLNs), introduced independently by [ 37 ] and [ 26 ] , have emerged as a particularly tractable framework. In shallow DLNs, the network function takes the form f_{\theta}(x)=\theta^{\top}x with parameters \theta=u\odot v , where \odot denotes the Hadamard product (see Figure 1 ). Despite their simplicity, DLNs display a variety of nontrivial optimization phenomena. Prior studies have examined how initialization scale, step size, or stochasticity influence convergence and the nature of the obtained solution [ 7 , 30 , 15 , 27 , 29 ] . These works highlight, among other insights, that mild overparameterization can already smooth the loss landscape, and that small versus large initializations give rise to qualitatively different training behaviors, mirroring the so called rich and lazy regimes of [ 9 ] . x^{i}_{1} x^{i}_{2} x^{i}_{3} x^{i}_{4} u_{1} u_{2} u_{3} u_{4} v_{1} v_{2} v_{3} v_{4} f_{\theta}(x^{i}) Figure 1: Representation of a Diagonal Linear Network In this paper, we extend this line of inquiry to Deep Diagonal Linear Networks, which generalize DLNs to an L -layer parametrization. In this model, the effective parameter is given by \theta=u^{1}\odot u^{2}\odot\dots\odot u^{L}, where each u^{j}\in\mathbb{R}^{d} . This formulation offers a closer abstraction of multilayer architectures while preserving analytical tractability, and it allows us to explore how depth influences the optimization landscape and the implicit dynamics induced by training. Note that a simpler version of the L -layers case has been studied in [ 37 , 26 , 10 ] considering the simpler parameterization \theta=u^{\odot L}-v^{\odot L} . Our goal is to understand the effect of this reparameterization when the layers weights (u^{j})_{j\in[L]} are trained using gradient flow. More precisely, we consider the problem \min_{\theta\in\mathbb{R}^{d}}\mathcal{L}(\theta), for some differentiable loss function \mathcal{L}:\mathbb{R}^{d}\rightarrow\mathbb{R} , and for some input-output data X:\mathbb{R}^{d}\rightarrow\mathbb{R}^{n} and y\in\mathbb{R}^{n} . We optimize \mathcal{L} by applying gradient flow with respect to the variables (u^{j})_{j\in[L]} . Our main contributions are summarized as follows: • We provide a mild initialization condition under which gradient flow on (u^{j})_{j\in[L]} induces a trajectory in \theta that follows a mirror-flow dynamic. • Under the same condition, we establish several convergence guarantees for the original optimization problem. In particular, when the loss function \mathcal{L} satisfies a Polyak-Łojasiewicz condition, the objective function in the \theta -space decreases exponentially toward its minimum. Our analysis also indicates that very small initializations lead to slow training, consistent with phenomena observed in related models. • We identify quantities that are conserved along the gradient-flow trajectory, in line with recent works on conservative laws [ 22 , 24 ] . In particular, this reveals that the dynamics of \theta are governed by those of an abstract layer associated with minimal components. We provide a detailed review of related topics in the literature in the next section. 1.1 Related works Conservation laws A common approach to studying the behavior of optimization dynamics in neural network models is to identify the conserved quantities that remain invariant during training. Such quantities often provide insight into convergence properties or implicit bias phenomena (see, e.g., [ 33 , 14 ] ). Characterizing these conservation laws is therefore crucial for understanding the overall training dynamics. Recent works have pursued this direction by establishing convergence laws for gradient flow in general neural network models [ 22 ] , momentum-based dynamics [ 23 ] , and discrete-time dynamics [ 24 ] . Hadamard parameterization to promote sparsity Diagonal Linear Networks are closely related to Hadamard parameterization (HP) and it is worth noticing that HP was used before for sparsity recovery. [ 19 ] takes benefit of HP to promote sparsity in a LASSO problem. [ 35 , 40 ] also consider an HP for a least squares problem inducing an implicit bias towards the minimal L^{1} -norm solution, enabling the use of early stopping strategies. In a comparable vein, [ 4 ] exploit a reparameterization \theta=u^{\odot 2} for a classification problem and [ 12 ] apply HP to solve non negative least squares. [ 31 ] elegantly reformulate a group-LASSO problem using some Hadamard parameterization in order to apply more efficient optimization methods. The geometry induced by HP is also a subject of interest and in this regard, [ 28 ] give necessary condition for a reparameterized function to satisfy the Kurdyka-Łojasiewicz property. Links with Mirror Flow As shown in [ 7 , 10 ] , Diagonal Linear Networks trained with a Gradient Flow (or Gradient Descent) share direct links with Mirror Flow (or Mirror Descent). One of the first appearances of the Mirror Flow dynamic can be attributed to [ 3 ] . Many years later, a number of studies are presented in the context of understanding the effects of reparameterization. The mirror gradient method having an implicit bias towards specific solutions, [ 36 ] propose early stopping strategies taking advantage of it. [ 21 ] provide assumptions that are necessary for a reparameterization trained with a Gradient Flow to be equivalent to a Mirror Flow. In their paper, [ 11 ] adopt an original point of view, seeing reparameterization as a way to enforce some implicit bias related to the corresponding mirror map. In this perspective, the authors give guidelines to define it efficiently. Matrix factorization A significant step toward understanding the optimization dynamics of overparameterized models is provided by [ 18 ] , who analyze gradient-based training in a simple least squares setting,