Layerwise LQR for Geometry-Aware Optimization of Deep Networks
Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu, Simon Lacoste-Julien, Aristide Baratin · 2026
Read Layerwise LQR for Geometry-Aware Optimization of Deep Networks by Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu, Simon Lacoste-Julien, Aristide Baratin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04230.txt.
Introduction
Simon Dufort-Labbé and four co-authors propose a new way to train deep networks that respects the geometry of the whole model, not just one layer at a time. Geometry-aware methods can train better, but fast versions usually cut the links between layers early to save computing cost.
Their approach recasts the ideal training step as a control problem across layers, then learns a structured stand-in that is cheap to reuse. The resulting optimizer wraps standard training methods while keeping a principled link to the full geometry. It aims for better conditioning without paying the full cost at every step.
Catalog record
| CREATOR | Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu, Simon Lacoste-Julien, Aristide Baratin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04230
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layerwise LQR for Geometry-Aware Optimization of Deep Networks Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu, Simon Lacoste-Julien, Aristide Baratin Abstract Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners usually impose structural approximations early, often discarding cross-layer interactions induced by the network computation. We introduce Layerwise LQR (LLQR), a framework for learning structured inverse preconditioners under a global layerwise optimal-control objective. The starting point is an exact equivalence: the steepest-descent step under a broad class of divergence-induced quadratic models—including Newton, Gauss–Newton, Fisher/natural-gradient, and intermediate-layer metrics—can be written as a finite-horizon Linear Quadratic Regulator (LQR) problem. This formulation serves as a reference that exposes the layerwise dynamics and cost matrices encoding the original dense geometry. We then derive a scalable relaxation that learns diagonal, (E-)Kronecker-factored, or other structured inverse preconditioners by minimizing the LQR objective and reusing them across iterations. The resulting optimizer wraps standard methods while retaining a principled connection to second-order geometry, without forming or inverting the global curvature matrix. Experiments on ResNets and Transformers show that LLQR improves optimization dynamics and often translates these gains into improved final test performance, while adding only modest wall-clock overhead. It establishes LLQR as a practical framework for geometry-aware second-order methods and a reference for evaluating scalable approximations. Implementation is available at: github.com/SimonDufLab/LLQR Figure 1 : ImageNet and IWSLT14 training curves under NGD induced divergence. Left : ResNet-50 trained for 100 epochs on ImageNet with SGDM, comparing the baseline, LLQR+KFAC, and LLQR+E-KFAC. Right : a Fairseq Transformer trained on IWSLT14 De–En with AdamW, comparing the baseline and LLQR+E-KFAC. 1 Introduction Many successful optimization algorithms in deep learning can be understood as steepest-descent methods under a chosen geometry. For a differentiable loss L(\bm{\theta}) with gradient {\bm{g}}=\nabla L(\bm{\theta}) , the gradient descent step is the solution of the variational problem \arg\min_{\Delta\bm{\theta}}\left[{\bm{g}}^{\top}\Delta\bm{\theta}+\frac{1}{2\eta}\|\Delta\bm{\theta}\|_{2}^{2}\right]=-\eta{\bm{g}} corresponding to steepest descent in the Euclidean norm ( 9 ) . Newton’s method uses the local loss Hessian; natural gradient uses the Fisher information matrix; and practical methods such as Adam, Shampoo, Muon, K-FAC, and related algorithms can be interpreted as using structured norms or preconditioners that capture partial information about the local geometry ( 7 ) . This viewpoint is increasingly important: the choice of geometry does not merely affect conditioning and convergence speed, but also changes the implicit bias of the training trajectory and can therefore influence the solutions found by learning 39 . The classical difficulty is that the most principled geometries are dense and expensive. A generic second-order or geometry-aware step minimizes a local quadratic model m(\Delta\bm{\theta})={\bm{g}}^{\top}\Delta\bm{\theta}+\frac{1}{2}\Delta\bm{\theta}^{\top}{\bm{H}}\Delta\bm{\theta} (1) where {\bm{H}} may be the regularized Hessian, the Gauss–Newton matrix, the Fisher matrix, or another divergence-induced metric. For modern neural networks, {\bm{H}} is a dense matrix coupling parameters across layers through the chain rule. Directly solving {\bm{H}}\Delta\bm{\theta}=-{\bm{g}} is infeasible at scale. Existing scalable methods therefore introduce structure early: for instance, K-FAC ( 28 ) approximates the Fisher by block-diagonal Kronecker-factored blocks. This makes preconditioning tractable, but it also removes inter-layer couplings before the optimization problem defining the update has been solved. This paper takes a different route. Rather than imposing structure directly on the curvature model, we first rewrite the dense geometry-aware update as a layer-coupled optimal-control problem. Building on the classical link between Newton steps and Linear Quadratic Regulator (LQR) ( 15 ) , we show that under a broad class of divergence-induced quadratic models, the dense global quadratic form ( 1 ) admits an exact layerwise factorization: the forward pass defines linear perturbation dynamics, and the chosen divergence defines the LQR costs. The resulting problem retains the cross-layer dependencies of the original dense quadratic model, but expresses them through layerwise dynamics and costs rather than through an explicitly formed global curvature matrix. The exact LQR formulation can be solved by Riccati recursions and recovers the corresponding exact geometry-aware update. However, for large neural networks, it still requires manipulating large Jacobian-dependent quantities and solving expensive matrix recursions. Our main contribution is therefore a scalable relaxation of this LQR problem. We parameterize the update as a preconditioned gradient, \Delta\bm{\theta}_{i}\;=\;{\bm{U}}_{i}\,\nabla_{\bm{\theta}_{i}}L(\bm{\theta}^{k}), (2) where {\bm{U}}=\mathrm{diag}({\bm{U}}_{0},\dots,{\bm{U}}_{N-1}) is a structured inverse preconditioner, for example diagonal or Kronecker-factored. Crucially, this block structure is imposed on the learned inverse preconditioner, not on the geometry before deriving the update objective. This distinction is important. A block-diagonal learned preconditioner does not represent the full dense inverse curvature matrix, and our relaxation is not exact. But unlike methods that begin by replacing the curvature matrix with a block-diagonal surrogate, our approach first derives a layer-coupled objective equivalent to the dense geometry-aware step, and only then restricts the class of inverse preconditioners used to approximate the resulting update. The learned blocks are therefore optimized in the presence of cross-layer couplings encoded by the LQR dynamics and cost matrices. This provides a systematic way to trade expressivity for scalability while retaining a principled connection to the exact second-order geometry. Our contributions are: 1. A layerwise optimal-control formulation of geometry-aware descent. We show that steepest descent under a broad class of divergence-induced quadratic models—including Newton, Gauss–Newton, natural gradient, and intermediate-layer metrics—can be written exactly as a finite-horizon LQR problem. This reformulation separates the network dynamics from the choice of descent geometry and provides an exact reference update via Riccati recursions. 2. A scalable relaxation based on learned structured inverse preconditioners. We propose to learn a structured inverse preconditioner directly by minimizing the LQR objective. This yields a practical family of geometry-aware optimizers that can use diagonal, Kronecker-factored, or other structured blocks, while preserving the layer-coupled objective induced by the original dense quadratic model. 3. A new optimizer wrapper for modern architectures. The relaxed LLQR update can be wrapped around standard optimizers such as SGDM or AdamW, reusing the learned preconditioner across iterations and avoiding explicit curvature inversion. This makes the method flexible with respect to both the underlying divergence and the chosen preconditioner structure. 4. Empirical validation on ResNets and Transformers. Experiments show that LLQR improves convergence and final performance on image classification and translation benchmarks, and accelerates grokking in Transformers, while adding only modest computational overhead. The results position LLQR as a practical and extensible framework for studying geometry-aware optimization at scale. 2 Related Work The connection between deep learning and optimal control has been recognized since the early days, with back-propagation framed as a control problem and linked to automatic differentiation ( 10 ; 4 ; 23 ) . More recently, neural networks have been cast as discrete-time nonlinear dynamical systems, where layers correspond to time steps and weights to control variables ( 16 ) . Within this view, tools such as Pontryagin’s Maximum Principle and Differential Dynamic Programming have been used to derive new training algorithms ( 24 ; 25 ; 27 ) . Our work differs in focus: rather than reformulating the training problem itself, we recast the steepest-descent step as an optimal control problem. Closer to our approach, 33 proposed a stagewise Newton method for multilayer perceptrons, relying on classical results from 14 —equivalent to 15 that we extend—to propagate Hessian information through the state space without explicitly forming Newton updates. Extensions followed ( 32 ; 35 ; 34 ) , but these remained