Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules

John J. Vastola, Samuel J. Gershman, Kanaka Rajan · 2025

Read Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules by John J. Vastola, Samuel J. Gershman, Kanaka Rajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26997.txt.

Introduction

John Vastola, Samuel Gershman, and Kanaka Rajan ask why some learning rules for tuning models work better than others. A learning rule tells a system how to update its settings to improve performance. Instead of taking rules as given, the authors treat them as policies for moving across a loss landscape, meaning a terrain where height marks error.

Framed as an optimal control problem, familiar rules fall out under different assumptions. Short horizons give plain gradient descent, longer planning gives momentum, and accounting for geometry gives natural gradients. Adaptive methods like Adam arise from tracking landscape shape online. Even resetting weights in continual learning fits as a response to task uncertainty, unifying many tricks under one objective.

Catalog record

CREATORJohn J. Vastola, Samuel J. Gershman, Kanaka Rajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules John J. Vastola, Samuel J. Gershman, Kanaka Rajan Abstract Learning rules—prescriptions for updating model parameters to improve performance—are typically assumed rather than derived. Why do some learning rules work better than others, and under what assumptions can a given rule be considered optimal? We propose a theoretical framework that casts learning rules as policies for navigating (partially observable) loss landscapes, and identifies optimal rules as solutions to an associated optimal control problem. A range of well-known rules emerge naturally within this framework under different assumptions: gradient descent from short-horizon optimization, momentum from longer-horizon planning, natural gradients from accounting for parameter space geometry, non-gradient rules from partial controllability, and adaptive optimizers like Adam from online Bayesian inference of loss landscape shape. We further show that continual learning strategies like weight resetting can be understood as optimal responses to task uncertainty. By unifying these phenomena under a single objective, our framework clarifies the computational structure of learning and offers a principled foundation for designing adaptive algorithms. 1 Introduction A central concern in machine learning is identifying parameters that optimize model performance. Because directly searching for optimal parameters (e.g., via a grid search) is prohibitively costly in the high-dimensional parameter spaces characteristic of models based on artificial neural networks, optimization typically involves seeking iterative improvements in performance rather than directly searching for a global optimum. Procedures for iterative parameter improvement, or learning rules , are most commonly some variant of gradient descent [ 1 , 2 ] , with the backpropagation algorithm [ 3 , 4 ] being a notable example. In biological neural networks, the plausibility of gradient descent is hotly debated [ 5 , 6 ] , and alternative rules that do not follow gradients have been proposed [ 7 , 8 ] . Variants of gradient descent can largely be classified in terms of the presence or absence of three elements: momentum [ 9 , 10 ] , an adaptive learning rate [ 11 , 12 , 13 ] , and loss approximation. From a geometric perspective, gradient descent corresponds to moving down the steepest part of the local loss landscape. In this view, momentum helps ensure smooth parameter changes, even when the loss landscape changes abruptly; an adaptive learning rate allows parameters to change more quickly when gradients are steady, or equivalently in regions where the loss landscape is flat; and computing the loss or its gradients approximately, for example over mini-batches, both improves efficiency and may add noise useful for generalization [ 14 , 15 , 16 ] . Why prefer one learning rule over another? Is a variant of gradient descent always optimal? If so, which one? If not, by what criteria do we construct or decide on an alternative? These questions are usually answered empirically, with optimizers like Adam [ 13 ] popular because they have proven performant in a wide variety of contexts [ 17 , 18 , 19 , 20 ] . It would be helpful to have a normative framework for answering them in a principled fashion, which, given a set of assumptions, identifies some learning rule as ‘optimal’. In this paper, our aim is to provide such a unifying framework. Our three key insights are as follows. First, one can improve performance by optimizing not just over the next small parameter update, but over the whole sequence of future updates; this allows optimization to be less ‘myopic’ and more ‘farsighted’. Second, optimal learning dynamics ought to be sensitive to structure in parameter space. This insight is related to, but slightly more general than, the line of thought that leads to natural gradient descent [ 21 ] . Third, one can view the loss landscape as being only partially observable, which implies optimal learning ought to depend on beliefs about the loss landscape. Assuming partial observability is a useful way to model the fact that the training loss is typically only a proxy for the test loss, which is the true optimization target. We show this idea naturally yields adaptive optimizers like Adam, which update parameters using inferred loss shape. Our framework is significant for two reasons. First, it makes it easier to generate new learning rules from a principled starting point, and to justify them without empirical guesswork. Second, it helps clarify which features of existing learning rules are essential for performance, and which are incidental. In the following sections, we discuss in more detail how different classes of well-known learning rules—including gradient descent with momentum (Sec. 3 ), natural gradient descent (Sec. 4 ), and rules with adaptive learning rates (Sec. 5 )—can all be derived from our framework. Finally, Sec. 6 uses our framework to justify recently-identified rules for continual learning [ 22 ] . 2 Mathematical formulation: learning rules as loss landscape navigation Gradient descent and Newton’s method as optimal single steps through parameter space. To motivate our framework, it is useful to observe that gradient descent and Newton’s method (a second-order analogue [ 23 ] ) minimize a certain objective. Given a loss \mathcal{L}(\boldsymbol{\theta}) that depends on a parameter vector \boldsymbol{\theta} , and a local first- or second-order approximation of it, we would like an update \Delta\boldsymbol{\theta} that decreases the loss as much as possible. Since large enough updates would invalidate our local loss approximation, we want to do this subject to the constraint that \|\Delta\boldsymbol{\theta}\| is not too large. We can do this by minimizing a combination of the loss and a step-size-related regularization term: J(\Delta\boldsymbol{\theta})=\frac{\|\Delta\boldsymbol{\theta}\|^{2}}{2\eta}+\mathcal{L}(\boldsymbol{\theta}+\Delta\boldsymbol{\theta})\approx\begin{cases}\frac{\|\Delta\boldsymbol{\theta}\|^{2}}{2\eta}+\mathcal{L}(\boldsymbol{\theta})+[\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta})]^{T}(\Delta\boldsymbol{\theta})\\ \frac{\|\Delta\boldsymbol{\theta}\|^{2}}{2\eta}+\mathcal{L}(\boldsymbol{\theta})+[\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta})]^{T}(\Delta\boldsymbol{\theta})+\frac{1}{2}(\Delta\boldsymbol{\theta})^{T}\boldsymbol{H}(\boldsymbol{\theta})(\Delta\boldsymbol{\theta})\end{cases} where \eta>0 is the learning rate (which determines the size of the ‘trust region’), and \boldsymbol{H}(\boldsymbol{\theta}) is the Hessian of \mathcal{L} at \boldsymbol{\theta} . Given the first-order loss approximation, the solution is \Delta\boldsymbol{\theta}=-\eta\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta}) , i.e., gradient descent with a learning rate \eta . Geometrically, this step is down the ‘steepest’ part of the loss landscape near \boldsymbol{\theta} . Given the second-order loss approximation, the solution is \Delta\boldsymbol{\theta}=-[\frac{1}{\eta}\boldsymbol{I}+\boldsymbol{H}(\boldsymbol{\theta})]^{-1}\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta}) , i.e., gradient descent with a curvature-sensitive effective learning rate \eta_{\text{eff}}:=[\frac{1}{\eta}\boldsymbol{I}+\boldsymbol{H}(\boldsymbol{\theta})]^{-1} . One still takes a step down the steepest part of the loss landscape, but takes a larger step if the landscape is very flat; this avoids the slowdown gradient descent faces near a local minimum, where gradients get smaller. Newton’s method technically only corresponds to the large \eta limit, where \eta_{\text{eff}}=\boldsymbol{H}(\boldsymbol{\theta})^{-1} ; more generally, we obtain a regularized version, which (given a specific Hessian approximation) is sometimes called the Levenberg-Marquardt algorithm [ 24 ] . Figure 1: Basic idea of our framework and a simple example. a. Single-step approaches (top) optimize over short-term changes to the loss, while a multi-step approach (bottom) optimizes over longer-term changes. b. Gradient descent vs multi-step optimization for a double-well loss, with the optimal multi-step trajectory computed by directly minimizing the objective (see Appendix B for details). Note that the multi-step rule converges to the global rather than local minimum. c. Values of the kinetic (top) and potential (bottom) terms along the optimal trajectory from (b). The loss/potential does not decrease monotonically, since the learner must first escape a local minimum. Learning rules as partially observable loss landscape navigation. The single-step view of loss optimization is arguably myopic, since the best short-term improvement

Read the complete text