Optimizers Qualitatively Alter Solutions And We Should Leverage This
Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, James Martens · 2025
Read Optimizers Qualitatively Alter Solutions And We Should Leverage This by Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, James Martens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12224.txt.
Introduction
The optimizer that trains a neural network is usually judged by how fast it learns, and Razvan Pascanu and co-authors say that misses half the story. Because network loss surfaces are nonlinear, different optimizers land on genuinely different solutions, not just the same answer at different speeds. The choice quietly encodes inductive biases and can change what a model class is able to express.
The authors argue the field should map the biases of existing optimizers on purpose. Better still, new optimizers could be designed to push solutions toward wanted properties directly. They frame optimizer design as a lever beside architecture and data in shaping what models become.
Catalog record
| CREATOR | Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, James Martens |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizers Qualitatively Alter Solutions And We Should Leverage This Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, James Martens Abstract Due to the nonlinear nature of deep neural networks, one can not guarantee convergence to a unique global minimum of the loss when using optimizers that rely only on local information, such as gradient descent. Indeed, this was a primary source of skepticism regarding the feasibility of neural networks in the early days of the field. The past decades of progress in deep learning have revealed this skepticism to be misplaced, and a large body of empirical evidence shows that sufficiently large networks following standard training protocols exhibit well-behaved optimization dynamics that converge to performant solutions. This success has biased the community to use convex optimization as a mental model for learning, leading to a focus on training efficiency — either in terms of required iteration, FLOPs or wall-clock time — when improving learning algorithms. We argue that, while this perspective has proven extremely fruitful, another perspective specific to neural networks has received considerably less attention: namely, that the choice of optimizer (or learning algorithm) not only influences the rate of convergence, but also the qualitative properties of the learned solutions. Restated, the choice of optimizer can and will encode inductive biases and change the effective expressivity of a given class of models . Furthermore, we believe that the choice of the optimizer can be an effective way of encoding desiderata in the learning process. We contend that the community should aim at understanding the biases of already existing methods, as well as aim to build new learning algorithms with the explicit intent of inducing certain properties of the solution, rather than solely judging them based on their convergence rates. We hope that our arguments will inspire research to improve our understanding of how the learning process can impact the type of solution we converge to, and lead to a greater recognition of learning algorithm design as a critical lever that complements the roles of architecture and data in shaping model outcomes. 1 1 footnotetext: Google DeepMind, UK 2 2 footnotetext: Mila, Québec AI Institute, Canada 3 3 footnotetext: Institute of Science and Technology Austria (ISTA) 4 4 footnotetext: Technische Universität Berlin, Germany & BIFOLD Berlin 5 5 footnotetext: Red Hat AI 6 6 footnotetext: Cambridge University, UK 7 7 footnotetext: Politechnique Montreal, Canada 1 Introduction Neural network research has a long and rich history, going back at least as far as McCulloch and Pitts 1943 . This journey has been marked by cycles of intense optimism followed by periods of skepticism regarding their efficacy, famously exemplified by the critique of the limitations of perceptrons to learn the XOR problem by Minsky and Papert 1969 . The current success of these models, starting with works like [ 36 , 11 , 65 ] and the pivotal result on ImageNet by Krizhevsky et al. 2012 , rely heavily on the adoption of gradient based learning rules, made possible by the back-propagation algorithm [ 67 ] . Succinctly, modern machine learning relies on an iterative local learning rule, updating model parameters along a descent direction given by the gradient, optionally preconditioned by a matrix P . Given the locality of the approach, and the non-convex nature of the neural network, a natural concern is: what kind of guarantees can one obtain in terms of the optimality of the solution ? Interestingly, this was among the main themes in the critique around the XOR problem put forward in [ 54 ] ; it resurfaced in different forms in the 90s and early 2000s, as a source of skepticism towards connectionist approaches to machine learning. For further discussion please see [ 75 , 10 , 9 ] . The field approached this question empirically, with initial successes coming from layer-wise pretraining [ 36 , 11 ] , which was assumed to initialize the model into a good basin of attraction as argued by the careful study of [ 25 ] . Shortly thereafter, works such as [ 51 , 31 ] showed that good initialization and careful choice of optimizer can lead to strong performance when training these models, despite the persistent worry of bad local minima . This has now become standard training protocol, achieving remarkable success. Figure 1 : Compared to the convex case, where optimizers can converge to a global minima, in the non-convex case (e.g. for neural networks), different optimizers can lead training to converge to different minima. We argue that for neural networks, the choice of optimizer can lead to qualitatively different kinds of solutions, and that one can effectively leverage this as a mechanism for introducing inductive biases in learning, similar to architecture design. Follow-up work tried to formulate theoretical arguments for these observations, and study this behavior empirically, e.g. [ 19 , 17 , 32 , 79 , 46 , 74 ] . These works suggested that, at scale, the loss landscape becomes well behaved, almost convex , and that all local minima lead to similar performance to global minima. This included theoretical arguments borrowed from statistical physics [ 19 , e.g.] or empirical studies of how loss varies when interpolating between initialization and the convergence point, showing a monotonically decreasing behavior [ 32 ] , akin to the convex case. Note that the result of Goodfellow et al. 2015 was later questioned by Frankle 2020 , however the original impact of the work on how learning was perceived cannot be ignored. Local minima in fact often form linearly-connected basins [ 77 ] , where a convex combination of two local minima will also be an (approximate) local minimum. It was also shown in [ 64 ] that it is possible to linearly interpolate between parameters fine-tuned towards different task reward functions without degrading average performance on these tasks. However, recent works added nuance to this perspective, arguing that learning is actually split into at least two stages, a short initial stage in which learning jumps from one basin of attraction to another [ 39 ] , and a second much longer stage wherein learning works within a locally convex region. Learning can fall within the lazy regime , characterized by the Neural Tangent Kernel (NTK) [ 38 ] , where model parameters remain close to initialization [ 16 ] , and, conversely, in the rich feature-learning regime , characterized by models achieving generalization capabilities [ 78 ] . While our understanding of learning dynamics has improved over time, such initial results biased the community towards not only confidently borrowing ideas from the convex optimization literature and applying them to neural networks, but also on overly emphasizing the importance of convergence speed—the standard metric in convex optimization—as the main target when developing new learning algorithms. In some sense, the role of the optimizer is typically seen as being to exploit local convexity and reach as fast as possible the minima of the basin of attraction the model finds itself in. The early literature providing the belief that generally for a sufficiently large model the basin of attraction the model finds itself in due to initialization and used protocols