Navigating Potholes with Geometry-Aware Sharpness Minimization

Simon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu, Ioannis Mitliagkas, Damien Scieur, Aristide Baratin · 2026

Read Navigating Potholes with Geometry-Aware Sharpness Minimization by Simon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu, Ioannis Mitliagkas, Damien Scieur, Aristide Baratin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16134.txt.

Introduction

Dufort-Labbé and five colleagues introduce LLQR+SAM, a sharpness-aware minimization method that respects loss geometry. Standard SAM perturbs parameters uniformly, ignoring that some directions curve more than others. A learned preconditioner from the LLQR framework captures a smoothed picture of the landscape, and SAM then probes curvature on top of it. Theory shows the preconditioner amplifies escape signals from locally sharp potholes while wide flat basins stay stable, and experiments beat both SAM and LLQR alone.

Catalog record

CREATORSimon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu, Ioannis Mitliagkas, Damien Scieur, Aristide Baratin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60125
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Navigating Potholes with Geometry-Aware Sharpness Minimization Simon Dufort-Labbé, Mehrab Hamidi, Razvan Pascanu, Ioannis Mitliagkas, Damien Scieur, Aristide Baratin Abstract Sharpness-aware minimization (SAM) encourages flat minima by perturbing parameters along directions of high loss curvature, but treats all parameter directions uniformly, ignoring the underlying loss geometry. We introduce LLQR+SAM, which combines SAM with a learned preconditioner obtained from the recently proposed LLQR framework–a second-order method that recasts steepest descent as a layerwise linear-quadratic regulator problem. The preconditioner is updated sparsely and maintained as a slow exponential moving average, so it captures a smoothed, low-resolution picture of the loss landscape geometry. The SAM perturbation then operates on top of this learned geometry, probing curvature at a faster timescale. We show that this two-timescale structure is not merely a computational convenience: theoretically, the preconditioner amplifies the SAM escape signal in directions that are flat under the average geometry but locally sharp (“potholes”). Wide, flat basins, by contrast, remain stable. Empirically, LLQR+SAM gives consistent gains over both SAM and LLQR alone across standard vision and sequence modeling benchmarks, supporting the view that slow learned geometry and fast sharpness correction are genuinely complementary. Figure 1: Interaction between FSAM and LLQR on ResNet-50/ImageNet. Left : Top-1 error for SGDM and FSAM, a SAM variant, with and without LLQR. Although both SAM-style methods and LLQR can be interpreted as curvature-correcting mechanisms, their combination yields gains over either component alone, suggesting complementary effects. Right : Test loss versus elapsed training time. Despite the usual concern that second-order methods are prohibitively expensive, LLQR remains efficient enough not to become the bottleneck: whenever the slowdown of SAM is acceptable, topping it with LLQR likely preserves affordability. 1 Introduction Modern deep-learning loss landscapes are widely thought to combine smooth, slowly-varying structure at large scales—wide basins, valleys, and ridges shaped by the architecture and data—with much sharper features at smaller scales: narrow valleys, spurious minima, and localized potholes that act as attractors of poor generalization. Two largely independent lines of optimization research target these two scales. Geometry-aware methods such as natural gradient ( 1 ) , Newton, K-FAC ( 19 ) and Shampoo ( 8 ) use estimates of curvature to reshape the descent direction, addressing the ill-conditioning of the average loss geometry. Sharpness-aware methods, beginning with SAM ( 7 ) , perturb the parameters before evaluating the gradient, biasing optimization toward flatter minima. These two mechanisms operate on different aspects of the geometry, and at very different time scales. A learned preconditioner accumulated over many gradient steps can only capture the slow average curvature of the landscape; it is by construction insensitive to sharp localized features. A SAM perturbation, evaluated on a single minibatch, probes local sharpness directly—but treats all parameter directions uniformly, ignoring the underlying optimization geometry. Each captures something the other misses. In this work, we study the combination of the two. We use LLQR ( 6 ) , a recently proposed framework that learns a structured inverse preconditioner U by minimizing a linear-quadratic-regulator (LQR) objective derived from the layerwise network dynamics and a chosen divergence on the loss. The preconditioner is updated sparsely (a few times per epoch) and maintained as an exponential moving average; it captures a smoothed, low-resolution picture of the average loss geometry. On top of this slow geometry we apply a SAM perturbation evaluated on each minibatch in the U -induced norm , and the resulting outer update uses the same learned geometry. The combined algorithm, LLQR+SAM , applies the SAM displacement and the gradient transport in a single learned metric. Our central claim is that this two-time-scale combination is synergistic , not merely additive. We make this precise on a model landscape consisting of a quadratic loss with a smooth average geometry, supplemented by a sharp localized pothole that creates a spurious minimum invisible to U . On this landscape the LLQR+SAM dynamics is closed-form and exposes a clean mechanism: SAM prevents the iterate from settling at any minimum, forcing it to hover at the scale of the SAM probe itself, while U shapes that scale according to the average geometry. Around a wide basin the hovering is appropriate to the basin width and the iterate is stable; around a pothole the same hovering scale exceeds the pothole’s basin of attraction and the iterate escapes. Crucially, the escape signal is amplified relative to vanilla SAM by a factor that grows as the surrounding basin becomes flatter—precisely the regime where preconditioning matters most. The same U that enables fast convergence in flat directions also enables pothole escape: U and SAM are not independent ingredients but two complementary uses of one learned geometry. These predictions are borne out by the toy experiments in Figures 2 and 3 and by the empirical study of Section 5 , which shows consistent gains over either component alone across CIFAR, TinyImageNet, ImageNet, and IWSLT14. Contributions. (i) We propose LLQR+SAM, combining a slowly-updated, EMA-smoothed second-order preconditioner with a SAM perturbation evaluated in the induced geometry. (ii) On a quadratic landscape with a sharp pothole perturbation we derive the exact LLQR+SAM dynamics and quantify the escape-amplification of the preconditioner relative to vanilla SAM, with the gain growing as the surrounding basin becomes flatter. (iii) We validate empirically that the combination yields consistent gains over either component alone across standard vision and sequence benchmarks. 2 Related Work Sharpness, flatness, and SAM. The relation between generalization and loss-landscape sharpness has long motivated methods that bias training toward flatter minima ( 11 ; 12 ; 21 ) , although this connection is neither universal nor invariant: sharpness can change under reparameterization without changing the represented function ( 4 ) . SAM ( 7 ) is the canonical sharpness-aware optimizer, replacing the training objective by a local min–max problem that probes the worst-case loss near the current parameters and descends using the gradient at the perturbed point. Many variants modify either the perturbation neighborhood or the adversarial-point estimate: ASAM ( 15 ) uses a scale-adaptive radius to reduce sensitivity to parameter rescaling, while Friendly SAM ( 18 ) reduces minibatch-noise sensitivity by averaging perturbation information across batches. A complementary view comes from gradient-norm penalization: 28 argue that penalizing the gradient norm encourages smaller local Lipschitz constants and show that SAM can be recovered as a special case of a loss augmented by a gradient-norm regularizer. Geometry-aware sharpness neighborhoods. Vanilla SAM defines sharpness through Euclidean neighborhoods in parameter space, which may poorly reflect the effective geometry of neural networks ( 17 ) . Fisher SAM addresses this by replacing the Euclidean ball with an ellipsoid induced by the Fisher information matrix, so the adversarial probe is measured in a geometry tied to the model distribution ( 13 ) . Riemannian SAM further generalizes this idea by defining the sharpness neighborhood through an arbitrary Riemannian metric ( 26 ) . In this view, Fisher SAM is a special case, and sharpness-aware perturbations are interpreted as steepest-ascent directions under a non-Euclidean local norm. Geometry-aware optimization and robustness. A separate line of work uses geometry to transport gradients rather than to define adversarial neighborhoods. Natural-gradient descent defines steepest descent with respect to the Fisher information metric ( 1 ) . K-FAC makes this principle scalable through layerwise Kronecker-factored Fisher

Read the complete text