Don’t Stop Me Yet : Sampling Loss Minima via Dissipative Riemannian Mechanics

Albert Kjøller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner, Georgios Arvanitidis, [-5pt] Section for Cognitive Systems, Technical University of Denmark · 2026

Read Don’t Stop Me Yet : Sampling Loss Minima via Dissipative Riemannian Mechanics by Albert Kjøller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner, Georgios Arvanitidis, [-5pt] Section for Cognitive Systems, Technical University of Denmark (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15459.txt.

Introduction

Albert Kjøller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner, and Georgios Arvanitidis tackle sampling the connected sets of equivalent neural-network minima rather than spreading loosely over low-loss regions. Their sampler, DiMS, uses a dynamical system with kinetic energy, a gravitational pull, and friction that drains energy, and is guaranteed to sample exactly from minimum level sets. Motivated by uncertainty quantification in Bayesian inference, it improves on earlier approaches in their tests.

Catalog record

CREATORAlbert Kjøller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner, Georgios Arvanitidis, [-5pt] Section for Cognitive Systems, Technical University of Denmark
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100409
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Don’t Stop Me Yet : Sampling Loss Minima via Dissipative Riemannian Mechanics Albert Kjøller Jacobsen, Leo Uhre Jakobsen, Johanna Marie Gegenfurtner, Georgios Arvanitidis, [-5pt] Section for Cognitive Systems, Technical University of Denmark Abstract The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS , is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches. 1 Introduction Figure 1 : Two trajectories through parameter space via different dynamical systems. The sphere constitutes a submanifold of parameters that minimize \mathcal{L}(\boldsymbol{\theta})=\sum_{i=1}^{K}\theta_{i}^{2}-1 . While a geodesic on the loss surface ( ) is the straightest path, it never converges to a minimum solution, even when starting from one. Our modified dynamics ( ) ensure energy dissipation which leads the particle to stop at some minimum solution. Training a deep neural network results in finding a single point estimate with minimum loss within a vast, high-dimensional parameter space. Yet the loss landscape has a rich geometric structure and a single point estimate can only capture local information about it. This can have important shortcomings for downstream tasks, for instance, in Bayesian inference. The architecture and the data together shape the loss landscape such that several parameter configurations perform equally well on the training data; the minimum level sets of modern overparameterized networks form extended, continuous submanifolds for which the model is reparameterization invariant on the observed data ( 7 ; 22 ; 13 ; 10 ) . In this paper we ask “how should we navigate the space of good solutions” rather than “where is a good solution” . We propose the Di ssipative M inima S ampler ( DiMS ), formulated as a dynamical system that navigates the loss landscape into low-loss regions while guaranteeing convergence to the submanifold of minimum solutions. As a result, the gathered samples correspond to a diverse set of functions that agree on the training data but vary in their predictions elsewhere. Recent works have focused on sampling low-loss regions by respecting the inherent geometry of the problem ( 1 ; 33 ; 12 ) , and even restricting samples to the reparameterization invariant set ( 30 ) . Our proposed method differ by guaranteeing convergence to the minimum manifold while allowing for escaping the initial loss basin to find other connected minima - a combination that neither optimization-based methods nor existing sampling-based methods are designed for. Our contributions include: 1. A deterministic sampler for the minimum submanifolds , continuously visiting distinct points on the minimum submanifolds of the training loss. We formalize geometry-driven dynamics on Riemannian manifolds induced by the loss with physically interpretable hyperparameters controlling the exploration behavior of the sampler. The resulting trajectories can be considered as an implicit approximate Bayesian posterior concentrated on the minimum manifold. 2. Theoretical guarantees that our dynamics explore the loss surface before converging to the minimum manifold in finite time. 3. Empirical validation that posterior approximations concentrated on the minimum manifold generalize well, and can improve predictive performance, calibration, and especially out-of-domain uncertainty in Bayesian inference settings. Intuition: The sampling process (Figure 1 ) starts by shooting out a particle from a fixed point \boldsymbol{\theta}^{\ast} that minimizes the loss. The direction and initial speed of the particle is randomly sampled from a distribution, \boldsymbol{v}\sim q(\boldsymbol{v}) , and determines the initial kinetic energy of the particle. By introducing external forces well-known from physics, we guide the particle toward minima, eventually stopping at a minimum level set. Every intersection with this set corresponds to a model that optimally fits the training data, yet exhibits different behavior in other data regions. This leads to high functional variation in regions away from observed data (Figure 2 ), which is desired in Bayesian inference settings. Remark that DiMS is not a traditional approximate posterior as it deliberately generates samples only at the maxima ridges of the true posterior. Figure 2 : We fit a neural network to N=7 training points and show function-space samples from two geometry-aware sampling schemes formulated as continuous dynamical systems. The Riemannian Laplace approximation ( left ) explores low-loss regions but does not enforce interpolating the training data. Our sampler DiMS ( center ) leverages an alternative formulation of the system and guarantees convergence to minimum training loss solutions, ensuring all function-space samples to interpolate the training data, while remaining in low-loss regions along paths generated by the dynamics ( right ). 2 Background - geometry of loss surfaces Let \mathcal{D}=\{(\boldsymbol{x}_{i},y_{i})\}_{i=1}^{N} denote a dataset with N paired input-output observations of \boldsymbol{x}_{i}\in\mathcal{X}\subseteq\mathbb{R}^{D} and y_{i}\in\mathcal{Y} where \mathcal{Y}\subseteq\mathbb{R} for regression or \mathcal{Y}=\{1,\dots,C\} for classification among C classes. In most predictive settings we can define a likelihood parameterized by a neural network, then following Bayes’ theorem the negative log-posterior equals the log-joint up to a constant: \mathcal{L}(\boldsymbol{\theta};\mathcal{D}):=-\log p(\mathcal{D}|\boldsymbol{\theta})-\log p(\boldsymbol{\theta})+\log p(\mathcal{D})=-\log p(\boldsymbol{\theta})-\sum\nolimits_{i=1}^{N}\log p\left(y_{i}|f_{\boldsymbol{\theta}}(\boldsymbol{x}_{i})\right)+C (1) where f_{\boldsymbol{\theta}}:\mathcal{X}\rightarrow\mathcal{Y} is the neural network with parameters \boldsymbol{\theta}\in\Theta and p(\boldsymbol{\theta}) is a prior distribution over the parameter space \Theta\subseteq\mathbb{R}^{K} . The posterior distribution p(\boldsymbol{\theta}|\mathcal{D}) is intractable when working with neural networks and we thus resort to an approximate posterior q(\boldsymbol{\theta}) rather than the true one. Using the generalized Bayesian framework, we formulate a Gibbs posterior over the parameter space that enables a Bayesian treatment of any neural network without requiring a likelihood specification. Figure 3 : The loss surface viewed as a Riemannian manifold (\mathcal{M},g) embedded in \mathbb{R}^{K+1} . The parameter space inherits the geometry of the loss surface via the pullback metric \mathbf{G}(\boldsymbol{\theta}) . A curve on the manifold \boldsymbol{\gamma}_{\boldsymbol{v}}(t)\in\mathcal{M} starting at \boldsymbol{\mu} is generated by the initial velocity \boldsymbol{v} and can be expressed through its corresponding path in the parameter space \boldsymbol{\alpha}_{\tilde{\boldsymbol{v}}}(t)\in\Theta induced by \tilde{\boldsymbol{v}} . We denote the mappings from the initial velocities to the curve at time t by \phi_{\boldsymbol{\mu},t}:\mathcal{T}_{\boldsymbol{\mu}}\mathcal{M}\rightarrow\mathcal{M} and \psi_{\tilde{\boldsymbol{\mu}},t}:\mathcal{T}_{\tilde{\boldsymbol{\mu}}}\Theta\rightarrow\Theta . We briefly review Riemannian geometry of loss surfaces, which forms the foundation of our approach. For notational convenience we suppress the dependency on the data \mathcal{D} unless explicitly needed. 2.1 Defining the loss surface as a Riemannian manifold Given a smooth loss function \mathcal{L}:\mathbb{R}^{K}\rightarrow\mathbb{R} , the graph of the function over the parameter space defines a smooth, K -dimensional submanifold embedded in \mathbb{R}^{K+1} , often called the loss surface: \mathcal{M}=\left\{h(\boldsymbol{\theta})\mid\boldsymbol{\theta}\in\Theta\right\}\subseteq\mathbb{R}^{K+1},\hskip 20.00003pt\textrm{with}\hskip 20.00003pth(\boldsymbol{\theta})=\left(\theta_{1},\dots,\theta_{K},\mathcal{L}(\boldsymbol{\theta})\right). (2) This definition of \mathcal{M} naturally induces a Riemannian metric on the parameter space, as we now establish. For a configuration \boldsymbol{\theta} , the ambient tangent space centered at h(\boldsymbol{\theta}) is spanned by the Jacobian \mathbf{J}_{h}(\boldsymbol{\theta})=\nabla_{\boldsymbol{\theta}}h(\boldsymbol{\theta})=\begin{bmatrix}\mathbb{I}_{K},\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta})\end{bmatrix}^{\top}\in\mathbb{R}^{(K+1)\times K} , and we can express two tangent space vectors \boldsymbol{v}_{1},\boldsymbol{v}_{2}\in\mathcal{T}_{\boldsymbol{p}}\mathcal{M} in parameter space coordinates as \boldsymbol{v}_{1}=\mathbf{J}_{h}(\boldsymbol{\theta})\tilde{\boldsymbol{v}}_{1} and \boldsymbol{v}_{2}=\mathbf{J}_{h}(\boldsymbol{\theta})\tilde{\boldsymbol{v}}_{2}

Read the complete text