Uncertainty quantification of neural network models of evolving processes via Langevin sampling

Cosmin Safta Sandia National Laboratories ,Livermore, CA 94551, Reese E. Jones Sandia National Laboratories ,Livermore, CA 94551Ravi G. Patel Sandia National Laboratories ,Albuquerque, NM 87185Raelynn Wonnacot University of Maryland ,College Park, MD 20742Dan S. Bolintineanu Sandia National Laborato · 2025

Read Uncertainty quantification of neural network models of evolving processes via Langevin sampling by Cosmin Safta Sandia National Laboratories ,Livermore, CA 94551, Reese E. Jones Sandia National Laboratories ,Livermore, CA 94551Ravi G. Patel Sandia National Laboratories ,Albuquerque, NM 87185Raelynn Wonnacot University of Maryland ,College Park, MD 20742Dan S. Bolintineanu Sandia National Laborato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14854.txt.

Introduction

Cosmin Safta and co-authors wrote this 2025 paper on putting honest error bars on neural models of processes that change over time. Their data model couples a neural differential equation, which tracks hidden states as they evolve, with a learned observation step. The catch is knowing how unsure the fitted settings are.

Their answer samples the settings with a Langevin method, a random walk guided by the slope of the probability landscape. That slope is itself learned alongside the model, inside a hypernetwork design that stays scalable. Users can trade computing effort between running the model and refining the uncertainty map. Tests on chemical reaction and materials data show the sampler holding its own against standard variational methods.

Catalog record

CREATORCosmin Safta Sandia National Laboratories ,Livermore, CA 94551, Reese E. Jones Sandia National Laboratories ,Livermore, CA 94551Ravi G. Patel Sandia National Laboratories ,Albuquerque, NM 87185Raelynn Wonnacot University of Maryland ,College Park, MD 20742Dan S. Bolintineanu Sandia National Laborato
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncertainty quantification of neural network models of evolving processes via Langevin sampling Cosmin Safta Sandia National Laboratories ,Livermore, CA 94551, Reese E. Jones Sandia National Laboratories ,Livermore, CA 94551Ravi G. Patel Sandia National Laboratories ,Albuquerque, NM 87185Raelynn Wonnacot University of Maryland ,College Park, MD 20742Dan S. Bolintineanu Sandia National Laboratories ,Albuquerque, NM 87185Craig M. Hamel Sandia National Laboratories ,Albuquerque, NM 87185Sharlotte L.B. Kramer Sandia National Laboratories ,Albuquerque, NM 87185 Abstract We propose a scalable, approximate inference hypernetwork framework for a general model of history-dependent processes. The flexible data model is based on a neural ordinary differential equation (NODE) representing the evolution of internal states together with a trainable observation model subcomponent. The posterior distribution corresponding to the data model parameters (weights and biases) follows a stochastic differential equation with a drift term related to the score of the posterior that is learned jointly with the data model parameters. This Langevin sampling approach offers flexibility in balancing the computational budget between the evaluation cost of the data model and the approximation of the posterior density of its parameters. We demonstrate performance of the ensemble sampling hypernetwork on chemical reaction and material physics data and compare it to standard variational inference. 1 Introduction Many models of physical and other processes require degrees of freedom to represent unobservable states that influence the observable response. The Luenberger observer [ 1 ] from control theory is a widely employed example that provides an estimate of the internal state while modeling measurements of the output of a system. Augmented neural ordinary differential equations (NODEs) [ 2 , 3 ] is another, now ubiquitous, data representation of this type that expands the modeled dynamics to hidden dimensions for greater expressiveness via latent variables. Generally speaking, since the hidden variables in these types of models are inferred or postulated (as opposed to directly measured), these states should be treated as uncertain. A wide variety of methods have been developed to perform uncertainty quantification (UQ). Expensive, sequential Markov chain Monte Carlo algorithms [ 4 , 5 , 6 ] are still considered the reference standards methods. Nevertheless, more efficient optimization-based variational inference (VI) methods [ 7 , 8 ] have arisen as practical alternatives. VI methods employ a parameterized surrogate posterior and posits UQ as an optimization problem for the parameters of the surrogate. More recently, hybrid methods that use an empirical posterior distribution composed of model realizations as well as optimization methods to drive these realizations to match data have arisen. These include Stein VI [ 9 ] and the method we employ, Langevin sampling (LS) [ 10 ] . The proposed hypernetwork framework consists of a hidden state model and an observation model that are parameterized by a sampler that is trained to data. Unlike the widely employed Hamiltonian Monte Carlo approach [ 11 , 12 , 13 ] , LS does not have a conditional acceptance criterion, which simplifies backpropagation. The hypernetwork implementation is relatively straightforward and is fully differentiable; furthermore, LS has a firm theoretical basis which we expand upon in App. A . We illustrate the performance of the framework with process modeling applications from chemical kinetics and material physics. After verifying the convergence properties on simpler problems, we demonstrate the efficiency and accuracy of modifying the method to treat some of the NN parameters with full UQ and others with reoptimized point estimates. In the next section, Sec.​ 2 , we put the proposed method in the context of related work including standard and recently developed UQ methods. Sec.​ 3 provides the theoretical development of the algorithm and implementation details. Then, in Sec.​ 4 , we provide demonstrations of the method’s performance in calibrating and providing UQ for selected physical models. Finally, Sec.​ 5 , provides a summary and avenues for future work. 2 Related work Uncertainty quantification (UQ) is a long-standing field of research with ongoing challenges. Two broad classes of UQ methods are: (a) sampling-based methods, such as Markov chain Monte Carlo (MCMC) methods [ 4 , 6 ] , which generate realizations of the posterior, and (b) optimization-based methods, such as VI [ 7 , 8 ] , which calibrate a surrogate distribution. MCMC algorithms are challenged by high-dimensional parameter spaces, as is the case with NN models with a large number of parameters, while tractable VI methods, such as mean-field VI [ 7 ] , suffer from limited expressiveness. Hybrid ensemble methods have emerged that use particles representing realizations and calibrated dynamics. Stein VI [ 9 ] utilizes Stein’s identity and a kernelized gradient flow to minimize the Kullback-Liebler (KL) divergence between the particles and the posterior underlying the data. Like Hamiltonian Monte Carlo methods [ 11 , 12 , 13 ] , Langevin sampling (LS) borrows from computational statistical mechanics [ 14 ] and was proposed by Welling and Teh [ 10 ] as a generally applicable ensemble method bridging sampling and optimization schemes. Later, LS was employed by Refs. [ 15 , 16 , 17 ] in various applications. LS relies on a particular form of the KL divergence between two random processes [ 18 , 19 , 20 ] , as opposed to random variables/distributions. In its simplest form, it generates samples from a Gaussian process [ 21 ] . This resembles techniques such as flow matching [ 22 ] , which seek to minimize the distance between two random processes. As with some diffusion models [ 23 , 24 , 25 ] and other paradigms [ 26 , 27 , 28 ] , it is based on learning a score function. Furthermore, methods have been designed that have strongly interacting Langevin diffusions [ 29 ] , unlike the approach adopted in this work. We formulate our LS method as a hypernetwork, where the learned score-based sampler provides parameters to the data model, which produces outputs that are compared to data. Hypernetworks have found wide-spread use in a variety of tasks [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 ] . Our utilization of a divided parameter space, where one partition is optimized to improve fit while the other pursues the UQ objective, resembles optimal brain surgery [ 39 , 40 ] , a deterministic method for model-parameter pruning. 3 Methodology The goal of this work is to provide comprehensive uncertainty quantification (UQ) of a class of generally applicable NN models. We combine stochastic sampling of a data matching model in a form of a hypernetwork where a trainable sampler provides weights to the data representation model whose predictions are compared to data. Hypernetwork framework The framework (refer to the schematic in Fig. 1 a) consists of data \mathsf{D} , a data model \mathsf{m} , a loss \mathcal{L} , and a parameter sampler \mathsf{q} . Data consists of input/output pairs \mathsf{D}=\{\mathsf{X},\mathsf{Y}\} . The data model \mathsf{m} takes weights \mathsf{w} from the sampler and data inputs \mathsf{X} to produce outputs \mathsf{Y} \mathsf{Y}_{\mathsf{w}}=\mathsf{m}(\mathsf{X};\mathsf{w}). (1) The sampler is a generative model for the weights \mathsf{w} of model \mathsf{m} \mathsf{w}\sim\mathsf{q}(\mathsf{w}\ |\ \mathsf{m},\mathsf{D};\bm{\theta}), (2) which is parameterized by \bm{\theta} , and relies on a stochastic differential equation (SDE): \mathrm{d}\mathsf{w}=\mathsf{f}(\mathsf{w};\bm{\theta})\mathrm{d}\tau+\gamma\sqrt{2}\mathrm{d}\mathsf{B}_{\tau} (3) to generate samples of the surrogate posterior \mathsf{q} . In Eq.

Read the complete text