Statistical accuracy of the ensemble Kalman filter in the near-linear setting

E. Calvello, J. A. Carrillo, F. Hoffmann, P. Monmarché, A. M. Stuart, U. Vaes · 2025

Read Statistical accuracy of the ensemble Kalman filter in the near-linear setting by E. Calvello, J. A. Carrillo, F. Hoffmann, P. Monmarché, A. M. Stuart, U. Vaes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16154.txt.

Introduction

Calvello and co-authors gave an overview in 2025 of recent work on the statistical accuracy of the ensemble Kalman filter. Estimating a hidden state from partial, noisy data matters for weather forecasting and epidemic prediction. Particle filters are accurate but collapse in high dimensions, while this filter keeps equal weights and avoids that fault.

Proofs of its accuracy, though, mostly cover linear models with linear measures, and the wider case is young. This short note makes recent first steps beyond the linear Gaussian setting accessible to readers. It surveys what is now known and where analysis still lags, for a method widely used in the geophysical sciences.

Catalog record

CREATORE. Calvello, J. A. Carrillo, F. Hoffmann, P. Monmarché, A. M. Stuart, U. Vaes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistical accuracy of the ensemble Kalman filter in the near-linear setting E. Calvello, J. A. Carrillo, F. Hoffmann, P. Monmarché, A. M. Stuart, U. Vaes Abstract . Estimating the state of a dynamical system from partial and noisy observations is a ubiquitous problem in a large number of applications, such as probabilistic weather forecasting and prediction of epidemics. Particle filters are a widely adopted approach to the problem and provide provably accurate approximations of the statistics of the state, but they perform poorly in high dimensions because of weight collapse. The ensemble Kalman filter does not suffer from this issue, as it relies on an interacting particle system with equal weights. Despite its wide adoption in the geophysical sciences, mathematical analysis of the accuracy of this filter is predominantly confined to the setting of linear dynamical models and linear observations operators, and analysis beyond the linear Gaussian setting is still in its infancy. In this short note, we provide an accessible overview of recent work in which the authors take first steps to analyze the accuracy of the filter beyond the linear Gaussian setting [ 3 , 4 , 1 ] . {resume} Estimer l’état d’un système dynamique à partir d’observations partielles et bruitées est un problème omniprésent dans de nombreuses applications, telles que la prévision météo et la prédiction des épidémies. Les filtres particulaires sont largement utilisés pour résoudre ce problème et fournissent des approximations des statistiques de l’état dont la précision peut être démontrée rigoureusement. Cependant, ils fonctionnent mal en grande dimension en raison de la disparité des poids associés aux particules. Le filtre de Kalman d’ensemble ne souffre pas de ce problème, car il repose sur un système de particules en interaction avec des poids égaux. Malgré sa popularité dans les sciences géophysiques, l’analyse mathématique de la précision de ce filtre est principalement limitée au cadre des modèles dynamiques linéaires et des opérateurs d’observation linéaires, et l’analyse au-delà de ce cadre en est encore à ses débuts. Dans cette courte note, nous présentons un aperçu accessible de travaux récents dans lesquels la précision du filtre est étudiée au-delà du cadre linéaire gaussien [ 3 , 4 , 1 ] . Introduction This note is partly based on [ 4 ] . Consider a discrete-time dynamical system with state \{v_{n}\}_{n\in\mathbb{N}} evolving in \mathbb{R}^{d} , observed through partial and noisy data \{y_{n}\}_{n\in\mathbb{N}} in \mathbb{R}^{K} . Suppose that the state and data are determined by the following system: \displaystyle\text{ \bf State evolution: }\qquad v_{n+1} \displaystyle=\Psi(v_{n})+\xi_{n}\,, \displaystyle\text{ \bf Data aquisition: }\qquad y_{n+1} \displaystyle=h(v_{n+1})+\eta_{n+1}\,. This holds for n\in\mathbb{N}=\{0,1,2,\cdots\} . We assume that the initial state of the system is a Gaussian random variable v_{0}\sim\mathcal{N}(m_{0},C_{0}) with known mean and covariance. In addition, we assume that the noise entering in the state evolution is distributed according to the normal distribution \xi_{n}\sim\mathcal{N}(0,\Sigma) , and that the noise affecting the observation satisfies \eta_{n+1}\sim\mathcal{N}(0,\Gamma) . We assume that both noise sequences are i.i.d. and satisfy the following independence assumption: v_{0}\perp\!\!\!\!\perp\{\xi_{n}\}_{n\in\mathbb{N}}\perp\!\!\!\!\perp\{\eta_{n+1}\}_{n\in\mathbb{N}}. The objective of the filtering problem is to determine, and update sequentially as new data arrives, the probability distribution on state v_{n} given all the data observed up to time n . To state the problem more precisely, we consider a given realization of the data denoted by a dagger and define Y^{\dagger}_{n}=\{y^{\dagger}_{\ell}\}_{\ell=1}^{n},\qquad v_{n}|Y^{\dagger}_{n}\sim\mathfrak{\mu}_{n}. The objective of filtering methods can then be reformulated precisely as follows: it is to approximate the filtering distribution or true filter \mathfrak{\mu}_{n} and update it sequentially in n . 1. Evolution of the filtering distribution In this section, we show that the law \mathfrak{\mu}_{n} evolves according to a nonlinear and nonautonomous dynamical system on the space of probability measures. To determine this dynamical system, and the approximation of it by the ensemble Kalman filter discussed in Section 2 , we denote by \mathcal{P}(\mathbb{R}^{r}) the set all probability measures on \mathbb{R}^{r} , and by \mathcal{G}(\mathbb{R}^{r}) all the set Gaussian probability measures on \mathbb{R}^{r} . For simplicity we will use the same symbol for measures and their densities throughout this article. To write the evolution of \mu_{n} in a compact manner, it is useful to introduce key operators on probability measures: linear operator \mathsf{P}\colon\mathcal{P}(\mathbb{R}^{d})\to\mathcal{P}(\mathbb{R}^{d}) is defined by \mathsf{P}\nu(v)=\frac{1}{\sqrt{(2\pi)^{d}\det\Sigma}}\int_{\mathbb{R}^{d}}\exp\left(-\frac{1}{2}|v-\Psi(u)|_{\Sigma}^{2}\right)\nu(u)\,\mathrm{d}u, and linear operator \mathsf{Q}\colon\mathcal{P}(\mathbb{R}^{d})\to\mathcal{P}(\mathbb{R}^{d}\times\mathbb{R}^{K}) is determined by \mathsf{Q}\mathfrak{\pi}(v,y)=\frac{1}{\sqrt{(2\pi)^{K}\det\Gamma}}\exp\left(-\frac{1}{2}\bigl\lvert y-h(v)\bigr\rvert_{\Gamma}^{2}\right)\mathfrak{\pi}(v). The linear operator \mathsf{P} determines the evolution of the law of Markov process \{v_{n}\} , while the linear operator \mathsf{Q} lifts the law of v_{n} to the joint space of state and data (v_{n},y_{n}). Lastly, we denote by \mathsf{B}(\,\cdot\,;y^{\dagger})\colon\mathcal{P}(\mathbb{R}^{d}\times\mathbb{R}^{K})\to\mathcal{P}(\mathbb{R}^{d}) the nonlinear operator effecting conditioning of a joint random variable (v,y) on observation y=y^{\dagger} : \mathsf{B}(\pi;y^{\dagger})(v)=\frac{\pi(v,y^{\dagger})}{\int_{\mathbb{R}^{d}}\pi(u,y^{\dagger})\,\mathrm{d}u}. Having introduced the necessary notation, we define the probability distribution \widehat{\mu}_{n+1} through v_{n+1}|Y^{\dagger}_{n}\sim\widehat{\mu}_{n+1} and observe that (1a) \displaystyle\widehat{\mu}_{n+1} \displaystyle=\mathsf{P}\mathfrak{\mu}_{n},\qquad \displaystyle v_{n+1}|Y^{\dagger}_{n}\sim\widehat{\mu}_{n+1} (1b) \displaystyle\pi_{n+1} \displaystyle=\mathsf{Q}\widehat{\mu}_{n+1},\qquad \displaystyle(v_{n+1},y_{n+1})|Y^{\dagger}_{n}\sim\pi_{n+1} (1c) \displaystyle\mathfrak{\mu}_{n+1} \displaystyle=\mathsf{B}(\pi_{n+1};y^{\dagger}_{n+1}),\qquad \displaystyle{\rm conditioning}\,{\rm on}\,y_{n+1}=y^{\dagger}_{n+1}. This update formula for \mu_{n} may be decomposed into a prediction step involving the dynamical model through \mathsf{P} , and an analysis step based on Bayes’ theorem through the composition \widehat{\mu}_{n+1}\mapsto\mathsf{B}(\mathsf{Q}\widehat{\mu}_{n+1};y^{\dagger}_{n+1}) . This iterative interleaving of prediction and analysis steps defines an infinite dimensional dynamical system over the space of probability measures, numerical approximation of which is a major challenge. 2. Ensemble Kalman Filter The ensemble Kalman filter was originally introduced by Evensen in [ 6 ] . We refer to the works [ 7 , 8 ] for a detailed review. In this section, we first present the basic particle formulation of the algorithm. Next, we show how this formulation can be derived as an approximation of a mean-field dynamics, an illuminating perspective fleshed out in [ 2 ] . Finally, we present stability and error estimates for the mean field filter, and show how these can be used to study the connection between the finite particle ensemble Kalman filter and the true filter [ 3 ] . Before presenting the particle formulation of the ensemble Kalman filter, we introduce useful notation: for a probability measure \pi\in\mathcal{P}(\mathbb{R}^{d}\times\mathbb{R}^{K}) , we write the mean under \pi as \mathcal{M}(\pi) and the covariance under \pi as \mathcal{C}(\pi)=\begin{pmatrix}\mathcal{C}^{vv}(\pi)&\mathcal{C}^{vy}(\pi)\\ \mathcal{C}^{vy}(\pi)^{\top}&\mathcal{C}^{yy}(\pi)\end{pmatrix}. For h\colon\mathbb{R}^{d}\to\mathbb{R}^{K} we also define {\mathcal{C}}^{hh}(\pi) as the covariance matrix of the vector h(v) , for v distributed according to the marginal of \pi on the state, and {\mathcal{C}}^{vh}(\pi) as the covariance between v and h(v) . An iteration of the ensemble Kalman filter then reads, for n\in\mathbb{N} , (2a) \displaystyle\widehat{v}_{n+1}^{(j)} \displaystyle=\Psi\bigl(v_{n}^{(j)}\bigr)+\xi_{n}^{(j)}, (2b) \displaystyle\widehat{y}_{n+1}^{(j)} \displaystyle=h(\widehat{v}_{n+1}^{(j)})+\eta_{n+1}^{(j)}, (2c) \displaystyle v_{n+1}^{(j)} \displaystyle=\widehat{v}_{n+1}^{(j)}+\mathcal{C}^{vh}\bigl(\pi^{{\rm EK},J}_{n+1}\bigr)\left(\Gamma+\mathcal{C}^{hh}\bigl(\pi^{{\rm EK},J}_{n+1}\bigr)\right)^{-1}\Bigl(y^{\dagger}_{n+1}-\widehat{y}_{n+1}^{(j)}\Bigr), (2d) \displaystyle\pi^{{\rm EK},J}_{n+1} \displaystyle=\frac{1}{J}\sum_{j=1}^{J}\delta_{\bigl(\widehat{v}_{n+1}^{(j)},\widehat{y}_{n+1}^{(j)}\bigr)}. Here \xi_{n}^{(j)}\sim\mathcal{N}(0,\Sigma) and \eta_{n}^{(j)}\sim\mathcal{N}(0,\Gamma) are i.i.d. with respect to both n and j , and the two sets are independent. For n=0 we choose independent v_{0}^{(j)}\sim\mathfrak{\mu}_{0} . From the particles evolving according to iteration ( 2 ), we define the following empirical measure as an approximation of the filtering distribution: (3) \mu^{{\rm EK},J}_{n}=\frac{1}{J}\sum_{j=1}^{J}\delta_{v_{n}^{(j)}}. Since an equal weight is assigned to all the particles, the ensemble Kalman filter cannot suffer from weight collapse, which contributes to its popularity among practitioners. Furthermore, in the setting where the functions \Psi,h are both affine and \mu_{0} is Gaussian, the empirical approximation \mu^{{\rm EK},J}_{n} converges to \mathfrak{\mu}_{n} at the Monte Carlo rate in the limit as J\to\infty , in an appropriate metric [ 12 , 13 ] . In this setting, the ensemble Kalman filter is an alternative to the classical Kalman filter [ 11 ]

Read the complete text