Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

Melissa Adrian, Daniel Sanz-Alonso, Rebecca Willett · 2026

Read Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms by Melissa Adrian, Daniel Sanz-Alonso, Rebecca Willett (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20891.txt.

Introduction

Adrian and co-authors present a framework that learns data assimilation pieces end to end. Data assimilation estimates a system's state from partial, noisy observations. Good results depend on costly tuning and an accurate dynamics model. Their method learns the state, the dynamics, and filter settings at once, using a theory based loss. They test it on systems from aerospace, weather science, and biology. They also show known methods can be learned or tuned inside the framework, and give practical setup guidance.

Catalog record

CREATORMelissa Adrian, Daniel Sanz-Alonso, Rebecca Willett
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102256
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms Melissa Adrian, Daniel Sanz-Alonso, Rebecca Willett Abstract Data assimilation algorithms estimate the state of a dynamical system from partial observations, where the successful performance of these algorithms hinges on costly parameter tuning and on employing an accurate model for the dynamics. This paper introduces a framework for jointly learning the state, dynamics, and parameters of filtering algorithms in data assimilation through a process we refer to as auto-differentiable filtering. The framework leverages a theoretically motivated loss function that enables learning from partial, noisy observations via gradient-based optimization using auto-differentiation. We further demonstrate how several well-known data assimilation methods can be learned or tuned within this framework. To underscore the versatility of auto-differentiable filtering, we perform experiments on dynamical systems spanning multiple scientific domains, such as the Clohessy-Wiltshire equations from aerospace engineering, the Lorenz-96 system from atmospheric science, and the generalized Lotka-Volterra equations from systems biology. Finally, we provide guidelines for practitioners to customize our framework according to their observation model, accuracy requirements, and computational budget. 1 Introduction In many applied scientific settings, practitioners have access to observational data that provides a partial and noisy snapshot of a dynamical system. This data may be utilized for a variety of purposes, including inferring interpretable parameters of the system, improving forecast skill, and reconstructing the underlying state of the system in the past. However, the noisy and incomplete nature of these observations poses significant challenges for both inference and prediction. Data assimilation provides a class of methods to combine data sources of various modalities, denoise, and “fill in the gaps” in a principled way. Data assimilation was proposed in Richardson, 1922 for atmospheric state estimation with the goal of improving weather prediction. Early data assimilation methods include the Kalman filter ( Kalman, 1960 ) and optimal interpolation ( Gandin, 1966 ) , which handle assimilating observations from linear systems, both in terms of dynamics and observation models. As the need increased for data assimilation algorithms that could handle nonlinearities, the variational approaches 3DVar ( Lorenc, 1986 ) and 4DVar ( Courtier et al., 1994 ) were developed. The landscape for data assimilation significantly changed when ensemble-based methods ( Palmer et al., 1992 ) were introduced, allowing for improved notions of uncertainty quantification for nonlinear systems. A prototypical ensemble-based data assimilation method is the ensemble Kalman filter ( Evensen, 1994 ; Evensen, 2003 , EnKF) , which inspired hybrid variational-ensemble approaches such as the ensemble 3DVar ( Hamill and Snyder, 2000 , Ens3DVar) and the ensemble 4DVar ( Liu et al., 2008 , Ens4DVar) . Among these choices, there is no one catch-all algorithm; practitioners weigh each approach’s advantages and disadvantages and choose the algorithm that balances specific considerations for their applied needs. One important challenge in data assimilation is that the dynamics model governing the evolution of the state is often expensive to simulate and subject to model error. A recent and promising approach to alleviate these issues is to leverage machine-learned surrogate models for the dynamics ( Chen, 2023 ; Bach et al., 2025b ; Adrian et al., 2025 ; Sanz-Alonso and Waniorek, 2025 ; Chakraborty et al., 2025 ) . The wide success of machine learning in building surrogates, particularly in emulating physical systems, can be attributed to the open availability of high-quality training data. In many domains, expensive processing to curate this high-quality training data needs to occur before surrogate learning even begins. Particularly for high-dimensional settings, fine-grained information may not be readily available, and data assimilation algorithms may be required to distill fine-scale features from coarse measurements. Thus, there is a bidirectional relationship between machine learning and data assimilation: data assimilation algorithms benefit from machine-learned dynamics models, and machine learning surrogate models benefit from training data processed using data assimilation algorithms. Data assimilation algorithms, though essential in many applications, require substantial tuning before they can be deployed in practice. For example, in 3DVar a background covariance matrix describing the time-independent covariances of forecast errors needs to be curated prior to using this algorithm. In atmospheric data assimilation, the background error covariance has been traditionally estimated via the National Meteorological Center (NMC) method ( Parrish and Derber, 1992 ) , which aims to estimate this error covariance matrix in the absence of complete ground truth data. Moreover, parameters such as covariance inflation and the covariance mixing parameter in ensemble schemes additionally require prior specification, requiring multiple runs of the algorithm for tuning. In this paper, we present a general framework that allows for learning updates to the dynamics model and data assimilation parameters based on noisy, potentially sparse observations of an underlying dynamical system, thereby streamlining the process to directly utilize observations. The framework can be used to learn nonlinear dynamics given linear observations of the state. As we will detail below, our framework generalizes existing learned data assimilation methods, including AD-EnKF ( Chen et al., 2022 ) and AD-3DVar-K ( Levine and Stuart, 2022 ) . Furthermore, our framework leads to new methods that are efficacious for time-varying observation models or which require fewer computational resources. We formalize the problem setting in the next section. 1.1 Problem setting Assume we collect observations y_{1:T} derived from the following hidden Markov model \displaystyle x^{*}_{t} \displaystyle=\mathcal{F}_{\theta_{1}^{*}}(x_{t-1}^{*})+\tilde{Q}_{\theta_{2}^{*}}\xi_{t},\qquad\xi_{t}\sim N(0,I_{d_{x}}), (1) \displaystyle y_{t} \displaystyle=H_{t}x_{t}^{*}+\eta_{t},\quad\quad\quad\quad\quad\ \ \eta_{t}\sim N(0,R_{t}), (2) \displaystyle x^{*}_{0} \displaystyle\sim p_{0}(x_{0}) (3) for time points t=1,\dots,T , where x_{t}^{*}\in\mathbb{R}^{d_{x}} is the true state vector at time t , \mathcal{F}_{\theta_{1}^{*}} is the true dynamics map parameterized by true parameters \theta_{1}^{*} , Q_{\theta_{2}^{*}}:=\tilde{Q}_{\theta_{2}^{*}}\tilde{Q}_{\theta_{2}^{*}}^{\top}\in\mathbb{R}^{d_{x}\times d_{x}} is the true forecast error covariance parameterized by \theta_{2}^{*} , y_{t}\in\mathbb{R}^{d_{y_{t}}} are noisy, potentially partial observations of x_{t}^{*} at time t , H_{t}\in\mathbb{R}^{d_{y_{t}}\times d_{x}} is the time-varying, known linear observation matrix, R_{t}\in\mathbb{R}^{d_{y_{t}}\times d_{y_{t}}} is the time-varying, known observation error covariance, and the initialization x^{*}_{0} is assumed to be unknown and drawn from a given prior distribution p_{0}. In this setting, we assume independence of x^{*}_{0} , \xi_{1:T} , and \eta_{1:T} . 1 1 1 To explain our notation, \xi_{1:T} , as an example, denotes the time series of forecast model errors for time points t=1,\dots,T . In practice, the true dynamics mapping \mathcal{F}_{\theta_{1}^{*}} is unknown, and instead, an imperfect model given by \mathcal{F}_{\theta_{1}} , may be used, where \theta_{1} are the parameters of this imperfect dynamics mapping. We note that \mathcal{F}_{\theta_{1}} may be structurally different from \mathcal{F}_{\theta_{1}^{*}} (e.g., \mathcal{F}_{\theta_{1}} could be a neural network with learnable parameters \theta_{1} , where these parameters do not have a one-to-one correspondence with the true governing parameters \theta_{1}^{*} ). This imperfect specification of the dynamics induces forecast errors \xi_{t} that are assumed to be Gaussian distributed as specified in ( 1 ). One goal in this setup is to compute, for each t=1,\ldots,T, a state estimate x_{t} , referred to as an analysis , given previous and current observations y_{1:t} . This goal is achieved by the use of filtering algorithms in data assimilation, which often require specifying filtering parameters \phi prior to deployment that control, for instance, the “trust” given to the forecasts compared to the observations. A key idea underlying our framework is that information obtained from computing these analyses via filtering can be used to update the imperfect forecast model’s parameters \theta:=\{\theta_{1},\theta_{2}\} and

Read the complete text