PnP-DA: Towards Principled Plug-and-Play Integration of Variational Data Assimilation and Generative Models
Yongquan Qu, Matthieu Blanke, Sara Shamekh, Pierre Gentine · 2025
Read PnP-DA: Towards Principled Plug-and-Play Integration of Variational Data Assimilation and Generative Models by Yongquan Qu, Matthieu Blanke, Sara Shamekh, Pierre Gentine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00325.txt.
Introduction
Yongquan Qu, Matthieu Blanke, Sara Shamekh, and Pierre Gentine merge data assimilation with generative models in a plug-and-play method for Earth forecasts, in a research paper from 2025. Forecast models drift as errors pile up, and blending in observations is the standard cure. Classic blending assumes bell-curve errors, a poor fit for chaotic weather.
Their method alternates two steps. A light gradient update pulls the forecast toward fresh observations. Then a trained generative model, conditioned on that forecast, redraws a realistic state in one pass. No gradients cross the big network, keeping the cycle cheap. Tests on chaotic benchmarks show the pairing beats classic assumptions.
Catalog record
| CREATOR | Yongquan Qu, Matthieu Blanke, Sara Shamekh, Pierre Gentine |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PnP-DA: Towards Principled Plug-and-Play Integration of Variational Data Assimilation and Generative Models Yongquan Qu, Matthieu Blanke, Sara Shamekh, Pierre Gentine Abstract Earth system modeling presents a fundamental challenge in scientific computing: capturing complex, multiscale nonlinear dynamics in computationally efficient models while minimizing forecast errors caused by necessary simplifications. Even the most powerful AI- or physics-based forecast system suffer from gradual error accumulation. Data assimilation (DA) aims to mitigate these errors by optimally blending (noisy) observations with prior model forecasts, but conventional variational methods often assume Gaussian error statistics that fail to capture the true, non-Gaussian behavior of chaotic dynamical systems. We propose PnP - DA , a Plug - and - Play algorithm that alternates (1) a lightweight, gradient - based analysis update—using a Mahalanobis-distance misfit on new observations—with (2) a single forward pass through a pretrained generative prior conditioned on the background forecast via a conditional Wasserstein coupling. This strategy relaxes restrictive statistical assumptions and leverages rich historical data without requiring an explicit regularization functional, and it also avoids the need to backpropagate gradients through the complex neural network that encodes the prior during assimilation cycles. Experiments on standard chaotic testbeds demonstrate that this strategy consistently reduces forecast errors across a range of observation sparsities and noise levels, outperforming classical variational methods. 1 Introduction Earth system modeling presents one of the most computationally demanding challenges in scientific computing, requiring simulation of complex physical processes across multiple spatial and temporal scales [ 29 ] . A fundamental limitation in this field is that computational constraints and incomplete physical understanding necessitate the use of simplified physical models that cannot fully resolve all scales. This scale disparity introduces systematic errors that propagate through the forecasts and diminish the prediction accuracy [ 30 ] . Data assimilation (DA) forms the foundation of modern weather forecasting and Earth system prediction, addressing the inverse problem of estimating the current state from sparse, noisy observations and prior prediction (called background) coming from the imperfect model [ 7 , 10 ] for a better initial condition (called analysis) of subsequent forecast (see Figure 1 ). It can also be used to perform parameter estimation to mitigate model misspecifications [ 12 , 34 ] . Figure 1 : Detailed sketch of data assimilation. While machine learning (ML) approaches have begun transforming the forecasting component of weather prediction [ 4 , 32 ] , their application to the underlying DA problem remains an active research area with substantial untapped potential [ 46 ] . Recent work has advanced the integration of DA with modern ML tools through differentiable numerical solvers or differentiable surrogates that facilitate gradient computation within variational DA frameworks [ 15 , 36 , 34 , 41 , 45 , 14 ] . Other researchers have addressed the probabilistic formulation of DA using modern generative models, for example, diffusion model [ 19 , 16 , 35 , 26 , 6 ] , while some have focused on developing novel filtering and optimization approaches with ML [ 3 , 5 , 31 ] . Traditional variational DA methods assume Gaussian error statistics for model errors. This assumption makes the assimilation problem tractable but blind to the non - Gaussian, state - dependent errors in chaotic flows. Some state-of-the-art strategies use ML for variational DA to address this limitation through two primary approaches: mapping background errors into a near Gaussian latent space via variational autoencoders [ 45 ] or mapping the physical state(s) into a latent space with near diagonal covariance structure [ 14 ] , both leveraging autoencoding procedures. While these methods have demonstrated great empirical performance with ML surrogates for atmospheric modeling, each has limitations. The former faces challenges because the change of measure introduces a costly Jacobian log-determinant which is both expensive to compute and numerically unstable in arbitrary deep networks. As a simplification, the Jacobian log-determinant term is therefore omitted for numerical stability in [ 45 ] . Consequently, high-density regions of the true posterior—where the decoder locally compresses latent volume—are no longer preferentially weighted. The resulting MAP estimate still fits the observations but may systematically mislocate modes relative to the correct Bayesian push-forward, and no longer corresponds to a true maximum-a-posteriori estimate under the intended generative model. By performing variational DA directly in a latent space representing the physical states—rather than transforming the error statistics—the approach avoids the need to compute an intractable Jacobian log-determinant; however, projection back to the physical space inevitably introduces reconstruction error arising from the reduced-order representation. Both of these algorithms require additional gradient backpropagation through the highly nonlinear neural network during optimization iterations. Building on this review of ML-enhanced variational DA, our approach departs fundamentally from latent-space autoencoding strategies. We aim to develop a more powerful background-state prior that can fully leverage the extensive record of historical forecasts and reanalysis data. We also seek to keep the analysis step numerically light by avoiding intractable Jacobian corrections or additional gradient backpropagation through learned autoencoders. Furthermore, instead of training models to simply match reanalysis data in an end-to-end fashion, our goal is to optimize them to perform well on future, unseen observations. The Plug-and-Play (PnP) framework offers an established approach for solving inverse problems by integrating powerful learned priors directly into the optimization process, without requiring an explicit regularization functional [ 43 ] . Recent advances in generative modeling have introduced expressive priors that naturally fit within this framework [ 48 , 44 , 28 ] . Data assimilation (DA) is fundamentally an ill-posed inverse problem, where the goal is to infer the true system state from incomplete and noisy observations. Variational DA addresses this by imposing a Gaussian prior centered on a background forecast to regularize the solution. In contrast, PnP methods with generative priors offer a more flexible alternative: instead of relying on simplified error statistics, they leverage rich, data-driven priors learned from historical data to constrain the solution space, improving robustness and realism in ill-posed settings. However, while these methods show promise for static inverse problems like image reconstruction, DA presents unique challenges. The solution to the inverse problem becomes an initial condition for a potentially chaotic dynamical system simulation, where visual quality doesn’t guarantee prediction accuracy or even numerical stability. More importantly, unlike traditional inverse problems, where a known degradation operator maps the true signal to the observed one, there is no explicit or well-defined mapping from the true physical state to the background forecast in data assimilation. This complicates the direct application of standard PnP methods, which typically rely on such forward models to guide optimization. In this paper, we propose a novel data-assimilation framework based on a Plug-and-Play algorithm that combines a variational DA update on incoming observations with a flow-matching prior on the background forecast, drawing inspiration from PnP-Flow [ 28 ] . Unlike the standard optimal transport (OT) flow matching used in PnP-Flow, we use the OT Bayesian flow matching [ 8 ] as the generative prior. This approach conditions the flow on the background forecast field by incorporating a conditional Wasserstein cost into the optimal transport coupling. This cost strongly penalizes deviations from the background, ensuring that the learned velocity fields serve as a Bayesian-optimal correction—specifically, they minimize the expected squared transport