A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems

Jingqiao Tang, Ryan Bausback, Feng Bao, Guannan Zhang, Phuoc-Toan Huynh · 2026

Read A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems by Jingqiao Tang, Ryan Bausback, Feng Bao, Guannan Zhang, Phuoc-Toan Huynh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14863.txt.

Introduction

Tang and co-authors fold a score filter into machine learning forecasts of dynamical systems. Learned forecasters drift as model error piles up over time. Data assimilation can pull forecasts back toward observations. The Ensemble Score Filter, a diffusion-based method, does that job here. Training and filtering alternate to refine the model in flight. High-dimensional nonlinear systems stay in scope. Lorenz-96 and Korteweg-de Vries tests show the hybrid at work. Long-term forecast uncertainty drops under the filter, they report.

Catalog record

CREATORJingqiao Tang, Ryan Bausback, Feng Bao, Guannan Zhang, Phuoc-Toan Huynh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57446
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14863

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems Jingqiao Tang, Ryan Bausback, Feng Bao, Guannan Zhang, Phuoc-Toan Huynh Abstract We introduce a score-filter-enhanced data assimilation framework designed to reduce predictive uncertainty in machine learning (ML) models for data-driven dynamical system forecasting. Machine learning serves as an efficient numerical model for predicting dynamical systems. However, even with sufficient data, model uncertainty remains and accumulates over time, causing the long-term performance of ML models to deteriorate. To overcome this difficulty, we integrate data assimilation techniques into the training process to iteratively refine the model predictions by incorporating observational information. Specifically, we apply the Ensemble Score Filter (EnSF), a generative AI-based training-free diffusion model approach, for solving the data assimilation problem in high-dimensional nonlinear complex systems. This leads to a hybrid data assimilation–training framework that combines ML with EnSF to improve long-term predictive performance. We Shall demonstrate that EnSF-enhanced ML can effectively reduce predictive uncertainty in ML-based Lorenz–96 system prediction and the Korteweg–De Vries (KdV) equation prediction. 1 Introduction Machine learning (ML) has emerged as a powerful tool across a wide range of impactful domains, including computer vision [ 17 , 63 , 71 ] , natural language processing [ 12 , 16 , 67 ] , and weather forecasting [ 1 , 13 , 34 ] . More recently, neural network (NN)-based approaches have been developed as surrogate models for approximating and accelerating the simulation of dynamical systems governed by differential equations [ 44 , 59 , 60 ] . Although a complete theoretical understanding remains an open question, these approaches can offer practical advantages over traditional solvers in some regimes. Firstly, once trained, they can provide rapid evaluations and can provide predictions at arbitrary spatial locations without extra interpolations. Secondly, for certain high-dimensional problems, learning-based surrogates can reduce the practical burden of spatial discretizations and thereby alleviate aspects of the curse of dimensionality. However, despite numerous advantages of ML models, significant challenges remain. While such models often demonstrate strong short-term predictive accuracy, they frequently struggle to reliably capture long-term dynamics, particularly in chaotic systems [ 69 ] . Without additional accurate inputs (e.g., new observational data for calibration) or continued retraining, these models can fail to generalize reliably to regimes far from the training distribution [ 66 ] . For instance, the “Aardvark Weather” model in [ 1 ] demonstrated strong forecasting skill near the initial time but exhibited progressively degraded performance at longer lead times. Similarly, the DSDL framework for chaotic dynamics introduced in [ 82 ] accurately tracks the Lorenz–63 system over short horizons but shows substantial divergence from the true trajectory during extended time integration. Another major challenge concerns the availability and quality of training data, which can significantly limit the development and performance of machine learning solvers. In practical applications, datasets are often limited in size, unevenly distributed, and sparsely sampled in space and time. Moreover, the available observations may only indirectly reflect the quantities of interest. In addition, training data are frequently affected by measurement noise and other sources of uncertainty. Such issues arise routinely in applications including weather and climate prediction, subsurface flow and transport, and ocean modeling, where data acquisition is costly and inherently uneven. As a consequence, even a well-trained surrogate model may inherit biases and systematic errors from the training data, ultimately leading to unreliable or unstable forecasts. To mitigate error accumulation in long-term forecasts and reduce biases arising from imperfect training data, one common strategy is to collect additional measurements to recalibrate ML-based predictions. In practice, however, online measurements and observational data acquired during the forecasting phase are often partial, irregular, and contaminated by noise, making them insufficient to support reliable retraining. Moreover, repeatedly updating the model can be computationally expensive and may even degrade performance when the newly acquired data are limited or biased. These challenges motivate our alternative strategy that incorporates imperfect measurements directly into the prediction process. Instead of modifying the trained model, this approach updates the state estimate and its associated uncertainty online using observational information. In this paper, we introduce a “data assimilation” framework that integrates model predictions with observational data to reduce uncertainty and enhance the accuracy of machine learning (ML)-based forecasts. The underlying mathematical formulation of data assimilation is the optimal filtering problem, which seeks the best estimate of the target system state conditioned on the available observations. In particular, the optimal state estimate is characterized by the conditional expectation of the state variable given the observational information. The standard approach for solving the optimal filtering problem is the recursive Bayesian filter, which consists of the Kalman filter [ 14 , 25 , 45 ] and the particle filter [ 3 , 65 , 70 , 2 , 29 , 46 ] . However, Kalman-type filters are often effective for high-dimensional problems, but they rely on linear or weakly nonlinear assumptions, which can limit their applicability in strongly nonlinear settings [ 25 , 42 ] . On the other hands, particle filters are, in principle, applicable to nonlinear and non-Gaussian problems, but they often suffer from severe weight degeneracy and the curse of dimensionality in high-dimensional state spaces because they are based on sequential Monte Carlo sampling [ 41 , 74 ] . To overcome the limitations of conventional methods, we adopt a score filter framework inspired by score-based diffusion models [ 77 ] , namely the Ensemble Score Filter (EnSF) [ 10 , 11 ] . The EnSF aims to estimate the conditional probability density of the target model state, referred to here as the filtering density, and encodes its information in the corresponding score function. This score function then serves as a guiding term in a reverse-time diffusion process that steers the generative dynamics to produce samples consistent with the target distribution. Leveraging the strong expressive and generative capabilities of diffusion models [ 57 , 78 ] , extensive studies have demonstrated that EnSF can effectively track stochastic nonlinear dynamical systems in very high-dimensional spaces with high accuracy and computational efficiency [ 11 , 9 , 5 , 55 ] . Building on its strong capability to filter uncertainties in predictive models, we integrate the EnSF framework with ML models to improve the predictive accuracy of dynamical systems learned from data. In this work, we consider two representative ML frameworks to demonstrate the performance of the EnSF-enhanced prediction: the long short-term memory (LSTM) network and operator-learning models. The LSTM network, a gated recurrent neural network (RNN) designed for sequential and time-series modeling, naturally provides a step-by-step state transition map that propagates the system state forward in time. Its gating mechanism enables the network to capture long-term temporal dependencies while alleviating vanishing gradient issues, making it well suited for modeling dynamical systems. In contrast, operator-learning models aim to learn mappings between function spaces and are typically trained to predict solutions over an entire spatiotemporal domain in a single forward pass. While effective for approximating complex dynamical systems, this global-in-time prediction structure does not naturally provide a recursive state-transition mechanism for time evolution. To match step-by-step state propagation, we adopt RNN-based architectures that advance the system state sequentially in time [ 28 , 31 , 56 ] . The rest of

Read the complete text