Enabling High-Accuracy Data Assimilation with Limited Ensembles via Machine Learning-Based Covariance Correction Journal: Computer Methods in Applied Mechanics and Engineering

Z. Yao, Z. Li, L. Zhao, Z. Liu, Z. Lu, S. Kim, G. Wang · 2026

Read Enabling High-Accuracy Data Assimilation with Limited Ensembles via Machine Learning-Based Covariance Correction Journal: Computer Methods in Applied Mechanics and Engineering by Z. Yao, Z. Li, L. Zhao, Z. Liu, Z. Lu, S. Kim, G. Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11639.txt.

Introduction

Yao and six colleagues present a machine learning-based ensemble Kalman filter for high-accuracy data assimilation with limited ensembles. Data assimilation merges forecasts with observations for state estimation, but ensemble methods depend heavily on ensemble size, trading accuracy against cost. Their multilayer perceptron predicts the covariance gap between a limited ensemble and a large one. That gap enters the filter through element-wise scaling, yielding an amended covariance matching true uncertainty better. Tests on Lorenz-63 and Lorenz-96 beat standard EnKF at the same limited size with higher accuracy.

Catalog record

CREATORZ. Yao, Z. Li, L. Zhao, Z. Liu, Z. Lu, S. Kim, G. Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31208
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11639

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enabling High-Accuracy Data Assimilation with Limited Ensembles via Machine Learning-Based Covariance Correction Journal: Computer Methods in Applied Mechanics and Engineering Z. Yao, Z. Li, L. Zhao, Z. Liu, Z. Lu, S. Kim, G. Wang Abstract Data assimilation (DA) integrates numerical model forecasts with observations to achieve the optimal state estimation. Ensemble-based methods, such as the ensemble Kalman filter (EnKF), are widely used for state estimation for high-dimensional and nonlinear dynamic systems. However, their performance strongly depends on the ensemble size, therefore causing a tradeoff problem between analysis accuracy and computational cost. To address this problem, this study presents a machine learning-based EnKF framework that maintains high accuracy with a relatively small ensemble size. Specifically, a multilayer perceptron (MLP) function is built to predict the difference between the forecast error covariances estimated from a limited ensemble and a sufficiently large ensemble, with the latter being assumed to be an accurate approximation of the underlying truth. This predicted covariance difference term is then incorporated into the EnKF algorithm via an element-wise scaling strategy, resulting in an amended forecast covariance matrix that better approximates the true uncertainty level and sequentially produces more accurate analysis results. To demonstrate the feasibility and robustness of the proposed algorithm, we perform a set of numerical experiments with the Lorenz-63 and Lorenz-96 systems under various configurations, and the results consistently indicate that the proposed algorithm can significantly outperform the standard EnKF with the same limited ensemble size, by achieving notably higher analysis accuracy while remaining computationally efficient. This approach provides a practical and feasible pathway to accurate and computationally efficient data assimilation for high-dimensional and nonlinear dynamic systems. 1 Introduction Data assimilation (DA) is a mathematical framework that incorporates the observations into numerical models to obtain an optimal state estimation via Bayesian inference ( 21 ) . Over the past few decades, DA methods have proven to be indispensable in diverse fields, including but not limited to weather forecasting ( 20 ; 23 ) , physical oceanography ( 1 ; 17 ; 18 ; 28 ; 29 ) , turbulence modeling ( 8 ; 32 ) , and material modeling ( 14 ; 19 ) . Among the various DA approaches, ensemble-based methods, particularly the ensemble Kalman filter (EnKF) ( 9 ; 10 ) , have gained widespread popularity due to their inherent natural ability to handle nonlinear systems and “plug-and-play” architecture. However, ensemble-based DA methods often face a practical and challenging dilemma, i.e., better accuracy usually comes at the price of higher computational cost ( 22 ; 30 ) . Specifically, accurate state estimation usually relies on a sufficiently large ensemble to properly characterize the true but usually poorly known forecast error covariances, i.e., uncertainty level; and a limited ensemble size may cause the rank-deficiency issues, which are usually manifested as spurious correlations and underestimation. However, the computational cost scales nearly linearly with the ensemble size, making large-ensemble DA impractical for high-dimensional dynamic systems. Two ad hoc remedies, including localization ( 4 ) and inflation ( 15 ) , have been proposed and widely applied to address these issues. However, both methods introduce additional parameters that often require careful and time-consuming tuning ( 7 ) , which therefore cannot effectively balance the analysis accuracy and computational efficiency. In recent years, machine learning (ML) has emerged as a promising tool to address the challenges faced by ensemble-based DA methods. Current efforts can be generally categorized into three types. The first type focuses on using neural networks (NNs) as surrogate models to accelerate the forecast step, i.e., replacing traditional computationally expensive models with learned functions ( 3 ; 26 ) . Although computationally efficient, such surrogates often struggle with generalization to out-of-distribution scenarios. The second category applies ML to enhance the analysis step, mainly by learning the update formula to handle missing or low-quality data ( 2 ; 31 ) . However, this type of methods may experience performance degradation in the scenario of low DA frequency, as shown by 16 . The third type, which is also the one most relevant to uncertainty quantification (UQ), employs ML to improve the representation of forecast uncertainty, either by ameliorating the forecast ensemble members or correcting deficiencies of the preliminary covariance estimations. This direction is especially meaningful and promising for operational DA, where the uncertainty level information is essential. For instance, 13 and 12 demonstrate that NNs can be utilized to predict ensemble spreads or spatiotemporal uncertainty maps with deterministic inputs, thereby significantly reducing the computational cost of running full ensemble systems. Similarly, 25 evaluate the capacity of different NNs to identify uncertainty sources, such as imperfect initial conditions and model parameters, using both direct (ensemble-based) and indirect (analysis-based) training strategies. Building on this, 24 propose one algorithm, which leverages convolutional neural networks to realize the online estimation of forecast error covariances from a single model integration for sequential DA. Despite these advances, two limitations still remain outstanding. First, these methods usually rely on the Gaussian assumption for error distributions, which may not always hold for highly nonlinear or chaotic systems and can lead to mis-calibrated uncertainty estimates. Second, most of the existing methods are not designed to handle low-frequency DA scenarios, where the forecast error covariance can vary significantly between assimilation cycles. This limitation can reduce their reliability in practical application situations with temporally sparse observation updates. In this study, we propose a novel machine learning-based covariance correction framework for the ensemble Kalman filter. In contrast to the existing approaches that get rid of the ensemble completely, our method retains a relatively small ensemble while employing a multilayer perceptron (MLP) to predict a correction term, which is then used to refine the preliminary estimated covariances. By incorporating this ML-predicted correction into the standard EnKF workflow via an element-wise scaling strategy, we aim to maintain the flow-dependent nature of the uncertainty while significantly mitigating the errors induced by the limited ensemble size. We demonstrate the feasibility and robustness of this approach through a set of numerical experiments based on Lorenz-63 and Lorenz-96 systems under various configurations, showing that it can significantly improve the overall accuracy while preserving computational efficiency. The paper is organized as follows. \lx@sectionsign 2 presents the methodological framework, including the EnKF formulation, the ML-based covariance correction strategy, and the complete assimilation algorithm. \lx@sectionsign 3 describes the experimental setup and presents the results of Lorenz-63 and Lorenz-96 systems, as well as the comprehensive sensitivity analyses. Finally, \lx@sectionsign 4 briefly summarizes this work. 2 Methods 2.1 Overview of the Proposed Framework The proposed data assimilation framework, which integrates the EnKF with a ML-based covariance correction function (named as EnKF-MLC), is illustrated in Fig. 1 . The simulation is initialized with a relatively small ensemble (size \mathfrak{N} ) of initial conditions, which are randomly sampled from a presumed distribution. Then the forecast step is performed for each ensemble member by running the forward integration model \mathcal{M} , which results in the forecast ensemble at the first DA time instant t=t_{1} . However, the forecast error covariance matrix ( \bm{P}^{\mathfrak{N}}_{f,1} ) estimated from this small ensemble may be inaccurate, exhibiting either underestimation or spurious correlations and potentially leading to suboptimal analysis or even filter divergence. To address this issue, a ML-based function

Read the complete text