Reinforcement Learning, Optimal Control, and Bayesian Filtering in Data Assimilation

Abed Hammoud · 2026

Read Reinforcement Learning, Optimal Control, and Bayesian Filtering in Data Assimilation by Abed Hammoud (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12158.txt.

Introduction

Hammoud gives one variational frame joining Kalman-type methods, KL-regularized control, Bayesian filtering and smoothing, and variational data assimilation. He proves identities showing the evidence is the global infimum and the true posteriors are the unique minimizers when they belong to the allowed classes. The frame clarifies targets often mixed up. Strong- and weak-constraint 4D-Var count as MAP estimators under Gaussian assumptions, and the ensemble Kalman filter is exact only in the limit of linear-Gaussian dynamics with infinite ensembles.

Catalog record

CREATORAbed Hammoud
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78538
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12158

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reinforcement Learning, Optimal Control, and Bayesian Filtering in Data Assimilation Abed Hammoud Abstract We give a finite-horizon variational formulation that places Bayesian filtering and smoothing, variational data assimilation, KL-regularized control, and Kalman-type methods inside one mathematically explicit hierarchy. For a discrete-time hidden Markov model and any admissible one-step candidate law q t q_{t} , we prove 𝒥 t ( q t ) = 𝔼 q t [ − log p ( y t ∣ X t ) ] + KL ( q t ∥ p t f ) = KL ( q t ∥ p t a ) − log p ( y t ∣ y 0 : t − 1 ) \mathcal{J}_{t}(q_{t})=\mathbb{E}_{q_{t}}\!\left[-\log p(y_{t}\mid X_{t})\right]+\operatorname{KL}(q_{t}\|p_{t}^{f})=\operatorname{KL}(q_{t}\|p_{t}^{a})-\log p(y_{t}\mid y_{0:t-1}) , and, for any admissible path law q q , 𝒥 path ( q ) = 𝔼 q [ − ∑ t = 0 T log p ( y t ∣ X t ) ] + KL ( q ∥ p ( x 0 : T ) ) = KL ( q ∥ p ( x 0 : T ∣ y 0 : T ) ) − log p ( y 0 : T ) \mathcal{J}_{\mathrm{path}}(q)=\mathbb{E}_{q}\!\left[-\sum_{t=0}^{T}\log p(y_{t}\mid X_{t})\right]+\operatorname{KL}(q\|p(x_{0:T}))=\operatorname{KL}(q\|p(x_{0:T}\mid y_{0:T}))-\log p(y_{0:T}) . These identities determine the evidence as the global infimum and make the analysis and smoothing posteriors the unique minimizers whenever those posterior laws belong to the admissible classes. This separates targets that are often conflated: strong- and weak-constraint 4D-Var are MAP estimators under the stated Gaussian assumptions; KL-regularized control recovers the Bayesian posterior only when the passive dynamics, likelihood cost, temperature, and a restrictive representability condition on the policy class are all matched correctly; and the linear-Gaussian specialization yields the Kalman analysis exactly. The ensemble Kalman filter then appears as a Gaussian and finite-ensemble approximation to the forecast-to-analysis map, exact only in the linear-Gaussian infinite-ensemble limit. This framework also clarifies RMSE-based RL data assimilation: such rewards may define effective estimators or pseudo-posteriors, but not exact posterior recovery unless they realize the likelihood-plus-KL objective. Keywords. data assimilation, Bayesian filtering, variational data assimilation, KL-regularized control, ensemble Kalman filtering MSC2020. 62F15, 62M20, 93E11, 93E20, 49L20 1 Introduction Data assimilation combines a dynamical model with partial and noisy observations in order to estimate the state of a system. In geophysical applications the problem is sequential, high dimensional, and often nonlinear. Three mathematical languages dominate the subject. The first is Bayesian filtering and smoothing , where the target is a posterior law conditioned on data. The second is variational data assimilation , where one minimizes a deterministic mismatch functional over states or trajectories. The third is the Kalman and ensemble Kalman language, where one propagates means, covariances, and their ensemble approximations through forecast and analysis updates. These languages are closely related, but they are not interchangeable, and reinforcement-learning terminology can make the distinctions even harder to state precisely. Historically, variational data assimilation emerged from optimal-control ideas and adjoint equations in meteorology [ 8 , 15 ] , while Bayesian interpretations of numerical weather prediction analysis were made explicit by Lorenc [ 10 ] . Ensemble Kalman methods turned Gaussian filtering formulas into practical ensemble algorithms for large systems [ 1 , 22 , 16 , 2 ] . Particle-filter viewpoints retained the goal of approximating the full nonlinear posterior while also clarifying the high-dimensional degeneracy barrier in geoscience data assimilation [ 3 , 21 ] . On a different track, linearly solvable Markov decision processes, KL-control, and control-as-inference showed that entropy-regularized control objectives can be rewritten as inference or KL minimization problems [ 20 , 17 , 18 , 19 , 6 , 12 , 9 ] . Controlled interacting particle systems, feedback particle filters, and Schrödinger or optimal-transport viewpoints supplied further bridges between filtering, control, and data assimilation [ 23 , 14 , 13 ] . Recent reinforcement-learning-based data assimilation papers, including Hammoud et al. [ 4 ] , make these bridges practically salient, but also increase the temptation to transfer posterior-exact language to objectives that do not actually compute posteriors. Our goal is conceptual rather than algorithmic. We do not propose a new approximate filter, prove finite-sample optimality of EnKF-type methods, or claim posterior recovery for generic RL rewards. Instead, we give a mathematically explicit synthesis of the exact finite-horizon statements that survive across these literatures and state explicitly where the usual slogans stop being theorems. The contributions are the following. 1. Posterior-as-variational theorems. We prove exact one-step and path-space identities for hidden Markov models, theorems 4.2 and 4.5 , and then state the corresponding unique-minimizer conclusions under explicit posterior-admissibility assumptions. 2. 4D-Var as MAP, not posterior-law optimization. Under the stated Gaussian assumptions, strong- and weak-constraint 4D-Var are negative-log-posterior minimization problems and therefore compute MAP estimators rather than posterior means or posterior laws; see theorems 3.1 and 3.3 . 3. A mathematically honest control-as-inference statement. KL-regularized control recovers the Bayesian posterior only when the passive dynamics, likelihood cost, temperature, and a restrictive representability condition on the admissible policy class are all matched correctly; see theorems 5.1 , 5.5 , and 5.9 . Outside that calibrated setting one obtains either a different Gibbs law or a restricted-family projection. 4. Kalman exactness and EnKF approximation. In the linear-Gaussian case the unrestricted optimizer is the Kalman analysis posterior, and Gaussian minimization reproduces the classical mean and covariance formulas; see theorems 6.2 and 6.3 . EnKF-type methods then appear as Gaussian and finite-ensemble approximations to the forecast-to-analysis map, not as generic exact posterior solvers at finite ensemble size; see section 7 . 5. Clarification for RMSE-based RL data assimilation. RMSE-style rewards can define useful estimation or control objectives, but they do not imply exact posterior recovery unless they realize the same likelihood-plus-KL structure as the Bayesian objective; see theorems 5.7 and 8 . The backbone of the paper is the pair of identities proved in theorems 4.2 and 4.5 . The one-step filtering identity reads \mathcal{J}_{t}(q_{t})=\mathbb{E}_{q_{t}}\!\left[-\log p(y_{t}\mid X_{t})\right]+\operatorname{KL}(q_{t}\|p_{t}^{f})=\operatorname{KL}(q_{t}\|p_{t}^{a})-\log p(y_{t}\mid y_{0:t-1}), for every admissible candidate analysis law q_{t} , while the path-space analogue is \mathcal{J}_{\mathrm{path}}(q)=\mathbb{E}_{q}\!\left[-\sum_{t=0}^{T}\log p(y_{t}\mid X_{t})\right]+\operatorname{KL}(q\|p(x_{0:T}))=\operatorname{KL}(q\|p(x_{0:T}\mid y_{0:T}))-\log p(y_{0:T}). These identities identify the evidence as the variational infimum and make the posterior the unique minimizer whenever the posterior itself belongs to the admissible family. Restricting that family then recovers familiar approximations: MAP in the point-mass or zero-variance limit, Gaussian variational approximations in Gaussian families, and ensemble approximations when the posterior update is replaced by sample-based Gaussian surrogates. Core notation is summarized in table 1 . Four qualifications are essential for the rest of the paper. First, the minimizer statements in sections 4 and 5 require posterior or Gibbs admissibility in addition to mere normalization of the evidence. Second, exact posterior recovery by a policy class is a reachability statement on state-action laws and is therefore not automatic under fixed transition kernels. Third, EnKF exactness means exactness only in the linear-Gaussian infinite-ensemble regime; finite-ensemble stochastic and deterministic square-root filters remain Gaussian or ensemble approximations. Fourth, except for the deterministic strong-constraint formulation in section 3 , the main theorems are written in density form for finite-dimensional Euclidean state spaces. The results below are intentionally finite-horizon and discrete-time. This keeps the main theorems exact and transparent, while still covering the standard filtering, smoothing, 4D-Var, Kalman, and EnKF settings used in data assimilation. Sections

Read the complete text