Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows

Defne E. Ozan, Andrea Nóvoa, Luca Magri · 2025

Read Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows by Defne E. Ozan, Andrea Nóvoa, Luca Magri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16588.txt.

Introduction

Defne Ozan, Andrea Nóvoa, and Luca Magri teach a controller to tame chaotic flows it can only partly see in a paper from 2025. Turbulent flow control matters for energy and transport, yet chaos and huge state spaces defeat classic methods. Learning controllers need full state data, which real sensors rarely give.

Their framework feeds partial, noisy measures into an echo state network that predicts the flow. An ensemble Kalman filter cleans the state estimate in real time. An actor-critic learner then picks control moves from those estimates. The test bed was the Kuramoto-Sivashinsky equation, a standard chaos model. Success there suggests a path toward control of real turbulent systems with sparse sensors.

Catalog record

CREATORDefne E. Ozan, Andrea Nóvoa, Luca Magri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows Defne E. Ozan, Andrea Nóvoa, Luca Magri Abstract The goal of many applications in energy and transport sectors is to control turbulent flows. However, because of chaotic dynamics and high dimensionality, the control of turbulent flows is exceedingly difficult. Model-free reinforcement learning (RL) methods can discover optimal control policies by interacting with the environment, but they require full state information, which is often unavailable in experimental settings. We propose a data-assimilated model-based RL (DA-MBRL) framework for systems with partial observability and noisy measurements. Our framework employs a control-aware Echo State Network for data-driven prediction of the dynamics, and integrates data assimilation with an Ensemble Kalman Filter for real-time state estimation. An off-policy actor-critic algorithm is employed to learn optimal control strategies from state estimates. The framework is tested on the Kuramoto-Sivashinsky equation, demonstrating its effectiveness in stabilizing a spatiotemporally chaotic flow from noisy and partial measurements. 1 Introduction Turbulence arises in many engineering applications involving energy and transportation and has implications for efficiency and energy consumption of the system of interest. Closed-loop control has been key for taming turbulence for drag reduction of transport vehicles, lift increase of airfoils, efficiency increase of wind turbines, and mixing enhancement of combustors [ 1 ] . While these control methods are well established for linear systems, finding effective and robust control strategies in turbulent systems is challenging due to the nonlinear and chaotic nature of the dynamics. Furthermore, turbulence spans over multiple spatial and temporal scales, requiring high-dimensional state spaces and expensive computations, which makes it difficult for control to work in real time. These challenges have motivated the development of data-driven control methods. Reinforcement learning (RL) has been imported in the field of flow control because it offers a model-free framework that can learn optimal control policies directly from data. Recent advancements in deep RL have demonstrated success in learning policies for flow control tasks such as drag reduction and separation control e.g., [ 9 ] . However, because RL methods are typically Markov Decision Processes, they require access to the full state information, i.e., full observability. This poses a limitation in applications in which only a subset of the full state can be measured as they are Partially Observed Markov Decision Processes [ 6 ] . Previous approaches have addressed partial observability by incorporating a memory of the system’s measurements [ 13 ] , recurrent or attention-based architectures [ 12 ] . Data assimilation methods, specifically sequential assimilation methods, are commonly used to enable real-time predictions in numerical weather forecasting and fluid dynamics in partially-observed systems [ 3 ] . In this paper, we employ sequential data assimilation to integrate noisy and sparse measurements with a predictive model in order to estimate the full state of a controlled system. We develop an RL framework for chaotic flow control with the following objectives: (i) learn a data-driven model of the actuated dynamics of a chaotic flow, (ii) estimate the full state of the system in real time from partial, noisy observations using data assimilation, and (iii) learn an optimal control policy from the estimated state. This paper is organized as follows: Sec. 2 sets the control problem, Sec. 3 presents the proposed framework, Sec. 4 demonstrates it on the Kuramoto-Sivashinsky (KS) equation, and Sec. 5 concludes the paper. 2 Problem formulation We consider a deterministic discrete-time dynamical system of the form \bm{s}(t+1)=\bm{\mathcal{F}}(\bm{s}(t),\bm{a}(t)), (1) where \bm{s}(t)\in\mathbb{R}^{n_{s}} is the system’s state, \bm{a}(t)\in\mathbb{R}^{n_{a}} is the control action and \bm{\mathcal{F}}:\mathbb{R}^{n_{s}}\times\mathbb{R}^{n_{a}}\rightarrow\mathbb{R}^{n_{s}} is the operator that governs the evolution of the system’s dynamics in time. We focus on systems with chaotic dynamics, which are extremely sensitive to initial conditions making the system unpredictable in long time intervals. The system is Markovian, meaning that \bm{s}(t) and \bm{\mathcal{F}} encapsulate all necessary information to determine the next state, \bm{s}(t+1) . Our goal is to find the optimal set of actions, \bm{a}^{*}(t) to optimize a certain objective functional constrained by the system’s dynamics ( 1 ). In RL, this objective is formulated through a reward function, r(\bm{s},\bm{a})\in\mathbb{R} , which evaluates the immediate effect of an action. The agent, i.e., the controller, selects actions based on a state-feedback policy, i.e., \bm{a}(t)=\bm{\pi}(\bm{s}(t)) . The value function V^{\pi}(\bm{s})=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(\bm{s}(t),\bm{a}(t))\mid\bm{s}(0)=\bm{s}\right] (2) quantifies the expected return under policy \bm{\pi} , with discount factor \gamma\in[0,1] , which weighs the importance of future states. The agent is trained to learn the optimal policy \bm{\pi}^{*} that maximizes the associated value function. In practical applications, we acquire observations of the system’s state, which are noisy and incomplete. We model the observation process as \bm{o}(t)=\bm{\mathcal{M}}(\bm{s}(t))+\bm{\epsilon}(t), (3) where \bm{o}(t)\in\mathbb{R}^{n_{o}} is the observation, \bm{\epsilon}(t)\in\mathbb{R}^{n_{o}} is a zero-mean measurement noise, and \bm{\mathcal{M}}:\mathbb{R}^{n_{s}}\rightarrow\mathbb{R}^{n_{o}} is the observation operator. 3 Methodology To tackle control from partial observations, we propose a framework, data-assimilated model-based reinforcement learning (DA-MBRL), that has three components: (i) a predictive model of the system’s dynamics, (ii) an ensemble-based data assimilation method for real-time state estimation, and (iii) an off-policy actor-critic RL algorithm for learning the control policy. If the system’s equations are unknown or are too expensive for real-time simulation, a data-driven model can be utilized to learn and forecast the dynamics. For this purpose, we propose a control-aware formulation of the Echo State Network (ESN) [ 11 ] . The ESN [ 5 ] is a type of recurrent neural network (RNN) that is computationally cheap to train. To model actuated dynamics, we extend the ESN by incorporating control actions into the reservoir state update \bm{h}(t+1)=(1-\alpha)\bm{h}(t)+\alpha\tanh\big(\bm{W}_{in}[(\bm{y}_{in}(t)-\bar{\bm{y}})\odot\bm{g}_{y};\bm{a}(t)]+\bm{W}\bm{h}(t)\big), (4) where \bm{y}_{in}(t)\in\mathbb{R}^{n_{y}} is the input vector, \bm{h}(t)\in\mathbb{R}^{n_{h}} is the reservoir state, \bm{W}_{in}\in\mathbb{R}^{n_{h}\times(n_{y}+n_{a})} is the input matrix, and \bm{W}\in\mathbb{R}^{n_{h}\times n_{h}} is the state matrix. The input vector is standardized by the mean \bar{\bm{y}}\in\mathbb{R}^{n_{y}} and the scaling \bm{g}_{y}\in\mathbb{R}^{n_{y}} , g_{y,i}=1/\sigma_{y,i} with \sigma_{y,i} being the standard deviation of each y_{i},\;i=1,2,\dots,n_{y} . The time series prediction of the dynamics is enabled by the readout, which is defined as \bm{\hat{y}}(t+1)=\bm{W}_{out}[\tilde{\bm{h}}(t+1);1],\;\tilde{h}_{k}(t)=\begin{cases}h_{k}(t)&\text{if }k\text{ is odd},\\ h_{k}^{2}(t)&\text{if }k\text{ is even},\end{cases}\;k=1,2,\dots,n_{h}, (5) where \bm{y}\in\mathbb{R}^{n_{y}} is the target output vector, \bm{\hat{y}}\in\mathbb{R}^{n_{y}} is the predicted output vector, and \bm{W}_{out}\in\mathbb{R}^{n_{y}\times(n_{h}+1)} is the output matrix. We include the squared reservoir state variables in the readout to break the antisymmetry of \tanh and enable closed-loop prediction. The output weights \bm{W}_{out} are trained in open-loop configuration via ridge regression, whilst the input and state matrices \bm{W}_{in} and \bm{W} are sparse, randomly generated, and not trained. The input matrix is obtained by first generating a matrix \tilde{\bm{W}}_{in} such that each row contains only one nonzero element drawn from a uniform distribution \sim\mathcal{U}(-1,1) . The inputs are then scaled as \bm{W}_{in}=[\xi_{in}\tilde{\bm{W}}_{in}^{y}\quad\xi_{a}\tilde{\bm{W}}_{in}^{a}] , where \tilde{\bm{W}}_{in}^{y}\in\mathbb{R}^{n_{h}\times n_{y}} and \tilde{\bm{W}}_{in}^{a}\in\mathbb{R}^{n_{h}\times n_{a}} are the weights associated with the input and action vectors, respectively. The state matrix \bm{W} is scaled as \bm{W}=\rho\tilde{\bm{W}} , where \tilde{\bm{W}} is an Erdős-Renyi matrix with a given average number of nonzero elements per row (connectivity, n_{conn} ) drawn from a uniform distribution \sim\mathcal{U}(-1,1) scaled to have unity spectral radius, so that \rho defines the spectral radius of \bm{W} . The details of optimal training, hyperparameter selection and validation of ESNs can be found in [ 10 ] . In this paper, we train the ESN on the full state data from the system generated by random actuations, i.e., \bm{y}_{in}(t)=\bm{y}(t)=\bm{s}(t) . After training, the closed-loop prediction is

Read the complete text