Sample-Efficient Reinforcement Learning of Koopman eNMPC

Daniel Mayfrank · 2025

Read Sample-Efficient Reinforcement Learning of Koopman eNMPC by Daniel Mayfrank (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18787.txt.

Introduction

Daniel Mayfrank and co-authors link reinforcement learning with a data-driven control method in two thousand twenty-five. The goal is a controller that plans ahead for best cost while using few costly trials to learn. Such tuning can adjust the model or the bounds inside the plan itself.

Their setup turns a Koopman-based controller into a policy whose parts can be tuned by gradient steps. It is tested on a stirred-tank reactor drawn from past work. The method beat both plain identified models and neural network controllers on control quality and on how few samples it needed. Adding partial knowledge of the physics helped it learn faster still.

Catalog record

CREATORDaniel Mayfrank
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18787

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sample-Efficient Reinforcement Learning of Koopman eNMPC Daniel Mayfrank Abstract a Forschungszentrum Jülich GmbH, Institute of Climate and Energy Systems, Energy Systems Engineering (ICE-1), Jülich 52425, Germany b RWTH Aachen University, Aachen 52062, Germany c JARA-ENERGY, Jülich 52425, Germany d RWTH Aachen University, Process Systems Engineering (AVT.SVT), Aachen 52074, Germany Reinforcement learning (RL) can be used to tune data-driven (economic) nonlinear model predictive controllers ((e)NMPCs) for optimal performance in a specific control task by optimizing the dynamic model or parameters in the policy’s objective function or constraints, such as state bounds. However, the sample efficiency of RL is crucial, and to improve it, we combine a model-based RL algorithm with our published method that turns Koopman (e)NMPCs into automatically differentiable policies. We apply our approach to an eNMPC case study of a continuous stirred-tank reactor (CSTR) model from the literature. The approach outperforms benchmark methods, i.e., data-driven eNMPCs using models based on system identification without further RL tuning of the resulting policy, and neural network controllers trained with model-based RL, by achieving superior control performance and higher sample efficiency. Furthermore, utilizing partial prior knowledge about the system dynamics via physics-informed learning further increases sample efficiency. Keywords: Economic model predictive control; Koopman; Model-based reinforcement learning; Policy optimization; Process control; Physics-informed neural networks; Demand response 1 Introduction Model predictive control (MPC) and its variants, e.g., economic nonlinear MPC (eNMPC), rely on dynamic models that (i) are accurate and (ii) lead to optimal control problems (OCPs) which are solvable in real-time. In process systems engineering, obtaining mechanistic models that fulfill these requirements can be difficult due to large scales, unknown parameters, and nonlinearities ( Tang and Daoutidis, (2022) ). As an alternative to mechanistic models, data-driven approaches can be employed. These approaches are typically based on system identification (SI) using either historical operating data or data generated with a mechanistic model of the physical system ( Tang and Daoutidis, (2022) ). Alternatively, using reinforcement learning (RL) methods, data-driven (eN)MPCs can be trained for optimal performance in specific control tasks (see Fig. 1(a) ), which may produce superior control performance compared to SI (see, e.g., Chen et al., (2019) ; Gros and Zanon, (2019) ; Mayfrank et al., 2024b () ; Mayfrank et al., 2024a () ). (a) (b) (c) Figure 1 : (a) Procedure for RL-based training of an (eN)MPC. (b) A differentiable eNMPC policy parameterized by the parameters \bm{\theta} of the dynamic model; takes as input the current state \bm{x}_{t} and computes the optimal control action \bm{u}^{*}_{t} based on the minimization of a cost function f , subject to inequality constraints \bm{g} , and the learnable dynamic model \bm{h_{\theta}} . (c) Differentiable eNMPC policy with parameterized inequality constraints \bm{g} , e.g., state bounds. Training this policy leaves the underlying dynamic model \bm{h} unchanged but adapts the inequality constraints to counteract model-plant mismatch. To this end, a differentiable (e)NMPC policy is constructed, wherein the learnable parameters can be parameters of the dynamic model (see Fig. 1(b) , e.g., Mayfrank et al., 2024b () ; Mayfrank et al., 2024a () ), or other parameters that appear in the objective function or constraints of the (e)NMPC ( Gros and Zanon, (2019) ; Brandner et al., (2023) ; Brandner and Lucia, (2024) ), e.g., the state bounds (see Fig. 1(c) ). The latter approach optimizes the policy by compensating for model errors, e.g., via bounds adaptation. Thereby, RL-based training/refinement of a highly parameterized dynamic model such as an artificial neural network can be avoided by optimizing few (interpretable) parameters, e.g., parameters that modify the state bounds. Therefore, this approach may lead to better convergence, compared to RL-based (re)training of the dynamic model itself. However, even positing good training convergence, the model-free 1 1 1 Due to the multitude of ways in which models can be used in RL, the distinction between model-free RL, model-based RL, and other control approaches is not consistent across the literature. This distinction might be especially confusing in the context of the present work since we study the training of eNMPC policies, i.e., inherently model-based policies. However, the distinction between model-free and model-based, which is most relevant to our work, is about whether the training algorithm uses an additional learned model of the environment to train the policy. Please refer to Section 2.1 for the exact definitions that we use in this work. RL algorithms that have so far been used for RL-based training of (eN)MPCs remain notoriously sample inefficient, essentially rendering them inapplicable to domains where interacting with the physical environment (sampling) is expensive ( Gopaluni et al., (2020) ), e.g., in industrial chemical process control applications. Model-based RL (MBRL) algorithms are designed to increase the sample efficiency of RL by concurrently learning a policy and a model of the environment, i.e., a function that allows the prediction of future states and rewards from state-action pairs. Figure 2 : Dyna-style ( Sutton, (1991) ) model-based RL framework. The three steps are repeated for a predefined number of steps, or until satisfactory control performance is reached. The seminal Dyna algorithm ( Sutton, (1991) ) iterates between three steps (see Figure 2 ): (i) The current policy interacts with the real environment to gather data about the system dynamics. (ii) Using the acquired data, a data-driven dynamic model of the environment is learned via SI. (iii) The learned model is used to optimize the policy via any suitable RL algorithm, e.g., Proximal Policy Optimization (PPO) ( Schulman et al., (2017) ) or Soft Actor-Critic ( Haarnoja et al., (2018) ). Numerous “Dyna-style algorithms”, i.e., algorithms that follow this basic framework, have been developed over the years. However, in this framework, the policy can learn to exploit the errors of the dynamic model, leading to overly-optimistic simulated results and corresponding policy failures in the real world ( Kurutach et al., (2018) ). However, recent contributions ( Kurutach et al., (2018) ; Clavera et al., (2018) ; Janner et al., (2019) ) have showcased ways to counteract this problem based on learning ensembles of dynamic models, leading to algorithms that can match the asymptotic control performance of model-free RL on certain problems while requiring orders of magnitude fewer interactions with the real environment. Ultimately, to maximize sample efficiency, any Dyna-style algorithm relies on learning reasonably well-generalizing dynamic models of the environment with as little data as possible. Therefore, another intuitive way to improve the performance of Dyna-style algorithms is to incorporate prior knowledge of the dynamics of the environment into the models, e.g., using physics-informed learning. Multiple contributions (e.g., Liu and Wang, (2021) ; Ramesh and Ravindran, (2023) ) have shown that using physics-informed neural networks (PINNs) ( Raissi et al., 2019a () ) in Dyna-style algorithms can increase both the sample efficiency and the performance of the resulting policies. The adoption of RL algorithms into the process control community is still in its infancy and has largely been limited to model-free RL ( Faria et al., (2022) ; Dogru et al., (2024) ). Gopaluni et al., (2020) describe model-free RL algorithms as insufficiently data-efficient for industrial process control applications. They identify the unification of model-based and model-free methods as a research area with tremendous potential to redefine automation in the process industry. Ponse et al.,

Read the complete text