Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems

Luning Sun, Xin-Yang Liu, Siyan Zhao, Aditya Grover, Jian-Xun Wang, Jayaraman J. Thiagarajan · 2025

Read Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems by Luning Sun, Xin-Yang Liu, Siyan Zhao, Aditya Grover, Jian-Xun Wang, Jayaraman J. Thiagarajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05588.txt.

Introduction

Luning Sun and co-authors blend cheap and costly simulators to teach control in a 2025 paper. Reinforcement learning needs many trials, but high-fidelity physics is slow and real rigs are scarce. Pure learned surrogates, meanwhile, drift badly on chaotic systems where pointwise accuracy collapses.

Their multi-fidelity framework corrects a physics-based hybrid model with a small ration of high-fidelity data, keeping it differentiable for learning. A spectrum-based reward grades the controller on statistical behavior rather than fleeting exact states. On two hard physics problems the learned control matched statistics from full high-fidelity runs and beat leading baselines.

Catalog record

CREATORLuning Sun, Xin-Yang Liu, Siyan Zhao, Aditya Grover, Jian-Xun Wang, Jayaraman J. Thiagarajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems Luning Sun, Xin-Yang Liu, Siyan Zhao, Aditya Grover, Jian-Xun Wang, Jayaraman J. Thiagarajan Abstract Controlling instabilities in complex dynamical systems is challenging in scientific and engineering applications. Deep reinforcement learning (DRL) has seen promising results for applications in different scientific applications. The many-query nature of control tasks requires multiple interactions with real environments of the underlying physics. However, it is usually sparse to collect from the experiments or expensive to simulate for complex dynamics. Alternatively, controlling surrogate modeling could mitigate the computational cost issue. However, a fast and accurate learning-based model by offline training makes it very hard to get accurate pointwise dynamics when the dynamics are chaotic. To bridge this gap, the current work proposes a multi-fidelity reinforcement learning (MFRL) framework that leverages differentiable hybrid models for control tasks, where a physics-based hybrid model is corrected by limited high-fidelity data. We also proposed a spectrum-based reward function for RL learning. The effect of the proposed framework is demonstrated on two complex dynamics in physics. The statistics of the MFRL control result match that computed from many-query evaluations of the high-fidelity environments and outperform other SOTA baselines. 1 Introduction Instabilities are universal in complex dynamical systems, and controlling their evolution has great value in science and engineering. For example, laser-plasma instability [ 1 , 2 , 3 , 4 , 5 ] is usually an undesirable phenomenon in inertial confinement fusion processes, where a great of input laser energy is dissipated during this process, increasing difficulties in getting power net gain. In fluid dynamics, Plateau-Rayleigh instabilities describe the process of falling liquid film breaking into small drops under a given threshold. Understanding when and why it happens also has great value in the field of multi-phase flow [ 6 , 7 ] . Furthermore, Rayleigh-Benard instabilities, caused by the buoyancy effect, are a widely investigated topic in thermal-driven flows [ 8 ] . Finally, Kevin-Helmholtz instabilities are formulated by the velocity shear [ 9 ] and can be observed in the atmosphere of planets. Given their broad utility, methods for controlling or mitigating such instabilities have gained significant research interest. In particular, deep reinforcement learning (DRL) has emerged as an important approach in different scientific applications. For example, in magnetic confinement nuclear fusion, DRL has witnessed success in controlling tokamak core plasma [ 10 ] and avoiding the plasma tearing instabilities [ 11 ] . There is also recent research about controlling instabilities and chaos in fluid dynamics [ 6 , 12 , 13 ] . These DRL algorithms typically learn policies through multiple interactions with the environments. Given that high-fidelity computational models in many scientific applications are prohibitively expensive, and that deep reinforcement learning (DRL) requires a large number of evaluations of these models to infer a reasonable policy, the feasibility of this approach is a practical concern. Using low-fidelity approximations of the computational model is an alternative solution. However, these approximations often rely on strong assumptions that may not hold in practice, potentially leading to non-physical results. To mitigate the computational burden, while also not sacrificing the performance significantly, we propose a multi-fidelity reinforcement learning framework that controls the high-fidelity model indirectly through a low-fidelity model with learnable correction terms. The key novelty of this work is twofold. First, we integrate additional domain knowledge of instabilities into the DRL framework. Second, we introduce learnable correction terms in a differentiable model to bridge the gap between models of varying fidelities, thereby reducing the number of evaluations required for high-fidelity models. Finally, we benchmark our framework’s ability to mitigate instabilities in complex dynamical systems through two use-cases. We introduced various physical criteria to evaluate the soundness of learning performance. The result shows that the statistics of the controlled result by our framework could match the result computed from many-query evaluations of the high-fidelity environments. The performance of the proposed model also outperforms other baselines. 2 Related Work Instabilities in Dynamical Systems Instabilities in dynamical systems represent a fascinating research area. They occur when small perturbations amplify with time, leading to unpredictable behavior that evolves with time. One area in which instabilities play an important role is laser-plasma interaction (LPI). Key instabilities matter are stimulated Raman and Brillouin scattering (SRS/SBS) [ 14 , 15 ] . Another area is fluid instabilities that are ubiquitously seen in nature, like turbulence, vortex formulation, and chaotic mixing. Classical phenomenons, such as Rayleigh-Bernard convection, and Kelvin-Helmholtz [ 16 , 17 ] have been investigated for decades by experiments and simulations. However, in the deep learning era, huge gaps exist in integrating learning techniques into the framework to control/optimize the instabilities and our work aims to bridge this gap. Deep Reinforcement Learning for Control Deep reinforcement learning (DRL) has demonstrated success in learning optimal policies through trial-and-error interactions with environments, achieving remarkable results in simulated environments like Minecraft and Atari games [ 18 , 19 ] . Recently, there have been attempts to apply RL to real-world physics control tasks, such as robotic fish [ 20 ] , turbulent flows [ 21 ] , and falling liquid films [ 6 ] , where traditional control methods are limited. However, these DRL methods require querying the environment to obtain a sufficient number of transitions for learning the policy, and in real-world, access to a high-fidelity simulator or controller can be expensive and sometimes infeasible. To address the challenge of sample efficiency, one approach is to use model-based RL, which learns a model of the transition dynamics to simulate additional data [ 22 , 23 ] . Leveraging simulators of different fidelity levels to train policies can also improve data efficiency, with techniques like Gaussian Processes and control variates used to help reduce variance in state-action value function [ 24 , 25 ] . Additionally, transfer learning across multi-fidelity data [ 26 ] or similar RL environments [ 27 ] enables knowledge transfer for better data efficiency. Multi-fidelity Modeling and Differentiable Programming The multi-fidelity modeling framework is beneficial for many-query applications like uncertainty quantification and design optimization. This framework usually delegates the majority of the computational budget to low-fidelity models whose output could be corrected by a small number of labeled high-fidelity data. The recent literature has invested in various settings for multi-fidelity modeling. The quantities of interest (QOI) could be either integrated properties [ 28 , 29 ] (e.g., draft/lift coefficient) or full field state variables [ 30 , 31 ] . The correction terms can also be higher-order POD modes [ 32 , 33 ] or missing closure terms [ 34 , 35 ] . Recently, there are also probabilistic multi-fidelity generative models that produce high-fidelity realizations by conditioning on corresponding low-fidelity data [ 36 , 37 , 38 ] . To the author’s knowledge, there is limited research on controlling the instabilities in multi-fidelity formulations. Sahil et al applied the multi-fidelity on a steady-state shape optimization problem [ 26 ] . The knowledge between different fidelity models is shared by the parameter of the policy network by transfer learning. 3 Proposed Approach In this section, we describe our multi-fidelity DRL approach and also discuss the implementation details. The focus of this paper is on using actuators to

Read the complete text