Digital Twin Calibration with Model-Based Reinforcement Learning

Hua Zheng, Wei Xie, Ilya O. Ryzhov, Keilung Choy · 2025

Read Digital Twin Calibration with Model-Based Reinforcement Learning by Hua Zheng, Wei Xie, Ilya O. Ryzhov, Keilung Choy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02205.txt.

Introduction

Hua Zheng and co-authors fold twin tuning into model based learning for control of wild, chance run plants, in a paper from 2025. A twin is a code copy of a real plant, used to plan moves. Old plans trust stiff forms for how the plant shifts and shrug off doubt in twin knobs, a flaw that bites in drug making where runs are scarce and paths are bent.

The frame tunes the twin and hunts the best move plan at once, so twin error shrinks as the plan grows. Trial paths are picked by how well the plan scores, which balances probe and use. Math proofs show the frame lands on the best plan. Bench tests in a drug plant world beat past methods by a clear edge.

Catalog record

CREATORHua Zheng, Wei Xie, Ilya O. Ryzhov, Keilung Choy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02205

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Digital Twin Calibration with Model-Based Reinforcement Learning Hua Zheng, Wei Xie, Ilya O. Ryzhov, Keilung Choy Abstract This paper presents a novel methodological framework, called the Actor-Simulator, that incorporates the calibration of digital twins into model-based reinforcement learning for more effective control of stochastic systems with complex nonlinear dynamics. Traditional model-based control often relies on restrictive structural assumptions (such as linear state transitions) and fails to account for parameter uncertainty in the model. These issues become particularly critical in industries such as biopharmaceutical manufacturing, where process dynamics are complex and not fully known, and only a limited amount of data is available. Our approach jointly calibrates the digital twin and searches for an optimal control policy, thus accounting for and reducing model error. We balance exploration and exploitation by using policy performance as a guide for data collection. This dual-component approach provably converges to the optimal policy, and outperforms existing methods in extensive numerical experiments based on the biopharmaceutical manufacturing domain. Keywords Model-based reinforcement learning, digital twin calibration, model uncertainty, infinite-horizon Markov Decision Processes (MDPs), biomanufacturing 1 Introduction This study is motivated by optimal control applications that exhibit high complexity, high uncertainty, and very limited data ( Wang et al. 2024 ; Zheng et al. 2023 ; Plotkin et al. 2017 ; Mirasol 2017 ) . In particular, all of these challenges are present in the domain of biopharmaceutical manufacturing, used for production of essential life-saving treatments for severe and chronic diseases, including cancers, autoimmune disorders, metabolic diseases, genetic disorders, and infectious diseases such as COVID-19 ( Zahavi and Weiner 2020 ; Teo 2022 ) . Using cells as factories, biomanufacturing involves hundreds of biological, physical, and chemical factors dynamically interacting with each other at molecular, cellular, and macroscopic levels and impacting production outcomes. Due to the complexity of these mechanisms, it is quite difficult to control production safely and effectively, especially in the presence of very limited data. Digital twins have proven very useful in guiding the control of complex physical systems ( Tao et al. 2018 ) . However, there are many situations where the system dynamics are not fully understood and only crudely approximated by digital twin models ( Zheng et al. 2023 ; Nguyen-Tuong and Peters 2011 ) . For example, in biopharmaceutical manufacturing and cell culture processes, a digital twin, built only on limited knowledge of the critical metabolic mechanisms and kinetics of cells, can lead to suboptimal control strategies, such as oxygen and nutrient feeding, that eventually impact the production outputs (e.g., yield and product quality). Therefore, the digital twin should be calibrated to reflect the physical system as accurately as possible, and it should be optimized to identify an effective control policy for the physical system. This paper considers a situation where the digital twin is specified by a finite number of calibration parameters . The parameters are set based on experimental data from the physical system, which are expensive to collect and therefore very limited; often sample sizes are no more than 10-20 ( Wang et al. 2024 ) . However, under a particular parameter setting, the control policy obtained by simulating the digital twin can be used to guide the acquisition of new data ( Powell and Ryzhov 2012 ) . Thus, there is a synergy between the physical and digital systems, with the overall goal being to optimize the control policy in the real world. Both calibration and data acquisition must be linked back to policy performance in the physical system. Therefore, digital twin calibration and informative data collection must be tailored to specific decision making of the real system: (i) the prediction of the mean response for system design; and (ii) the dynamics characterized by a state-action transition model for system control. Most existing studies on computer model calibration (see Sung and Tuo 2024 for a recent review) focus on statistical accuracy and uncertainty quantification for system design rather than control. Many of these studies build on the KOH framework, introduced by Kennedy and O’Hagan 2001 , which uses Gaussian processes (GPs) to model the simulation output as well as the discrepancy from the physical system. Tuo and Wu 2015a refined the KOH framework to handle identifiability issues between calibration parameters and model discrepancy. Further improvements and extensions were developed by Plumlee et al. 2016 and Plumlee 2017 . None of these papers considered the problem of control or modeled any policy optimization problem. In marked contrast, we model both the physical and digital systems as infinite-horizon Markov decision processes (MDPs) with continuous state spaces. The goal in such problems is to learn an optimal policy function mapping a state to an action (or to a distribution over actions). The transition dynamics may have some known structure, but depend on parameters that are unknown for the physical system. The digital twin can be simulated with any choice of parameters, but since we do not know the “true” values, the simulation output may be an inaccurate representation of the real process ( Luo et al. 2020 ) . Our proposed approach integrates parameter calibration with optimal control, strategically prioritizing experiments on the physical system to acquire information that yields the greatest benefit for policy optimization. The problem of learning a state transition model characterizing the dynamics of a physical system, and then using this model to guide the search for a control policy, has been considered in the literature on model-based reinforcement learning (MBRL). See Moerland et al. 2023 for a survey describing various parametric and nonparametric state transition models. One widely used approach is based on GPs ( Deisenroth and Rasmussen 2011 ; Kurutach et al. 2018 ) , which allow for a form of uncertainty quantification that can be explicitly considered when searching for a policy. However, it is quite difficult for GP-based MBRL to incorporate known structural information (for example, physics- and chemistry-based cell growth models). A different approach by Wagenmaker et al. 2021 imposes linear structure on the transition dynamics, but linearity is unable to capture the full complexity of many real-world systems. This reference paper also did not extensively study the connection between the data acquisition strategy, the model error, and the search for the optimal policy. Our proposed “Actor-Simulator” framework uses a general nonlinear state transition model, which incorporates existing structural information (e.g., physics-based models of bioprocess dynamics) as well as uncertainty in the form of unknown parameters. Our approach alternates between calibrating the digital twin, identifying information to be acquired in the next experiment, and learning an reinforcement learning (RL) policy to optimize the physical system. A crucial component of this framework is a novel uncertainty function quantifying the discrepancy between the two systems with respect to the RL objective (i.e., the cumulative discounted reward). We use this criterion in two ways: during calibration, we select the next experiment to maximize the uncertainty function, and during policy optimization, we augment the reward function of the MDP by penalizing high-uncertainty actions. This approach leads to more effective targeted exploration of the state-action space, identifying those parts of it that yield the greatest benefit for policy optimization. We present both theoretical and empirical evidence in favor of the new approach. First, on the theoretical side,

Read the complete text