Closing the Sim2Real Performance Gap in RL

Akhil S Anand, Shambhuraj Sawant, Jasper Hoffmann, Dirk Reinhardt, Sebastien Gros · 2025

Read Closing the Sim2Real Performance Gap in RL by Akhil S Anand, Shambhuraj Sawant, Jasper Hoffmann, Dirk Reinhardt, Sebastien Gros (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17709.txt.

Introduction

Akhil S Anand and five co-authors tackle a stubborn problem in robot learning. Policies trained inside a simulator often fail in the real world. The authors call this mismatch the sim-to-real gap. Simulators are built to predict well, while policies aim to perform tasks well.

The team works out how a trained policy changes when simulator settings change. They study policies learned with a common actor-critic method. That math supports a two-level approach that tunes the simulator itself. Tuning uses grades from real-world performance, not just prediction error. A test version built on PPO shows the idea can work.

Catalog record

CREATORAkhil S Anand, Shambhuraj Sawant, Jasper Hoffmann, Dirk Reinhardt, Sebastien Gros
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Closing the Sim2Real Performance Gap in RL Akhil S Anand, Shambhuraj Sawant, Jasper Hoffmann, Dirk Reinhardt, Sebastien Gros Abstract Sim2Real aims at training policies in high-fidelity simulation environments and effectively transferring them to the real world. Despite the developments of accurate simulators and Sim2Real RL approaches, the policies trained purely in simulation often suffer significant performance drops when deployed in real environments. This drop is referred to as the Sim2Real performance gap. Current Sim2Real RL methods optimize the simulator accuracy and variability as proxies for real-world performance. However, these metrics do not necessarily correlate with the real-world performance of the policy as established theoretically and empirically in the literature. We propose a novel framework to address this issue by directly adapting the simulator parameters based on real-world performance. We frame this problem as a bi-level RL framework: the inner-level RL trains a policy purely in simulation, and the outer-level RL adapts the simulation model and in-sim reward parameters to maximize real-world performance of the in-sim policy. We derive and validate in simple examples the mathematical tools needed to develop bi-level RL algorithms that close the Sim2Real performance gap. SDM Sequential Decision Making MDP Markov Decision Process RL Reinforcement Learning SL Supervised Learning SSL Self-Supervised Learning DP Dynamic Programming ADP Approximate Dynamic Programming SP Stochastic Programming LP Linear Programming MPC Model Predictive Control EMPC Economic Model Predictive Control TD Temporal Difference DPG Deterministic Policy Gradient SPG Stochastic Policy Gradient MBRL Model-based Reinforcement Learning DT Digital Twin ML Machine Learning LSTM Long Short-Term Memory RNN Recurrent Neural Network DNN Deep Neural Network DNNs Deep Neural Networks GP Gaussian Processes AI Artificial Intelligence MLE Maximum Likelihood Estimation LQR Linear Quadratic Regulator GPU Graphics processing unit TPU Tensor processing unit GANs Generative Adversarial Networks CNNs Convolutional Neural Networks LS Least Squares MSE Mean Squared Error MAP Maximum A Posteriori GMM Gaussian Mixture Model HMM Hidden Markov Model ELBO Evidence Lower Bound VAE Variational Autoencoder GAN Generative Adversarial Network ODE Ordinary Differential Equations UDE Universal Differential Equations HAM Hybrid Analysis and Modeling VI Variational Inference EM Expectation-Maximization LSTDQ Least Square Temporal Difference Q-learning RLDP Reinforcement Learning of Decision Processes OR Operations Research MPPI Model Predictive Path Integral CEM Cross Entropy Method IFT Implicit Function Theorem PPO Proximal Policy Optimization SAC Soft Actor Critic LLM Large Language Model NN Neural Network 1 Introduction Sim2Real rl ( rl ) addresses the challenge of obtaining real-world data by training rl agents in simulation and then transferring them to real-world environments ( Talvitie 2018 ) . The overarching objective of Sim2Real rl is to train agents capable of achieving optimal real-world performance, which we refer to as Sim2Real optimality . Sim2real rl has been supported by significant advances in capabilities and fidelity of simulators ( Makoviychuk et al. 2021 ; Todorov et al. 2012 ; Collins et al. 2021 ) . Despite these advances, the inevitable discrepancies between simulated and real-world conditions lead to a significant performance gap upon policy transfer, often referred to as the Sim2Real gap ( Da et al. 2025 ) . To mitigate this gap, methods such as domain randomization ( Tobin et al. 2017 ; Zhao et al. 2020 ) or domain adaptation ( Höfer et al. 2021 ) are used to ensure that the agent is robust to distribution shifts that occur while transferring to the real world. However, robustness does not yield Sim2Real optimality ( Zhou et al. 1996 ) . Sim2Real rl methods typically do not directly optimize policy performance for the real-world conditions; instead, they rely on proxies such as simulator accuracy, parameter variability, or prediction fidelity ( Da et al. 2025 ) . However, the conditions required for Sim2Real optimality are distinct from conditions on simulator accuracy or robustness ( Skelton 1989 ; Anand et al. 2024 ) . Consequently, these methods cannot inherently guarantee optimal real-world performance. The Sim2Real performance gap in rl is closely tied to the objective mismatch in mbrl ( mbrl ) ( Wei et al. 2024 ; Eysenbach et al. 2022 ; Janner et al. 2019 ) , emphasizing that standard metrics for prediction accuracy in simulation models often fail to correlate with the real-world performance of the policy developed in simulation. In theory, for any stochastic real-world system, it is possible to adjust the simulation environment such that the optimal policy in simulation is also optimal in the real world ( Anand et al. 2025 ) . However, there is still no general consensus on the correct objective for training the simulation model, beyond the intuitive goal that both the learned model and policy should ultimately optimize the expected return in the real-world environment ( Wei et al. 2024 ) . Following previous works ( Anand et al. 2025 ; Nikishin et al. 2022 ; Amos and Yarats 2020 ) , we frame the problem of finding or adapting the simulation environment parameters that will guarantee Sim2Real optimality, and cast it as an rl problem on the real environment data. This forms a bi-level rl framework, where the inner-level consists of a standard in-simulation rl agent, and the outer-level comprises a real-world rl agent that updates the simulator parameters by evaluating the real-world performance of the in-sim agent. In this paper, we consider the general case where the rl agent in simulation is obtained through spg ( spg ) methods, considering it is generalizable to both discrete and continuous action spaces. To that extent, we derive the mathematical tools that serve as the foundation for bi-level policy gradient rl algorithms that can achieve Sim2Real optimality both in Sim2Real rl and dyna-style mbrl frameworks. The central contribution is the estimation of sensitivities of the in-sim rl process using spg . Figure 1 : (left) Classical Sim2Real rl , (right) the proposed bi-level rl framework. 2 Related Work Sim2Real RL Approaches: The problem of the Sim2Real gap has been traditionally addressed by either enriching the simulator fidelity or exposing agents to diverse simulated scenarios. Domain randomization is a common approach that perturbs the simulator parameters during training to improve robustness, such that the resulting in-sim policy generalizes better to the real-world ( Tobin et al. 2017 ; Duan et al. 2024 ) . Such approaches often demand extensive domain knowledge and manual feature engineering. Few works have explored automated curriculum-based randomization to reduce manual tuning ( Akkaya et al. 2019 ; Yuan et al. 2024 ) . In another approach, domain adaptation rather than randomizing aligns feature or parameter distributions between the simulator and real environment, commonly via adversarial training on observations ( Chen et al. 2020 ; Long et al. 2018 ; Pei et al. 2018 ; Höfer et al. 2021 ) . In meta rl , the agent is trained on a distribution of environments with the goal that the agent can then quickly adapt to the real environment Finn et al. 2017 ; Brown et al. 2020 . Here, the environment distribution during training needs to be carefully designed such that it covers the real-world environment during testing. Other approaches include grounded action methods to adjust simulator dynamics to match observed real trajectories ( Hanna and Stone 2017 ; Desai et al. 2020 ) and distributionally robust rl formulates the problem as

Read the complete text