Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

Matthew Vandergrift, Esraa Elelimy, Martha White · 2026

Read Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy by Matthew Vandergrift, Esraa Elelimy, Martha White (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28433.txt.

Introduction

Matthew Vandergrift, Esraa Elelimy, and Martha White wrote this position paper to separate two jobs that reinforcement learning researchers often blur. Simulators can be the thing being solved, or a stand-in used to learn about decision-making in the real world. Chasing a high simulator score quietly converts the second job into the first.

Vandergrift and colleagues argue the two uses differ in what an agent may do with the simulator, which algorithms suit, and which metrics mean anything. Conflating them produces misleading conclusions about deployed learning, a risk they document with examples and simple experiments. Solving simulators is worthwhile research, they stress, but it answers a different question. Their call is for authors to state plainly which use they intend, so readers can judge the evidence correctly.

Catalog record

CREATORMatthew Vandergrift, Esraa Elelimy, Martha White
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28433

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy Matthew Vandergrift, Esraa Elelimy, Martha White Abstract One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing exclusively on solving the simulator. To achieve high scores, researchers may adopt solutions exclusively meant for solving simulators, rather than learning while the agent is deployed outside a simulator. Solving simulators is also worthy of investigation, but it is a fundamentally different RL research question. In this paper, we argue that RL researchers need to distinguish between two use cases of simulators: solving simulators and using simulators as a proxy for learning in deployment. We first discuss how these two use-cases are importantly different, in terms of constraints on how the agent can use the simulator, which algorithms are appropriate, and which evaluation metrics are appropriate. We then highlight several issues and misleading conclusions that can occur by not making the distinction between these two settings clear, supported with examples and simple experiments. This work is a call to the community to begin clearly distinguishing how they are using simulators in their work, hopefully sparking further discussion on which empirical practices work best in each setting. 1 Introduction Most RL papers do empirical comparison of algorithms in simulators. There are a host of typical research simulators, including classic control, MuJoCo ( Todorov et al. 2012 ) , Arcade Learning Environment ( Bellemare et al. 2013 ) , Behaviour Suite ( Osband et al. 2020 ) , Minecraft ( Guss et al. 2019 ) , and DeepMind-Control ( Tunyasuvunakool et al. 2020 ) . Other papers use RL algorithms in more realistic simulators, such as for a Tokomak reactor ( Degrave et al. 2022b ) or the advanced simulator tools from the process control community, such as Aspen HYSYS or DWSim, for problems like carbon capture ( Al-Sakkaria et al. 2024 ) and water treatment ( Gao et al. 2023 ) . The ubiquitous use of simulators can mask two very different goals in papers: solving simulators or using simulators as a proxy for learning in deployment. For the first group, the primary goal is to solve a specific simulator. For example, an engineer may have a simulation of a wastewater treatment plant they intend to build, and would like to identify an optimal controller to assess the design. As another example, a games researcher may want to find the best Go player, as was done by AlphaGo ( Silver et al. 2016 ) . Critical research questions can be answered through solving simulators. For instance, whether Strassen’s two-level algorithm was optimal for 4\times 4 matrix multiplication was answered by AlphaTensor ( Fawzi et al. 2022 ) , and protein docking molecules were discovered by solving a protein docking simulation ( Kim et al. 2024 ) . For the second group, the simulator is only used as a proxy for learning in deployment. We define the deployment setting as the setting where the dynamics of the environment are outside the control of the practitioner, as is the case when controlling physical systems or interacting with people. Additionally, for learning in deployment, the agent interacts with this single deployment environment, and rewards during that interaction matter. For many researchers in this setting, it would be preferable to run all experiments in deployment. For example, to truly test whether a new RL algorithm can reduce energy costs in a data center, it would be ideal to deploy it directly in a data center and observe any reduction in energy use as it learns and acts on the physical system. But, naturally, running such an experiment is much more onerous. For algorithm development, research simulators allow investigating new ideas quickly and cheaply, and allow running controlled experiments. Some brave researchers have taken the next step of also running RL experiments in real deployment scenarios, such as in robotics ( Büchler et al. 2022 ; Liu et al. 2025 ; White et al. 2012 ; Cheng et al. 2019 ; Ma et al. 2023 ) , recommender systems ( Bietti et al. 2021 ) , and for data center cooling ( Luo et al. 2022 ) . But the vast majority of RL researchers stick to experiments in simulation. When doing experiments in simulation rather than in deployment, it is easy to start inadvertently cheating and exploiting the fact that an algorithm is being tested on a simulator. Thus, a researcher must be hyper-vigilant to constrain the use of the research simulator, to ensure it remains a reasonable proxy for deployment scenarios. When running experiments, however, the short-term goal can morph into solving the research simulator to get good benchmark scores, even though the long-term goal is to understand the algorithms faithfully in simulation, ultimately for practical use. Due to the pressures from benchmarking, it is no surprise that the line between these two goals can become murky. As an anecdotal example, consider the introduction of learning in parallel in Atari with A2C ( Mnih et al. 2016 ) . This method is implicitly designed for the problem of solving simulators, because we cannot have multiple parallel copies of the environment in deployment settings. 1 1 1 There can be instances where deployment involves something resembling parallelism, for instance, a recommendation algorithm can asynchronously provide recommendations to many users. In this case, however, one simulated environment would be the collection of users, i.e., it would include that type of parallelism, whereas parallelizing a simulator would correspond to increasing the number of recommendation platforms. In this work, we define our deployment setting to mean that the agent interacts in a single environment. Yet, the A2C paper did not explicitly state this as its goal and instead stated a goal of simply having a better general RL agent. Many following methods built on this innovation, such as vectorized PPO ( Schulman et al. 2017 ) , IMPALA ( Espeholt et al. 2018 ) , PQN ( Gallici et al. 2025 ) , because it is a great way to solve a simulator more quickly. These papers similarly did not state that their methods were restricted to solving simulators. The ensuing arms race to improve performance in Atari led to Go-Explore ( Ecoffet et al. 2021 ) , which further exploited access to the simulator by resetting the agent to different states for faster learning. There was some negative reaction to this paper, because such resetting was “cheating”, again indicating confusion in the community around these goals. If the Go-Explore paper had stated a goal of solving simulators (getting state-of-the-art on Atari in this case), then resetting is not cheating, and such a negative reaction should not occur. Of course, outrage is a minor consequence; in this paper, we outline that there are many more serious issues with the lack of clarity between these two goals in the RL community. In this paper, we take the position that RL researchers need to distinguish between solving simulators and using simulators as a proxy for learning in

Read the complete text