Evaluating Robot Policies in a World Model
Julian Quevedo 1, Percy Liang 1, Sherry Yang 1 2 3, New York University 2, Google DeepMind 3 · 2025
Read Evaluating Robot Policies in a World Model by Julian Quevedo 1, Percy Liang 1, Sherry Yang 1 2 3, New York University 2, Google DeepMind 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00613.txt.
Introduction
Julian Quevedo and co-authors propose grading robot control software inside a learned dream of the world, in this 2025 paper. Testing a robot policy in reality is slow and expensive, while hand-built simulators take endless manual work to look right. Their system, WorldGym, instead trains a video-generating model that predicts how the world responds to actions, then lets the policy play inside that model.
Starting from a single real camera frame, the world model rolls the scene forward while a vision-language model hands out rewards. When the team graded real robot policies this way, success rates inside the dream tracked real-world success closely, and the rankings between policy versions, sizes, and training stages were preserved. The setup also exposed weaknesses: modern robot policies still confuse object shapes and get distracted by fake look-alike objects painted on the scene.
Catalog record
| CREATOR | Julian Quevedo 1, Percy Liang 1, Sherry Yang 1 2 3, New York University 2, Google DeepMind 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Robot Policies in a World Model Julian Quevedo 1, Percy Liang 1, Sherry Yang 1 2 3, New York University 2, Google DeepMind 3 Abstract Robotics has broad applications from automating house chores to taking care of patients. However, evaluating robot control policies is challenging, as real-world testing is expensive, while handcrafted simulations often fail to accurately reflect real-world conditions, resulting in poor correlation between simulated evaluation and real-world outcomes. In this work, we investigate World-model-based Policy Evaluation (WPE). We first train an action-conditioned video generation model as a proxy to real-world environments. To enable efficient rollouts of hundreds of interactive steps while mitigating error accumulation in the world model, we propose an inference scheme which we call Blockwise-Autoregressive Diffusion Transformer with adjustable context and decoding horizon lengths. To ensure that the world model indeed follows action input, we propose metrics based on the agreement between the ground truth video and generated video conditioned on the same sequence of actions to evaluate the world model. We then use the world model for policy evaluation by performing Monte Carlo rollouts in the world model while employing a vision-language model (VLM) as a reward function. Interestingly, we found that WPE tends to underestimate the policy values for in-distribution actions and overestimate policy values for out-of-distribution actions. Nevertheless, WPE preserves the relative rankings of different policies. In emulating real robot executions, WPE achieves high fidelity in mimicing robot arm movements as in real videos, while emulating highly realistic object interaction remains challenging. Despite this limitation, we show that a world model can serve as a starting point for evaluating robot policies before real-world deployment. 1 1 1 See videos and code at https://world-model-eval.github.io 1 Introduction Robots can help humans in ways that range from home robots performing chores shafiullah2023bringing ; liu2024ok to hospital robots taking care of patients soljacic2024robots . One of the major road blocks in the development robots lies in evaluation — how should we ensure that these robots will work reliably without causing any physical damage when deployed in the real world? Traditionally, people have used handcrafted software simulators to develop and evaluate robot control policies tedrake2019drake ; todorov2012mujoco ; erez2015simulation . However, handcrafted simulation based on our understanding of the physical world can be limited, especially when it comes to hardcoding complex dynamics with high degrees of freedom or complex interactions such as manipulating soft objects sunderhauf2018limits ; afzal2020study ; choi2021use . As a result, the sim-to-real gap has hindered progress in robotics zhao2020sim ; salvato2021crossing ; dulac2019challenges . With the development of generative models trained on large-scale video data ho2022imagen ; villegas2022phenaki ; singer2022make , recent work has shown that video world models can visually emulate interactions with the physical real world, by conditioning on control inputs in the form of text yang2023learning ; videoworldsimulators2024 or keyboard strokes bruce2024genie . This brings up an interesting question — could video world models be used to emulate robot interactions with the real world, hence being used to evaluate robot policies in the world model before real-world testing or deployment? Learning a dynamics model from past experience and performing rollouts in the learned dynamics model has been extensively studied in model-based reinforcement learning (RL) fonteneau13batch ; zhang2021autoregressive ; kaiser2019model ; yu2020mopo . However, most of the existing work in model-based RL considers single-task settings, which puts itself at a disadvantage compared to model-free RL, since learning a dynamics model can be much harder than learning a policy in the single-task setting. Nevertheless, we make the important observation that While there can be many tasks and policies, there is only one physical world in which we live that is governed by the same set of physical laws . This makes it possible to learn a single world model that, in principle, can be used to evaluate any policies on any tasks. Figure 1: Overview of WPE. Generated video frames (bottom) are compared against ground-truth video frames (top) given the same action sequence. Semantic segmentation and tracking can evaluate the dynamics of the generative simulator, while VLMs can be used as reward models to compute policy values. Agreement between real-world videos and generated videos reflects the quality of the world model. Inspired by this observation, we propose World-model-based Policy Evaluation (WPE), as shown in Figure 1 , by learning a single world model that generates videos conditioned on actions from diverse tasks and environments. Since a policy might require thousands of interactive steps to complete a task, naïvely rolling out a policy in a next-frame prediction model (similar to existing model-based RL) leads to slowdowns and error accumulations. To overcome this challenge, we propose a Blockwise Autoregressive Transformer Diffusion inference scheme with adjustable context and decoding lengths. With video rollouts from the world model, WPE uses a vision-language model (VLM) to determine tasks success from generated videos. We further offer practical insights on the choices of action representations, context lengths, and block-wise decoding lengths for best generation results. Before evaluating robot policies, we first verify the dynamics and reward function of the world model by measuring their agreement both with a handcrafted software simulator liu2024libero and with a set of real-robot videos. We then use the world model to evaluate a simulated robot policy by rolling out the policy both in the software simulator and in the world model, and compare their success rates (policy values). Our result suggests that WPE preserves the relative rankings of different policies. Interestingly, WPE underestimates in-distribution actions (actions from the policy that curated the data for training the world model) while overestimates the value for out-of-distribution actions. In emulating real-robot interactions, WPE can simulate gripper controls across different control axes highly effectively. Although simulating realistic object interactions remains challenging, we believe WPE can serve as a highly useful tool to sanity check robot policies before deploying them on real robots. Key contributions of this paper include: • We propose to use video world model to evaluate robot policies and perform a comprehensive set of studies using software simulators and real-robot videos to understand its feasibility. • We propose an inference scheme (Blockwise Autoregressive Transformer Diffusion) for fast inference and minimal error accumulation rollouts with hundreds of interactive steps. • We propose evaluation metrics for the world model based on agreement between generated videos and ground-truth videos conditioned on the same sequences of actions. • We study how action representation, context length, and generation horizon length affect the fidelity of simulation, offering practical advice for future development of generative simulators. 2 Problem Formulation In this section, we first define relevant notations and review the formulation of offline policy evaluation (OPE). Next, we situate OPE in practical settings with partially observable environments and image-based observations, and motivate a model-based approach for tackling this practical problem. Multi-Task POMDP. We consider a multi-task, finite-horizon, partially observable Markov Decision Process (POMDP) puterman2014markov ; kaelbling1995partially , specified by \mathcal{M}=(S,A,O,G,R,T,\mathcal{E},H) , which consists of a state space, action space, observation space, goal space, reward function, transition function, emission function, and horizon length. A policy \pi interacts with the environment for a goal starting from an initial state g,s_{0}\sim G , producing a distribution \pi(\cdot|s_{t},g) over A from which an action a_{t} is sampled