WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models

R. Khorrambakht, Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, Ludovic Righetti · 2025

Read WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models by R. Khorrambakht, Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, Ludovic Righetti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03077.txt.

Introduction

R. Khorrambakht and six colleagues plan robot actions by imagining their outcomes in a learned visual world model. Copying human demos task by task transfers poorly and needs constant resets. Their 2025 system, WorldPlanner, instead learns from a few hours of unstructured play data. It builds an action-conditioned video model, a diffusion sampler for candidate actions, and optionally a reward model.

A Monte Carlo tree search then plans long action sequences inside the imagined world, and a simple predictive controller executes the plan on the real robot. The action sampler curbs the world model's hallucinations during planning. On three real robot tasks of varied difficulty, planning beat behavior cloning baselines by a clear margin on a standard manipulation test set.

Catalog record

CREATORR. Khorrambakht, Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, Ludovic Righetti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models R. Khorrambakht, Joaquim Ortiz-Haro, Joseph Amigo, Omar Mostafa, Daniel Dugas, Franziska Meier, Ludovic Righetti Abstract Robots must understand their environment from raw sensory inputs and reason about the consequences of their actions in it to solve complex tasks. Behavior Cloning (BC) leverages task-specific human demonstrations to learn this knowledge as end-to-end policies. However, these policies are difficult to transfer to new tasks, and generating training data is challenging because it requires careful demonstrations and frequent environment resets. In contrast to such policy-based view, in this paper we take a model-based approach where we collect a few hours of unstructured easy-to-collect play data to learn an action-conditioned visual world model, a diffusion-based action sampler, and optionally a reward model. The world model – in combination with the action sampler and a reward model – is then used to optimize long sequences of actions with a Monte Carlo Tree Search (MCTS) planner. The resulting plans are executed on the robot via a zeroth-order Model Predictive Controller (MPC). We show that the action sampler mitigates hallucinations of the world model during planning and validate our approach on 3 real-world robotic tasks with varying levels of planning and modeling complexity. Our experiments support the hypothesis that planning leads to a significant improvement over BC baselines on a standard manipulation test environment. I Introduction For a robot to operate in real-world environments, it must reason about physical interactions based on raw sensory observations and the actions it takes. A common approach in the literature, known as Behavior Cloning (BC), implicitly encodes this knowledge into a policy by training on offline human teleoperation data [ 6 , 22 ] . However, such episodic, task-oriented data must meet certain standards (e.g., low entropy [ 24 ] and sufficient coverage) and requires frequent environment resets, making it prohibitively expensive to collect at scale. In this paper, we adopt a different formulation: we first learn a dynamics and reward model from unstructured play data and then use it to adapt state-of-the-art planning and control algorithms to synthesize trajectories for new tasks. Recently, with the advent of high-fidelity video generative models [ 25 ] and diffusion architectures equipped with efficient sampling strategies [ 14 ] , explicit modeling of environment dynamics directly in raw image space has seen major breakthroughs [ 25 ] . In the context of robotic control and planning, action-conditioned variants of these architectures have been proposed [ 1 ] , demonstrating long-horizon future state prediction with minimal degradation. Building on this progress, we propose a framework that, instead of memorizing optimal actions as in BC, synthesizes them through a search process in the action/observation distribution of an unstructured play dataset. Importantly, since such data imposes no constraints on entropy or environment resetting, it can be collected far more easily, either via direct teleoperation or automated exploration policies. In this paper, we show that this play distribution contains action primitives useful for solving new goal-specific tasks. Specifically, we formulate a Monte Carlo Tree Search (MCTS) planner and a zeroth-order Model Predictive Controller (MPC) built around a small auto-regressive diffusion-based visual world model [ 2 ] and several image-conditioned reward functions. Our model is trained solely on a small play dataset (few hours of play) on a single local GPU within a few days. A compact world model not pretrained on prior data allows us to properly evaluate the feasibility of generating trajectories solely from prior-play data. However, note that our planning formulation is directly applicable to bigger foundation world models as well. We validate our approach on a range of representative real-world manipulation tasks of varying difficulty, involving both rigid and deformable objects. Our contributions are as follows: • We propose a control and planning framework based on learned world models from unstructured play data , avoiding the costly requirements of collecting goal-directed optimal demonstrations and enabling the synthesis of behaviors not directly seen in the play data. • We propose to discretize the MCTS search space using a stochastic diffusion model capturing the play distribution. This minimizes out-of-distribution rollouts by constraining search in the training distribution of the world model, thus enabling reliable planning. • We conduct our study using real hardware/data and demonstrate quantitative/qualitative validations in manipulation tasks with varying levels of complexity. II Related Work II-A Short Horizon Planning with World Models To our knowledge, most contributions focusing on the run-time generation of actions with world models are limited to short-horizon planning, either due to error accumulation in traditional world models with highly compressed latent spaces [ 10 , 8 , 9 ] or due to the prohibitive compute requirements of more recent foundation world models [ 1 ] . A recent representative example is [ 3 ] , which models the dynamics in the latent space of a foundation video encoder and uses it to formulate a one-step MPC planner. The authors of [ 3 ] also report one-step MPC with Cosmos foundation world model [ 1 ] , in which case each control update takes up to several minutes, demonstrating the need for more efficient models in planning. In this paper, we show that a relatively small diffusion-based world model originally proposed for simple Atari games [ 19 ] can instead be used to learn high-fidelity world models for real-world robotic tasks, avoiding the prohibitive computation cost due to its size and ensuring long-horizon simulation due to the diffusion retraction to the image manifold at each prediction step. Additionally, we train our world model from scratch from a small-scale (few hours) dataset, showing the possibility of adoption as a local visual dynamical model for planning and control. II-B Long Horizon Planning The prior research on real-world robotic long-horizon planning with visual world models is highly sparse, with only very recent examples such as [ 15 ] that adopt a simulator as the world model and a pretrained vision-language-action model as an action proposal module for the MCTS. Alternative examples include the AlphaGo line of work [ 20 ] that proposes the adoption of learning in MCTS in achieving remarkable play performance in game environments. To our knowledge, this paper is the first showing long-horizon multi-step planning in real-world robotic settings using learned world models. It demonstrates that with suitable world and action proposal models, such run-time reasoning and planning solutions may also be adopted to solve robotic tasks. Furthermore, we demonstrate that planning in low-level action space may be rendered feasible by discretizing the action space using a diffusion action model trained on unstructured play datasets. III Method Our method consists of two stages. In the first stage, we train a world model, a reward model, and an action prior model from a dataset of unstructured play trajectories. In the second stage, we use these models to implement an MCTS planner and an MPC controller to generate new motions. We present the notation and formal problem definition in Sec. III-A , describe the learned models in Secs. III-B , III-C , and III-D , and then introduce the MCTS planner in Sec. III-E and the MPC controller in Sec. III-F . III-A Notation and Problem Definition We adopt the standard

Read the complete text