DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models

Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang · 2026

Read DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models by Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16860.txt.

Introduction

Jia and co-authors present DreamPlan, a way to fine-tune vision-language planners without costly real robot trials. Such planners reason well but lack grounded physics, so errors compound in hard tasks like handling deformable objects. DreamPlan first lets the planner gather exploratory data in the real setup. That imperfect data trains an action-conditioned video world model. The planner is then fine-tuned inside this imagined world with ORPO reinforcement learning. The authors report clear gains in manipulation success. No large-scale real-world data collection is needed.

Catalog record

CREATOREmily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34220
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16860

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models Emily Yue-Ting Jia, Weiduo Yuan, Tianheng Shi, Vitor Guizilini, Jiageng Mao, Yue Wang Abstract Robotic manipulation requires sophisticated commonsense reasoning, a capability naturally possessed by large-scale Vision-Language Models (VLMs). While VLMs show promise as zero-shot planners, their lack of grounded physical understanding often leads to compounding errors and low success rates when deployed in complex real-world environments, particularly for challenging tasks like deformable object manipulation. Although Reinforcement Learning (RL) can adapt these planners to specific task dynamics, directly fine-tuning VLMs via real-world interaction is prohibitively expensive, unsafe, and sample-inefficient. To overcome this bottleneck, we introduce DreamPlan , a novel framework for the reinforcement fine-tuning of VLM planners via video world models. Instead of relying on costly physical rollouts, DreamPlan first leverages the zero-shot VLM to collect exploratory interaction data. We demonstrate that this sub-optimal data is sufficient to train an action-conditioned video generation model, which implicitly captures complex real-world physics. Subsequently, the VLM planner is fine-tuned entirely within the “imagination” of this video world model using Odds Ratio Policy Optimization (ORPO). By utilizing these virtual rollouts, physical and task-specific knowledge is efficiently injected into the VLM. Our results indicate that DreamPlan bridges the gap between semantic reasoning and physical grounding, significantly improving manipulation success rates without the need for large-scale real-world data collection. Our project page is https://psi-lab.ai/DreamPlan/ . I INTRODUCTION Robotic manipulation requires the ability to reason semantically and plan coherent multi-step behaviors. Large-scale Vision-Language Models (VLMs), trained on broad multimodal datasets, naturally possess strong semantic priors and reasoning ability, making them highly attractive as zero-shot planners. In this paradigm, a VLM can directly generate high-level plans, subgoals, or tool-use commands from visual observations and language instructions, which are subsequently executed by downstream low-level controllers. While this reasoning-centric approach demonstrates impressive generalization in simple, rigid-object scenarios with predictable dynamics, with contact-rich tasks that involve complex physical dynamics—most notably, the manipulation of deformable objects. The primary bottleneck is that zero-shot VLM planners inherently lack task-specific physical knowledge. Consequently, the generated actions may be semantically plausible yet physically ineffective, as the model cannot reliably anticipate how objects will respond to contact, force, and gravity. This reality gap is severely magnified in deformable object manipulation, where object dynamics are complex, and minor deviations in action can lead to drastically different topological deformations. Although reinforcement learning (RL) offers a principled way to adapt VLM planners to specific task dynamics, directly fine-tuning these massive models via real-world robot interaction is prohibitively expensive, unsafe, and extremely sample-inefficient. To satisfy the massive interaction data requirements of RL, prior works mostly turn to simulation-based training [ 21 , 18 ] . However, constructing high-fidelity, computationally efficient simulators for deformable objects remains a tremendous challenge [ 20 , 25 , 34 ] , inevitably leading to a severe sim-to-real gap. More recently, video generation models have been explored as visual planners or implicit simulators [ 30 , 29 , 31 , 40 ] . However, current action-conditioned visual models are largely limited to simple rigid-body interactions and struggle to realistically simulate complex deformable behaviors, limiting their usefulness for reliable policy fine-tuning. To overcome these fundamental limitations, we propose DreamPlan , a novel world-model-assisted RL framework designed to efficiently align VLM planners with real-world physics via model-generated virtual rollouts. DreamPlan initiates the learning process by deploying the zero-shot VLM to collect exploratory interaction data in the target environment. Although the initial planner suffers from a low success rate—yielding a dataset dominated by failed or sub-optimal attempts—this data still contains dense, rich signals regarding action-outcome causality. We demonstrate that such sub-optimal exploratory data is surprisingly sufficient to train an action-conditioned video world model capable of predicting plausible object deformation. To effectively condition the video diffusion backbone on low-level robotic actions, we render the kinematic configuration of the robot arms corresponding to the commanded motions and apply a ControlNet-style architecture [ 37 ] to inject these rendered visual action cues into the diffusion model. This allows the world model to accurately predict the physical evolution of the target objects under specific robot actions. Empowered by the learned action-conditioned world model, DreamPlan shifts policy adaptation entirely into the model’s imagination. Instead of relying on costly real-world interaction, we generate synthetic rollouts from the world model to provide virtual supervision for adapting the VLM planner. However, directly performing reinforcement learning with repeated video generation remains computationally expensive, as diffusion inference is inherently slow. To address this, we adopt an efficient Best-of- K strategy: for each training input, we sample a batch of candidate actions from the planner and use the world model to predict their outcomes. The action that achieves the highest task-level objective under the predicted rollouts is selected as the positive sample, while the remaining actions are treated as negatives. These world-model-informed preferences are then used to fine-tune the planner via Odds Ratio Policy Optimization (ORPO), which increases the likelihood of preferred actions relative to disfavored ones without requiring repeated world-model queries during optimization, enabling more efficient policy adaptation. In summary, our main contributions are: • We introduce DreamPlan , a highly efficient framework that adapts zero-shot VLM planners to complex physical tasks via offline reinforcement fine-tuning within video-generated virtual rollouts. • We propose an efficient reinforcement adaptation strategy that integrates Best-of- K sampling from the world model with Odds Ratio Policy Optimization (ORPO). This strategy decouples the computationally expensive video diffusion inference step from policy optimization, enabling highly efficient VLM adaptation with physics-informed supervision. • We validate DreamPlan on challenging deformable object manipulation tasks ( e.g. , cloth, ropes, and soft toys). Our results demonstrate that DreamPlan significantly outperforms zero-shot baselines, efficiently injecting crucial physical grounding into VLMs without the heavy burden of large-scale real-world data collection. II RELATED WORK II-A Vision-Language(-Action) Models for Robotics Recent works have extensively explored adapting pretrained Vision-Language Models (VLMs) and Vision-Language-Action (VLA) architectures for robotic manipulation. Works such as [ 1 , 15 , 6 ] successfully align high-level semantic reasoning with robot execution via prompting, value grounding, or multimodal fine-tuning. However, these approaches typically rely on predefined skill primitives or require extensive expert demonstrations, limiting their ability to handle complex physical interactions. To refine performance in specific tasks, recent frameworks leverage reinforcement learning (RL) for scalable policy post-training. Methods such as [ 19 , 16 ] use real interaction data or simulator-generated data to improve policy performance. Closely related to our approach, several emerging methods propose using learned world models as virtual environments to generate synthetic experience for policy fine-tuning. Works such as [ 10 , 22 , 31 , 40 , 23 , 24 , 38 ] integrate world models into the policy optimization loop, enabling imagined rollouts to reduce reliance on real-world interaction. However, these methods primarily focus on rigid-body manipulation or relatively simple deformations, while we try to address the more challenging deformable manipulation task. Moreover, prior approaches typically perform heavy world model inference within the RL loop, whereas we introduce a more efficient Best-of- K strategy that reduces video generation during policy updates while preserving physics-informed supervision. II-B World Models for Robotics World models learn predictive environment dynamics and have been widely studied for improving data efficiency in

Read the complete text