Planning with Reasoning using Vision Language World Model
Delong Chen, Théo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, Pascale Fung · 2025
Read Planning with Reasoning using Vision Language World Model by Delong Chen, Théo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, Pascale Fung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02722.txt.
Introduction
Delong Chen and six co-authors present a 2025 AI paper introducing a Vision Language World Model, a foundation model that plans by predicting how scenes change after actions. Good planning needs world models that reason at a semantic level over long horizons, which current models rarely do.
Trained on natural videos, the model first infers the goal, then predicts a trajectory alternating actions and state changes. Training targets come from an iterative self-refine process guided by compressed descriptions of future frames. The model supports two planning styles: a fast reactive decoder and a reflective search that scores imagined futures against the goal with a learned critic. On assistance planning benchmarks and human evaluations, the reflective mode scored clearly higher. It also beat strong vision-language baselines on robot question answering.
Catalog record
| CREATOR | Delong Chen, Théo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, Pascale Fung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Planning with Reasoning using Vision Language World Model Delong Chen, Théo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, Pascale Fung Abstract Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the Vision Language World Model (VLWM) , a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions . The VLWM learns both an action policy and a dynamics model, which respectively facilitates reactive system-1 plan decoding and reflective system-2 planning via cost minimization . The cost evaluates the semantic distance between the hypothetical future states given by VLWM roll-outs and the expected goal state, and is measured by a critic model that we trained in a self-supervised manner. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations, where system-2 improves the Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmark. † † affiliation: Meta FAIR † † contribution: Joint first author † † date: August 10, 2026 † † correspondence: Delong Chen [email protected] , Pascale Fung [email protected] 1 Introduction Figure 1 : Example of a VLWM action-state trajectory given a video observation and a goal. VLWM can either generate a plan using one roll-out (system-1), or search over multiple actions by inferring the new world states and minimizing a given cost function (system-2). World models enable AI agents to optimize action plans internally instead of relying on exhaustive trial-and-error in real environments ( LeCun 2022 ; Ha and Schmidhuber 2018 ; Fung et al. 2025 ) , showing strong performance in planning across low-level, continuous control tasks such as robotic control ( Oquab et al. 2023 ; Yang et al. 2024 ; Assran et al. 2025 ; Pan et al. 2025 ) and autonomous driving ( Hu et al. 2023 ; Wang et al. 2024c ) . However, learning world models for high-level task planning – where actions involve semantic and temporal abstraction ( Sutton et al. 1999 ; Chen et al. 2025 ) – remains an open challenge. Bridging this gap could unlock a wide range of practical applications, such as AI agents in wearable devices assisting humans in complex tasks and embodied agents capable of autonomously pursuing long-horizon goals. To obtain a high-level world model, existing approaches fall short. Prompting-based practices ( Hao et al. 2023 ; Tang et al. 2024 ; Wang et al. 2024b ; Gu et al. 2024 ) is straightforward but inadequate as LLMs are not directly grounded in sensory experience. VLMs are primarily trained for visual perception and instead of action-conditioned prediction of world-state transitions. Meanwhile, learning from simulation environments ( Hafner et al. 2024 ; Wang et al. 2025b ) cannot scale to divers real-world activities. Existing world models learned from natural videos often rely on generative architectures ( e.g., diffusion models) to generate future observations ( Yang et al. 2023b ; Brooks et al. 2024 ; Agarwal et al. 2025b ) . Such formulation is not only ill-posed due to partial observability and uncertainty, but also inefficient, capturing task-irrelevant details and imposing high computational costs for long-horizon roll-outs. These limitations highlight the need for world models that predict in abstract representation spaces , rather than raw pixels. In this work, we propose to learn a world model that leverages natural language as its abstract world state representation. We introduce the Vision Language World Model (VLWM) , which perceives the environment through visual observations and predicts world evolution using language-based abstraction (Figure 1 ). Language inherently provides semantic abstraction and is significantly more computationally efficient to generate compared to raw sensory observations. In comparison with latent embeddings in Joint Embedding Predictive Architecture (JEPA)-based world models ( LeCun 2022 ; Zhou et al. 2024 ; Assran et al. 2025 ) , language-based abstraction is intuitive, interpretable, and enables seamless integration with prior knowledge and extensive engineering ecosystems developed for LLMs/VLMs. Compared to current LLMs/VLMs paradigms that primarily focus on perception ( Cho et al. 2025 ) , behavior cloning (SFT) ( Zeng et al. 2023 ) , or reinforcement learning with verifiable rewards ( Shao et al. 2024 ) , we propose to perform direct world modeling as an objective based on massive, uncurated videos, i.e., reward-free offline data ( Sobal et al. 2025 ) . Figure 2 : Overview of VLWM. (a) VLWM is a JEPA-style world model that predict abstract representation of future world states, instead of generating noisy and high-volume raw observations. (b) Given video contexts, VLWM’s prediction target is a structured textual representation of the unobserved future. It includes goal and interleaved action ( A ) world state changes ( \Delta S ), all extracted automatically. (c) VLWM can infer possible goals from the context, and interpret them with current initial state and the expected final state. It supports both fast reactive system-1 plan generation and reflective system-2 reasoning based on cost minimization. An overview of the framework is shown in Figure 2 . To construct training prediction targets, VLWM employs an efficient abstraction pipeline that first compresses raw video into a hierarchical Tree of Captions , then refines it into structured goal-plan descriptions using an LLM-based Self-Refine ( Madaan et al. 2023 ) . The model is trained to predict these abstractions—capturing goal description, goal interpretation, actions ( A ) and world state changes ( \Delta S ) – conditioned on visual context from past observations. From this, both a predictive world model (S_{t},A_{t}\rightarrow S_{t+1}) and an action policy (S_{t}\rightarrow A_{t+1}) are learned. It enables straightforward plan generation via text completion, using the proposed action directly as policy. We term this approach system-1 planning. However, the autoregressive nature of token decoding limits foresight and reflection, as each action decision become irreversible once made. Additionally, when training on large-scale, real-world video datasets which typically contain imperfect demonstrations, the resulting policy will also clone those suboptimal behaviors present in the data. To unleash the full potential of VLWM, we introduce a reflective system-2 “planning with reasoning” mode. In this mode, VLWM first generates multiple roll-out based on action candidates (either proposed by itself or externally provided) and predicts resulting world states. We then search for the candidate action sequence that minimize a scalar cost , which is evaluated by a critic module that assess the desirability of candidate plans. This critic is a language model trained through a self-supervised objective: it learns to assign lower costs to valid progress toward the goal and higher costs to counterfactual or irrelevant actions, effectively measuring how closely each candidate action aligns with the desired goal state. The process of optimizing action plan by searching for a cost-minimizing candidate is a form of reasoning ( LeCun 2022 ) . It enables the agent to perform trial-and-error internally with its learned world model to obtain the optimal action plans. The VLWM is