Hierarchical Planning with Latent World Models

Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, Nicolas Ballas · 2026

Read Hierarchical Planning with Latent World Models by Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, Nicolas Ballas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03208.txt.

Introduction

Zhang and colleagues tackle long-horizon robot planning, where prediction errors compound and naive search explodes with the horizon. Their HWM system learns world models at several time scales in one latent space, letting long-horizon predictions set subgoals for short-horizon control without task-specific rewards. On a real Franka arm, pick-and-place from one goal image succeeded 70 percent of the time, versus zero for single-level planning.

Catalog record

CREATORWancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, Nicolas Ballas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72080
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03208

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Planning with Latent World Models Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, Nicolas Ballas Abstract Model predictive control (MPC) with learned world models has emerged as a promising paradigm for embodied control, particularly for its ability to generalize zero-shot when deployed in new environments. However, learned world models often struggle with long-horizon control due to the accumulation of prediction errors and the exponentially growing search space. In this work, we address these challenges by learning latent world models at multiple temporal scales and performing hierarchical planning across these scales, enabling long-horizon reasoning while substantially reducing inference-time planning complexity. Our approach serves as a plug-in planning abstraction that applies across diverse latent world-model architectures and domains. We demonstrate that this hierarchical approach enables zero-shot control on real-world non-greedy robotic tasks, achieving a 70% success rate on pick-&-place using only a final goal specification, compared to 0% for a single-level world model. In addition, across physics-based simulated environments including push manipulation and maze navigation, hierarchical planning achieves higher success while requiring up to 3 × 3\times less planning-time compute. † † affiliation: FAIR at Meta † † affiliation: New York University † † affiliation: Mila - Québec AI Institute † † affiliation: Brown University † † contribution: Work done at Meta † † contribution: Joint last author † † date: August 10, 2026 † † correspondence: Wancong Zhang at [email protected] † † Code: https://github.com/kevinghst/HWM_PLDM † † Website: https://kevinghst.github.io/HWM/ High Level WM Low Level WM generate subgoal Goal Obs Subgoal High Level Plan to Goal Low Level Plan to Subgoal Current Obs actions World Model Task SR SR w/ hierarchy \Delta SR % VJEPA2-AC Pick&Place (oracle subgoals) 80% 80% 0 Pick&Place (no oracle) 0% 70% +70 Open / Close Drawer 30% 70% +40 DINO-WM Push-T 17% 61% +44 PLDM Diverse Maze 44% 83% +39 Figure 1 : Top: Hierarchical planning in latent space. A high-level planner optimizes macro-actions using a long-horizon world model to reach the goal; the first predicted latent state serves as a subgoal for a low-level planner, which optimizes primitive actions with a short-horizon world model. Solid borders denote ground-truth observations; others are decoder reconstructions shown for interpretability only. Bottom Left: Empirical coverage. Hierarchical planning improves success on non-greedy, long-horizon tasks across multiple latent world models. \Delta SR denotes the absolute success-rate gain from hierarchy. Oracle subgoals use externally provided subgoal images; otherwise, planning relies only on the final goal. Bottom Right: Planning efficiency. Success rate versus normalized planning time for flat and hierarchical planners; hierarchical planning matches or exceeds success with \approx 3\times lower planning time. Highlighted HWM results prioritize planning efficiency rather than peak performance. 1 Introduction Planning with learned world models has emerged as a powerful paradigm for embodied decision making. By learning predictive models of environment dynamics from experience and using them to simulate future outcomes, world models enable agents to reason about the long-term consequences of actions ( Sutton 1981 ; Deisenroth and Rasmussen 2011 ) . Recent advances in representation learning and latent-space modeling have improved the scalability of this approach, making it possible to train world models directly from high-dimensional observations such as images and videos ( Ha and Schmidhuber 2018 ; Hafner et al. 2023 ; Hansen et al. 2023 ) . More recently, self-supervised learning has enabled training world models from large collections of unlabeled, task-agnostic offline trajectories, allowing zero-shot deployment in downstream robotic control and navigation ( Assran et al. 2025 ; Sobal et al. 2025 ; Bar et al. 2025 ; Zhou et al. 2024 ; Terver et al. 2026b ) . Despite this progress, learned world models continue to struggle with long-horizon control due to two fundamental challenges: (i) prediction errors compound over time, leading to increasingly inaccurate rollouts ( Talvitie 2014 ; Janner et al. 2019 ) , and (ii) even under a perfect model, planning becomes computationally expensive as the search space grows exponentially with horizon ( Ichter et al. 2020 ) . A classic approach to long-horizon control is to introduce hierarchy into the decision-making structure. By reasoning at multiple temporal scales, agents can plan coarsely over long horizons while retaining fine-grained control at short horizons. This intuition underlies a large body of work on temporal abstraction in reinforcement learning, including options, skills, and hierarchical policies ( Sutton et al. 1999 ; Bacon et al. 2017 ; Hafner et al. 2022 ; Park et al. 2023 ) , as well as hierarchical world models that capture dynamics at multiple timescales ( Gumbsch et al. 2023 ; Chitnis et al. 2024 ; Schiewer et al. 2024 ; Hansen et al. 2024 ) . In parallel, the optimal control literature has explored hierarchical model predictive control (MPC) for multi-task settings ( Fang et al. 2019 ; Li et al. 2021 ; Kögel et al. 2025 ) . However, these approaches remain limited in complementary ways. Reinforcement learning methods typically rely on task-specific rewards, which restrict their applicability in zero-shot settings; even goal-conditioned and zero-shot RL methods ( Park et al. 2023 ; Park et al. 2024b ) still struggle to generalize to environments and tasks beyond the training distribution ( Sobal et al. 2025 ) . Conversely, existing hierarchical MPC methods are not task-specific but are largely restricted to low-dimensional state spaces, hand-engineered representations, or known dynamics, limiting their applicability to high-dimensional observation spaces such as raw pixels. As a result, it remains unclear how to realize hierarchical planning for zero-shot control directly on learned world models from high-dimensional observations. In this work, we propose Hierarchical Planning with Latent World Models (HWM), a framework for zero-shot hierarchical model predictive control using learned latent world models. Rather than learning hierarchical policies, HWM introduces temporal hierarchy directly at inference time by planning over multiple timescales on task-agnostic world models trained via next-latent prediction. Crucially, our approach is defined by how the hierarchical levels interact. HWM learns world models operating at different temporal resolutions within a shared latent space, enabling a direct coupling between levels: high-level predictions act as subgoals for low-level MPC via latent-state matching. This removes the need for inverse models, skill learning, or goal-conditioned policies, and allows hierarchical planning to operate entirely on pretrained world models. To support efficient long-horizon planning, we further introduce a learned action encoder that compresses sequences of primitive actions between waypoint states into latent macro-actions. This reduces the dimensionality of the high-level search space and makes planning more tractable. By decoupling long-horizon reasoning from fine-grained control, HWM mitigates error accumulation and reduces overall planning complexity. z_{1} E s_{1} P^{(2)} l_{t_{1}} \hat{z}_{t_{2}} P^{(2)} l_{t_{2}} P^{(2)} l_{t_{H-1}} \hat{z}_{H} \cdot\ \ \cdot\ \ \cdot\ \ \cdot\ \ \cdot P^{(1)} a_{1} P^{(1)} a_{h} \hat{z}_{h} \dots z_{\text{goal}} E s_{\text{goal}} \mathcal{L}_{\text{low}} \mathcal{L}_{\text{high}} High-Level Planning to Goal \arg\min_{\{l\}}\mathcal{L}_{\text{high}} Low-Level Planning to 1^{\text{st}} Subgoal \arg\min_{\{a\}}\mathcal{L}_{\text{low}} Model key P^{(1)} Low-level world model P^{(2)} High-level world model Figure 2 : Hierarchical planning in latent space. A high-level planner optimizes macro actions using a long-horizon latent world model to reach the final goal embedding. The resulting first predicted latent state serves as a subgoal for low-level planning, where a short-horizon world model optimizes primitive actions to reach this subgoal.

Read the complete text