MinD: Unified Visual Imagination and Control via Hierarchical World Model
Xiaowei Chi 1,2 ∗† , Kuangzhi Ge 3 ∗ , Jiaming Liu 3 † , Siyuan Zhou 1,2 , Peidong Jia 3 , Zichen He 3 , Sirui Han 2 , Yuzhen Liu 1 , Tingguang Li 1 , Lei Han 1 , Shanghang Zhang 3 ✉ , Yike Guo 2 ✉ 1 Tencent Robotics X; 2 Hong Kong University of Science and Technology; 3 Peking University ∗ Equal co · 2025
Read MinD: Unified Visual Imagination and Control via Hierarchical World Model by Xiaowei Chi 1,2 ∗† , Kuangzhi Ge 3 ∗ , Jiaming Liu 3 † , Siyuan Zhou 1,2 , Peidong Jia 3 , Zichen He 3 , Sirui Han 2 , Yuzhen Liu 1 , Tingguang Li 1 , Lei Han 1 , Shanghang Zhang 3 ✉ , Yike Guo 2 ✉ 1 Tencent Robotics X; 2 Hong Kong University of Science and Technology; 3 Peking University ∗ Equal co (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18897.txt.
Introduction
Chi and eleven co-authors present a robot world model in a paper from 2025 whose head title frames it as learning a dual-system model for real-time planning and risk analysis. Video generation models can predict future scenes for robots, but full frame-by-frame generation is too slow for live control. Their system, MinD, splits the work between two diffusion processes running at unlike speeds.
A slow generator sketches future scenes while a fast policy outputs actions, fed by rough one-step scene sketches instead of finished frames. A matcher module keeps the two in sync. The system succeeded often on lab and real robot tasks and flagged most likely failures in advance.
Catalog record
| CREATOR | Xiaowei Chi 1,2 ∗† , Kuangzhi Ge 3 ∗ , Jiaming Liu 3 † , Siyuan Zhou 1,2 , Peidong Jia 3 , Zichen He 3 , Sirui Han 2 , Yuzhen Liu 1 , Tingguang Li 1 , Lei Han 1 , Shanghang Zhang 3 ✉ , Yike Guo 2 ✉ 1 Tencent Robotics X; 2 Hong Kong University of Science and Technology; 3 Peking University ∗ Equal co |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18897
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MinD: Unified Visual Imagination and Control via Hierarchical World Model Xiaowei Chi 1,2 ∗† , Kuangzhi Ge 3 ∗ , Jiaming Liu 3 † , Siyuan Zhou 1,2 , Peidong Jia 3 , Zichen He 3 , Sirui Han 2 , Yuzhen Liu 1 , Tingguang Li 1 , Lei Han 1 , Shanghang Zhang 3 ✉ , Yike Guo 2 ✉ 1 Tencent Robotics X; 2 Hong Kong University of Science and Technology; 3 Peking University ∗ Equal contribution, † Project lead, ✉Corresponding author Abstract Video generation models (VGMs) offer a promising pathway for unified world modeling in robotics by integrating simulation, prediction, and manipulation. However, their practical application remains limited due to (1) slow generation speed, which limits real-time interaction, and (2) poor consistency between imagined videos and executable actions. To address these challenges, we propose Manipulate in Dream (MinD) , a hierarchical diffusion-based world model framework that employs a dual-system design for vision-language manipulation. MinD executes VGM at low frequencies to extract video prediction features, while leveraging a high-frequency diffusion policy for real-time interaction. This architecture enables low-latency, closed-loop control in manipulation with coherent visual guidance. To better coordinate the two systems, we introduce a video-action diffusion matching module(DiffMatcher), with a novel co-training strategy that uses separate schedulers for each diffusion model. Specifically, we introduce a diffusion-forcing mechanism to DiffMatcher that aligns their intermediate representations during training, helping the fast action model better understand video-based predictions. Beyond manipulation, MinD also functions as a world simulator, reliably predicting task success or failure in latent space before execution. Trustworthy analysis further shows that VGMs can preemptively evaluate task feasibility and mitigate risks. Extensive experiments across multiple benchmarks demonstrate that MinD achieves state-of-the-art manipulation 63+% in RL-Bench, advancing the frontier of unified world modeling in robotics. Our demo page: https://manipulate-in-dream.github.io/ Figure 1: M anipulate in D ream( MinD ): A video-action unified generation world model that can manipulate, imagine, and simulate. MinD integrates LoDiff-Visual for low-frequency video generation and HiDiff-Policy for high-frequency action planning. A dynamic feature adapter, DiffMatcher , bridges motion features between the two systems, ensuring consistency across video and action. 1 Introduction World models aim to endow agents with the ability to simulate, predict, and plan based on internal representations of their environment ha2018world . In robotics, such models are critical for enabling intelligent behavior under uncertainty and for reasoning about long-horizon consequences of actions. Recent progress in generative modeling, especially video generation models (VGMs) sora ; agarwal2025cosmos , has opened up new opportunities to build unified world models that directly incorporate perception, imagination, and control from raw visual inputs. In parallel, the field of embodied AI has increasingly emphasized the need for unified models that can support perception, prediction, and decision-making in a coherent framework. By enabling agents to "imagine" the visual consequences of future actions, video generation models (VGMs) offer a promising route to this goal agarwal2025cosmos . Recent work has demonstrated how VGMs can be conditioned on action sequences and language instructions, forming the basis for integrated systems that combine simulation, planning, and control yang2023unisim ; RoboDreamer . In this view, VGMs are not merely generative tools, but potential foundations for general-purpose world models. Despite their potential, the existing VGMs world model faces two critical challenges that hinder their application in embodied tasks. First, their generation speed is often too slow for real-time decision-making, particularly due to the multi-step nature of diffusion-based video generation chen2024videocrafter2 ; xing2023dynamicrafter . Second, there is a lack of consistency between imagined visual trajectories and the action instructions needed to execute them, leading to a disconnect between planning and control Du2023-vlp ; cheang2024gr . To address the limitations of existing VGMs in robotic manipulation, we propose Manipulate in Dream (MinD), a hierarchical diffusion-based world model for vision-language manipulation tasks. As shown in Fig. 1 , MinD consists of two modules operating at different temporal resolutions: a low-frequency video generator (LoDiff-Visual) that imagines coherent visual futures, and a high-frequency diffusion policy (HiDiff-Policy) for real-time control. To connect these asynchronous processes, we introduce the Action-Video Diffusion Matching Model (DiffMatcher), which injects intermediate features from LoDiff into HiDiff as dynamic guidance. This design enables MinD to generate consistent visual predictions while supporting low-latency policy execution. However, aligning LoDiff and HiDiff is challenging due to their separate noise schedules and temporal resolutions. When LoDiff and HiDiff are trained independently, the two models tend to diverge—each generating outputs in isolation, leading to semantic inconsistency between imagined videos and executed actions. This issue is further exacerbated during asynchronous inference, where HiDiff consumes LoDiff features from varying noise time steps. To improve alignment, we introduce a co-training strategy: inspired by the diffusion-forcing mechanism chen2024diffusion , we construct paired video features at different noise levels, and apply a spatio-temporal contrastive loss to supervise the DiffMatcher in extracting stable and semantically meaningful features across diffusion steps. Co-training helps the fast action model better understand video features on different time steps, and gives consistent and coherent planning. We validate MinD across multiple manipulation benchmarks and demonstrate its superior performance in planning speed, visual-action consistency, and data efficiency. Our key contributions are: (1) We propose MinD, a hierarchical diffusion-based world model that unifies visual imagination and action policy learning for robotic manipulation. MinD consists of a low-frequency video generator and a high-frequency action policy, enabling coherent visual prediction and real-time control. (2) We introduce a DiffMatcher module comprising two key components: the dual alignment mechanism for bridging asynchronous diffusion schedulers, and the diffusion-forcing technique for aligning latent representations across visual and policy streams via spatio-temporal contrastive learning. (3) We demonstrate that our method achieves state-of-the-art performance across multiple manipulation benchmarks. Furthermore, through cross-analysis, we show that video observations can serve as reliable predictors of task success or failure. 2 Related Work World Models for Embodied AI World models aim to provide agents with internal simulations of the environment to support prediction, planning, and decision-making ha2018world . Early efforts in this direction employed latent dynamics models for model-based reinforcement learning wu2023daydreamer , while more recent approaches integrate vision and language to support open-world reasoning driess2023palm ; brohan2023rt . In the context of embodied AI, such models are increasingly required to generate coherent future states from raw visual observations, enabling long-horizon planning and closed-loop control. However, most prior work either focuses on purely reactive policies or lacks the ability to generate semantically meaningful future observations. Video Generation Models for Imagination and Planning Video generation models (VGMs), especially those based on latent diffusion long2024videodrafter ; sora , have recently shown strong potential in synthesizing realistic and temporally coherent video sequences yang2023unisim ; chen2024videocrafter2 ; xing2023dynamicrafter ; chi2024eva ; li2025manipdreamer . These models can "imagine" multiple future possibilities conditioned on actions or language, making them attractive candidates for visual world modeling RoboDreamer ; agarwal2025cosmos ; huang2025enerverse ; li2025unified ; zhen2025tesseract . However, VGMs are often computationally expensive and suffer from long inference times, which limits their applicability in real-time robotics. Furthermore, their integration with downstream control policies remains underexplored, often resulting in disjointed simulation-control pipelines. Multimodal Policy Learning and Vision-Language Manipulation Recent advancements in vision-language action (VLA) modeling have enabled robots to follow natural language instructions to perform manipulation tasks liu2025hybridvla ; black2024pi_0 .