ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance CCS: Computing methodologies Vision for robotics
Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, Shanghang Zhang · 2025
Read ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance CCS: Computing methodologies Vision for robotics by Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, Shanghang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16464.txt.
Introduction
Ying Li and co-authors improve the world models that let robots imagine the results of their actions, in a 2025 paper. Such models generate video of a manipulation as it unfolds, and two things matter: following the instruction, and looking right. Earlier work split instructions into primitive steps but ignored how the steps relate, and it skipped visual guidance such as depth. Their model, ManipDreamer, fixes both gaps.
The instruction is drawn as an action tree, and each node carries a learned embedding, so a command collects the embeddings along its path and keeps the step relationships. A visual adapter feeds depth and semantic guidance into the generator, steadying time consistency and physical plausibility. On standard manipulation benchmarks the model lifted video quality scores on both familiar and new tasks. It also raised real task success by about two and a half percent on average across six test tasks.
Catalog record
| CREATOR | Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, Shanghang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16464
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance CCS: Computing methodologies Vision for robotics Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, Shanghang Zhang Abstract . While recent advancements in robotic manipulation video synthesis have shown promise, significant challenges persist in ensuring effective instruction-following and achieving high visual quality. Recent methods, like RoboDreamer, utilize linguistic decomposition to divide instructions into separate lower-level primitives. They condition the world model on these primitives to achieve compositional instruction-following. However, these separate primitives do not take into account the relationships that exist between them. Furthermore, recent methods neglect valuable visual guidance, including depth and semantic guidance, both crucial for enhancing visual quality. This paper introduces ManipDreamer, an advanced world model based on the action tree and visual guidance. To better learn the relationships between instruction primitives, we represent the instruction as the action tree and assign embeddings to tree nodes. Therefore, each instruction can acquire its embeddings by navigating through the action tree. The instruction embeddings can be used to guide the world model. To enhance visual quality, we combine depth and semantic guidance by introducing a visual guidance adapter compatible with the world model. This visual adapter enhances both the temporal and physical consistency of video generation. Based on the action tree and visual guidance, ManipDreamer significantly boosts the instruction-following ability and visual quality. Comprehensive evaluations on robotic manipulation benchmarks reveal that ManipDreamer achieves large improvements in video quality metrics in both seen and unseen tasks, with PSNR improved from 19.55 to 21.05, SSIM improved from 0.7474 to 0.7982 and reduced Flow Error from 3.506 to 3.201 in unseen tasks, compared to the recent RoboDreamer model. Additionally, our method increases the success rate of robotic manipulation tasks by 2.5% in 6 RLbench tasks on average. 1. Introduction Video generation technologies have developed into three main technical paradigms based on control modalities: text-to-video (T2V), image/video-to-video (I2V), and other modalities to video. Early works focus on generating video content that is conditioned by textual input (T2V) ( Pan et al. 2017 ; Marwah et al. 2017 ; Li et al. 2018 ; Gupta et al. 2018 ; Liu et al. 2019 ; Hong et al. 2022 ; Ho et al. 2022 ; Singer et al. 2022 ) . Recent advancements in I2V ( Xing et al. 2024b ; Peng et al. 2025 ; Wang et al. 2025 ) and V2V ( Agarwal et al. 2025 ) techniques are focused on generating high-fidelity videos with complex motions and interactions. These developments illustrate the potential of using video generative models as world models. Other modalities of video generation methods have been proposed, such as motion trajectory-controlled approaches ( Wang et al. 2024b ; Yin et al. 2023 ) that enable precise perspective movement, depth-guided synthesis ( He et al. 2023 ; Xing et al. 2024a ) that enhances realism and spatial coherence, and pose-driven ( Karras et al. 2023 ; Hu 2024 ; Wei et al. 2024a ; Wei et al. 2024b ; Chen et al. 2024 ; Chen et al. 2025 ) methods that generate human animation videos. Creating high-quality datasets is essential for data-intensive fields, such as embodied intelligence ( Zhou et al. 2024 ; Wang et al. 2024a ; Gao et al. [n. d.] ; Chi et al. 2024 ) and autonomous driving ( Ni et al. 2025 ; Russell et al. 2025 ; Zhao et al. 2024 ; Huang et al. 2024 ; Wei et al. 2024c ) , where collecting physical data is often prohibitively expensive. Recent developments in generative models have enhanced the creation of world models, providing a cost-effective solution for generating synthetic data. In autonomous driving, world models have shown impressive versatility by integrating various control signals to create physically consistent scenarios. Many works ( Gao et al. 2024 ; Gao et al. 2023 ; Ni et al. 2025 ; Russell et al. 2025 ; Zhao et al. 2024 ; Lu et al. 2024b ) utilize control signals such as velocity profiles, steering angles, and traffic rule embeddings to condition the generation process, which is essential for applications like safety validation and rare-scenario augmentation. Methods ( Ni et al. 2025 ) based on diffusion transformers leverage MAE-style feature learning to generate multi-view driving videos based on driving actions and text prompts. Large-scale models ( Russell et al. 2025 ) integrate LiDAR point clouds, BEV semantic maps, and traffic agent trajectories to generate geographically diverse multi-camera streams, facilitating closed-loop simulations for planning algorithms. 4D scene reconstruction ( Zhao et al. 2024 ) utilizes world model priors to synthesize novel trajectory videos, showcasing the synchronized spatial-temporal evolution of dynamic objects and static environments. In contrast, robotic video generation focuses on task-oriented synthesis guided by specific inductive biases. Typically, a robotic world model takes a scene snapshot and generates robotic videos conditioned on language instructions. Unlike approaches in autonomous driving, most existing methods rely solely on the RGB modality, which provides limited multi-modal information. Robodreamer ( Zhou et al. 2024 ) breaks down language instructions into lower-level primitives for the synthesis of compositional manipulation. Wang et al. ( Wang et al. 2024a ) combine gesture-based spatial references with deictic language to clarify ambiguities in open-world environments. For action-conditioned generation, FLIP ( Gao et al. [n. d.] ) introduces flow-centric planning, where optical flow fields act as intermediate representations of actions to guide video generation. Recent efforts, such as EVA ( Chi et al. 2024 ) , have introduced an Understand–Imagine–Predict world model loop. However, it still relies solely on RGB images as the input perception modality, limiting its ability to capture richer spatial and semantic context. By overcoming these limitations, world models can achieve a more accurate understanding of spatial affordances and physical constraints, ultimately enhancing their performance in sim-to-real transfer and complex manipulation tasks. While recent advancements in robotic world modeling show promise, significant challenges remain in ensuring effective instruction following and achieving high visual quality. This paper proposes ManipDreamer, an advanced robotic manipulation world model based on the action tree and multi-modal visual guidance. For instruction following, instead of breaking down language instructions into lower-level primitives, we represent each instruction as an action tree. We assign embeddings to the nodes of the action tree, allowing each instruction to derive its own embeddings through a traversal of the tree. These instruction embeddings are then utilized to guide the world model. Regarding visual quality, we observe that generated robotic videos exhibit significant artifacts. These issues include structural anomalies in the robotic arms, geometric deformations of objects, and temporal inconsistencies such as cross-frame attribute mutations. As a result, the fidelity of the synthetic content is compromised, affecting both spatial continuity and temporal coherence. We introduce a visual adapter to leverage the depth and semantic information in the video generation process to enhance the visual quality. The adapter features a multi-scale encoder that progressively incorporates depth-aware geometric constraints into the video diffusion pipeline. Our experiments demonstrate that ManipDreamer significantly improves the performance of robotic manipulation world models. Specifically, it achieves notable enhancements in video quality metrics such as PSNR, SSIM, and Flow Error, surpassing the recent RoboDreamer model. Additionally, our method shows