MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation
Yu Shang Yangcheng Yu Xin Zhang Xin Jin Haisheng Su Wei Wu Yong Li, Tsinghua University Manifold AI Shanghai Jiao Tong University · 2025
Read MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation by Yu Shang Yangcheng Yu Xin Zhang Xin Jin Haisheng Su Wei Wu Yong Li, Tsinghua University Manifold AI Shanghai Jiao Tong University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21797.txt.
Introduction
Robots must turn camera views and spoken orders into exact actions, and world models help them plan by predicting how a scene will change. Models that rebuild every pixel waste effort on detail that does not matter. Compact models save effort but miss fine detail that precise hand work needs. Yangcheng Yu and six co-authors present MoWM in this 2025 paper to get both strengths at once.
MoWM mixes motion-aware features from a compact latent world model with detail from a pixel-space model. The blend helps the robot pick out the visual details that matter for choosing an action. On the CALVIN benchmark and on real robot tasks, MoWM reached top success rates. It also handled new settings better than rivals, and the paper compares the strengths of each feature type for future work.
Catalog record
| CREATOR | Yu Shang Yangcheng Yu Xin Zhang Xin Jin Haisheng Su Wei Wu Yong Li, Tsinghua University Manifold AI Shanghai Jiao Tong University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation Yu Shang Yangcheng Yu Xin Zhang Xin Jin Haisheng Su Wei Wu Yong Li, Tsinghua University Manifold AI Shanghai Jiao Tong University Abstract Embodied action planning is a core challenge in robotics, requiring models to generate precise actions from visual observations and language instructions. While video generation world models are promising, their reliance on pixel-level reconstruction often introduces visual redundancies that hinder action decoding and generalization. Latent world models offer a compact, motion-aware representation, but overlook the fine-grained details critical for precise manipulation. To overcome these limitations, we propose MoWM, a mixture-of-world-model framework that fuses representations from hybrid world models for embodied action planning. Our approach uses motion-aware representations from a latent model as a high-level prior, which guides the extraction of fine-grained visual features from the pixel-space model. This design allows MoWM to highlight the informative visual details needed for action decoding. Extensive evaluations on the CALVIN benchmark demonstrate that our method achieves state-of-the-art task success rates and superior generalization. We also provide a comprehensive analysis of the strengths of each feature space, offering valuable insights for future research in embodied planning. The code is available at: https://github.com/tsinghua-fib-lab/MoWM . 1 Introduction Embodied action planning represents a core research direction in embodied intelligence, aiming to enable robots to generate precise executable actions from environmental observations and language instructions ( Ma et al. 2024 ; Fung et al. 2025 ) . Early methods primarily relied on imitation learning (IL) ( Jang et al. 2022 ; Chi et al. 2023 ) from expert demonstration trajectories; however, such approaches often exhibit limited generalization and struggle to adapt to novel scenarios. With recent advances in large models, Vision-Language-Action (VLA) models ( Kim et al. 2024 ; Black et al. 2024 ; Cheang et al. 2025 ) have emerged as a promising alternative, offering enhanced capabilities in complex task understanding. Despite these improvements, their training paradigm remains fundamentally based on imitation learning and relies on high-quality demonstration data and faces challenges in achieving broad generalization. In parallel, another line of research explores video-based world models ( Hu et al. 2024 ; Feng et al. 2025 ; Liao et al. 2025 ) for action planning. This paradigm involves pre-training a world model on large-scale video datasets to learn general physical dynamics, followed by establishing a mapping between visual observations and robot actions. This approach offers greater data efficiency and is expected to have potential for cross-domain generalization acquired from rich video dynamics learning. Despite its promise, a key limitation of current world model-based embodied planning lies in the visual representation learning. These models typically rely on features from diffusion-based video generation models ( Blattmann et al. 2023 ; Yang et al. 2024 ; Wan et al. 2025 ) pre-trained with pixel-level reconstruction objectives. Such objectives emphasize detailed pixel recovery, yet many robotic tasks do not require perfect reconstruction of all visual elements ( Assran et al. 2025 ) . Uniformly encoding all pixels may introduce irrelevant signals that hinder action decoding, complicate the learning of vision-action mappings, and hurt generalization due to overfitting to some task-irrelevant visual details. Alternatively, another line of research explores latent world models ( Assran et al. 2025 ; Zhou et al. 2024 ; Baldassarre et al. 2025 ) , which learn state transitions in a compressed feature space rather than reconstructing raw pixels. These models employ an encoder to project video sequences into a compact latent representation and a predictor to model dynamics in this space. By focusing on learning motion-aware representations, such methods are more suitable for guiding global action planning. However, they may overlook fine-grained visual details, potentially leading to inaccuracies in tasks requiring dense object interaction. To address the redundancy of visual representations in video world models, we propose MoWM, a hybrid world model framework for effective embodied planning. Our motivation is to leverage the latent world modeling features as a high-level prior to guide the extraction of fine-grained visual knowledge from the pixel space. This not only eliminates redundant low-level visual information but also avoids the loss of crucial details when using coarse-grained latent features alone. To be specific, our method consists of two stages. In the first stage, we individually train a pixel world model based on video diffusion and a latent world model on embodied manipulation data. Both models are trained to predict future states in their respective spaces conditioned on text instructions. In the second stage, we combine the two world models. The latent world model’s representations are used to modulate the pixel world model’s representations, yielding a fused motion-aware low-level visual representation. In this way, we can direct attention to the visual information most relevant to action decoding, which is then fed into an inverse dynamics model for end-to-end action decoding. We evaluate our approach on the standard embodied manipulation benchmarks CALVIN ( Mees et al. 2022 ) , comparing it against imitation learning-based, VLA-based and world model-based action planning methods. Experimental results demonstrate that our proposed MoWM achieves state-of-the-art performance in task success rates, highlighting the significant potential of hybrid world modeling for embodied action planning. Furthermore, we provide an in-depth analysis of the pixel-level and latent-level visual features during action planning, offering practical guidance for model selection in real-world applications. The main contributions of this work are summarized as follows: • We propose MoWM, a hybrid world model architecture for embodied action planning, which integrates the motion-aware advantages of a latent world model with the low-level detail generation capabilities of a pixel space world model. • We explored the interactions of visual features from both pixel and latent space world models. We propose an innovative fusion scheme and offer a comprehensive analysis of its effectiveness. • Extensive experimental results demonstrate the superiority of our proposed method in both task success rate and generalization to unseen scenarios in embodied action planning. 2 Related Works 2.1 Vision-language-action models for embodied planning Vision-Language-Action (VLA) models, which use a large language model backbone enhanced with a vision encoder and an action decoder to predict executable robot actions based on the text instruction, current observation and robot state. Representatively, Pi0 ( Black et al. 2024 ) uses a pre-trained VLM as its foundation and adds an action expert to map VLM tokens to the action space, which is trained with a flow matching objective. Octo ( Team et al. 2024 ) employs a transformer-based LLM to deal with interleaved language, visual observation and action tokens, enabling it to flexibly adapt to new observations and action types. More recent works have focused on improving the complex planning and reasoning ability of VLAs ( Zhao et al. 2025 ; Intelligence et al. 2025 ; Huang et al. 2025 ) . For example, CoT-VLA ( Zhao et al. 2025 ) introduces intermediate thinking steps such as goal state prediction to enhance action planning. Despite these advancements, VLA models face several limitations. The high cost of collecting teleoperation data makes it difficult to cover a wide range of tasks and diverse scenarios, leading to poor generalization beyond the training environment. Furthermore, their reliance on imitation learning limits their ability to perform counterfactual reasoning or handle complex tasks. 2.2 World models for