Enhancing Policy Learning with World-Action Model
Yuci Han, Alper Yilmaz · 2026
Read Enhancing Policy Learning with World-Action Model by Yuci Han, Alper Yilmaz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28955.txt.
Introduction
Han and Yilmaz present the World-Action Model, a world model that reasons over future images and the actions behind state changes. Standard world models train on image prediction alone. Their model adds an inverse dynamics aim to DreamerV2, so it predicts actions from latent state shifts. That aim makes the learned features capture action relevant structure. On eight CALVIN manipulation tasks, a diffusion policy trained on these features rose from 59.4 to 71.2 percent success. After PPO fine-tuning, success reached 92.8 percent, with far fewer training steps than the baseline.
Catalog record
| CREATOR | Yuci Han, Alper Yilmaz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26940 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Policy Learning with World-Action Model Yuci Han, Alper Yilmaz Abstract This paper presents the World-Action Model (WAM), an action-regularized world model that jointly reasons over future visual observations and the actions that drive state transitions. Unlike conventional world models trained solely via image prediction, WAM incorporates an inverse dynamics objective into DreamerV2 that predicts actions from latent state transitions, encouraging the learned representations to capture action-relevant structure critical for downstream control. We evaluate WAM on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark. We first pretrain a diffusion policy via behavioral cloning on world model latents, then refine it with model-based PPO inside the frozen world model. Without modifying the policy architecture or training procedure, WAM improves average behavioral cloning success from 59.4% to 71.2% over DreamerV2 and DiWA baselines. After PPO fine-tuning, WAM achieves 92.8% average success versus 79.8% for the baseline, with two tasks reaching 100%, using 8.7 × 8.7\times fewer training steps. Index Terms —Reinforcement Learning, Robotics and World Model, Diffusion Policy. I Introduction Fig. 1 : (a) Standard world models predict only future observations, treating actions solely as conditioning inputs. (b) Our World-Action Model adds an inverse dynamics head that jointly predicts observations and actions during training. (c) The action-aware world model serves as a learned simulator for offline policy fine-tuning via PPO. World models have become a critical tool for learning environment dynamics in robotics, allowing agents to plan and refine behaviors through imagined rollouts rather than expensive real-world interactions [ 5 , 7 , 6 , 8 , 9 ] . These models compress high-dimensional observations into compact latent spaces that capture the temporal and physical structures of environments, enabling sample-efficient policy learning through imagined experiences in latent space. Recent work has shown that policies trained entirely within learned world models can reach strong performance on robotic manipulation benchmarks [ 4 , 12 , 3 ] , and in some cases transfer directly to physical hardware without further fine-tuning [ 17 ] . However, conventional world models are trained solely to predict future observations [ 5 , 7 , 6 , 8 , 17 , 12 ] . They forecast future visual states conditioned on past observations and actions, but never explicitly model the actions themselves. This asymmetry limits the quality of learned representations. The world model encodes each observation into a latent state z_{t} , which serves as the direct input to the downstream diffusion policy in model-based pipelines such as DiWA [ 3 ] —yet z_{t} is optimized only for pixel prediction and KL regularization, with no explicit pressure to encode action-relevant structure. Recent work on unified action-world models [ 2 ] has begun to address this gap by jointly generating actions and images within a single autoregressive architecture, but these approaches rely on large foundation models and fundamentally redesign the architecture rather than improving the representations learned by existing world models. We take a different and complementary approach: rather than redesigning the architecture, we augment the training objective of DreamerV2 [ 8 ] with an inverse dynamics head that predicts actions from consecutive encoder embeddings, regularizing the learned representations toward action-relevant structure. We argue that requiring the model to predict the action that caused a given state transition forces the encoder to capture fine-grained information about how the environment responds to agent behavior. This information is critical for downstream policy learning but is overlooked when the training signal comes only from observation reconstruction. This idea draws on inverse dynamics models in self-supervised representation learning [ 14 , 1 ] , where predicting actions between consecutive states yields representations focused on controllable aspects of the environment while filtering out task-irrelevant distractors. We present the World-Action Model (WAM) and evaluate it on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark [ 11 ] . We first pretrain a diffusion policy via behavioral cloning on world model latents, then refine it with model-based PPO using the frozen world model as a simulator. Compared to the DiWA baseline [ 3 ] , WAM improves average behavioral cloning success from 59.4% to 71.2% and achieves 92.8% average success after PPO fine-tuning versus 79.8%, with two tasks reaching 100%, using 8.7\times fewer training steps. Our main contributions are: • We present WAM, a lightweight extension of DreamerV2 that augments the training objective with an inverse dynamics head, explicitly regularizing latent representations toward action-relevant structure. • We demonstrate that action regularization improves world model generation quality on the CALVIN benchmark. WAM matches or exceeds DreamerV2 [ 8 ] on LPIPS, PSNR, SSIM, and FVD with fewer training steps. • We show that the improved representations enhance downstream policy learning, outperforming the DiWA baseline on both behavioral cloning and PPO fine-tuning across all eight tasks on the CALVIN benchmark. II Related Works II-A World Models World models learn a predictive representation of environment dynamics, enabling agents to optimize policies through imagined trajectories without requiring additional real-world data collection. Ha and Schmidhuber [ 5 ] introduced an early approach encoding observations into a latent space with an RNN for dynamics modeling. PlaNet [ 7 ] proposed the Recurrent State-Space Model (RSSM), combining deterministic and stochastic components for long-horizon prediction. The Dreamer series [ 6 , 8 , 9 ] progressively extended RSSM-based latent imagination for behavior learning. DreamerV2 replaced Gaussian latents with categorical representations, while DreamerV3 introduced robustness techniques that allowed a single set of hyperparameters to work across diverse domains. However, these world models are trained purely as observation predictors without explicitly reasoning about the actions that drive state transitions. Our work introduces action regularization into the world model, jointly predicting both actions and future observations to better capture environment dynamics. II-B World Models for Policy Learning Beyond modeling dynamics, world models can be directly used as complementary environments for policy optimization, where agents learn behaviors through imagined rollouts within the learned model. The Dreamer series [ 6 , 8 , 9 ] first demonstrated that actor-critic policies can be trained entirely through imagined latent rollouts, and DayDreamer [ 17 ] demonstrated this paradigm on physical robots. In offline settings, DITTO [ 4 ] proposed imitation learning by optimizing a distance metric in a frozen world model’s latent space, and LUMOS [ 12 ] extended this to language-conditioned multi-task learning. DiWA [ 3 ] fine-tuned diffusion policies via PPO within a learned world model, achieving sample efficiency orders of magnitude beyond model-free baselines. More recently, WMPO [ 19 ] introduced pixel-based world models for on-policy GRPO optimization of Vision-Language-Action models, and World4RL [ 10 ] employed diffusion-based world models for end-to-end policy refinement. While these works confirm the effectiveness of world-model-based policy learning, they rely on world models trained solely for observation prediction. Our approach enriches the world model with action prediction, providing a stronger training signal for downstream policy optimization. III Method III-A World-Action Model We propose to regularize the world model with an inverse dynamics objective, producing representations that are simultaneously predictive of visual dynamics and aware of the actions driving state transitions. III-A 1 World Model Backbone WAM builds on the RSSM architecture from DreamerV2 [ 8 ] . A dual-stream CNN encoder processes static and gripper camera images,