AdaWorld: Learning Adaptable World Models with Latent Actions
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan · 2025
Read AdaWorld: Learning Adaptable World Models with Latent Actions by Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18938.txt.
Introduction
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan teach world models to adapt fast in a 2025 paper. World models predict how a scene will change under actions. Most need huge sets of action-labeled data to train. New settings with new actions are hard for them to learn.
Their model, AdaWorld, learns hidden actions from raw video. It pulls out the key change between frames on its own. A prediction model then works under those hidden actions. The model can pick up new actions with few tries and light tuning. Tests across many settings show top scores for scene quality. Visual planning tasks also improved with the method.
Catalog record
| CREATOR | Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18938
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaWorld: Learning Adaptable World Models with Latent Actions Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan Abstract World models aim to learn action-controlled future prediction and have proven essential for the development of intelligent agents. However, most existing world models rely heavily on substantial action-labeled data and costly training, making it challenging to adapt to novel environments with heterogeneous actions through limited interactions. This limitation can hinder their applicability across broader domains. To overcome this limitation, we propose AdaWorld , an innovative world model learning approach that enables efficient adaptation. The key idea is to incorporate action information during the pretraining of world models. This is achieved by extracting latent actions from videos in a self-supervised manner, capturing the most critical transitions between frames. We then develop an autoregressive world model that conditions on these latent actions. This learning paradigm enables highly adaptable world models, facilitating efficient transfer and learning of new actions even with limited interactions and finetuning. Our comprehensive experiments across multiple environments demonstrate that AdaWorld achieves superior performance in both simulation quality and visual planning. adaptable-world-model.github.io 1 Introduction Figure 1: Different world model learning paradigms. Prior methods often require expensive labeling and training to achieve action controllability in new environments. To overcome this, we introduce latent actions as a unified condition for action-aware pretraining from videos, enabling highly adaptable world modeling. Our world model, dubbed AdaWorld , can readily transfer actions across contexts without training. By initializing the control interface with the corresponding latent actions, AdaWorld can also be adapted into specialized world models efficiently and achieve significantly better planning results than the action-agnostic baseline. Intelligent agents should perform effectively across various tasks ( Reed et al., 2022 ; Lee et al., 2022 ; Durante et al., 2024 ; Raad et al., 2024 ) . A promising solution to this objective is developing world models that can simulate different environments ( Wu et al., 2023 ; Wu et al., 2024 ; Yang et al., 2024c ; Hansen et al., 2024 ) . Recent world models are typically initialized from pretrained video models ( Xiang et al., 2024 ; Agarwal et al., 2025 ; He et al., 2025 ) . Despite improved generalization, these models still require substantial action labels and high training costs to acquire precise action controllability. While pseudo labels can be annotated for videos ( Baker et al., 2022 ; Zhang et al., 2022 ) , defining a unified action format for general environments is challenging. As a result, existing methods often require costly training when adapting to new environments with varying action specifications ( Gao et al., 2024 ; Chi et al., 2024 ; Che et al., 2025 ) . These limitations pose great challenges for transferring and learning new actions based on limited interactions and finetuning. As humans, we can estimate the effects of different actions through limited experiences ( Ha & Schmidhuber, 2018 ) . This ability likely arises from our internal representations of actions learned from extensive observations ( Rizzolatti et al., 1996 ; Romo et al., 2004 ; Dominici et al., 2011 ) . These common knowledge can be reused across different contexts and associated with specific action spaces efficiently ( Rybkin et al., 2019 ; Schmeckpeper et al., 2020 ; Sun et al., 2024 ) . Consequently, humans can easily transfer observed actions to various contexts and imagine the transitions of new environments through a few interactions ( Poggio & Bizzi, 2004 ) . These insights motivate us to ponder: can we achieve human-like adaptability in world modeling by learning transferrable action representations from observations? In this paper, we propose AdaWorld , an innovative pretraining approach for highly adaptable world models. Unlike previous methods that only pretrain on action-agnostic videos, we argue that incorporating action information during pretraining will significantly enhance the adaptability of world models. As illustrated in Figure 1 , the adaptability of AdaWorld primarily manifests in two aspects: (1) Given one demonstration of an action, AdaWorld can readily transfer that action to various contexts without further training. (2) It can also be efficiently adapted into specialized world models with raw action inputs via minor interactions and finetuning, enabling more effective planning in various environments. AdaWorld consists of two key components: a latent action autoencoder that extracts actions from unlabeled videos, and an autoregressive world model that takes the extracted actions as conditions. Our main challenge is that in-the-wild videos often involve complicated contexts ( e.g . , colors and textures), which hinders effective action recognition. To overcome this challenge, we introduce an information bottleneck design to our latent action autoencoder. Specifically, the latent action encoder extracts a compact encoding from two consecutive frames. We refer to this encoding as latent action hereinafter, as it is used to represent the transition between these two frames. Based on the latent action and the former frame, the latent action decoder makes its best effort to predict the subsequent frame. By minimizing the prediction loss using the minimal information encoded in the latent action, our autoencoder is encouraged to disentangle the most critical action from its context. Unlike previous methods ( Bruce et al., 2024 ; Chen et al., 2024b ; Ye et al., 2025 ) that focus on playability and behavior cloning, we compress the latent actions into a continuous latent space to maximize expressiveness and enable flexible composition. We find that our latent actions are context-invariant and can be effectively transferred across different contexts. We then pretrain an autoregressive world model that conditions on the latent actions. Thanks to the strong transferability of our latent actions, the resulting world model is readily adaptable to various environments. In particular, since our world model has learned to simulate different actions represented by any latent actions, adapting to a new environment is akin to finding the mapping of corresponding latent actions for its action space. Given one demonstration of an action, our model can readily transfer the demonstrated action by extracting the latent action with latent action encoder and reusing it across different contexts. When action labels are provided, we can similarly obtain their latent actions and initialize the control interface efficiently. This enables us to adapt our model to specialized world models with minimal finetuning. When only a limited number of interactions are available ( e.g . , 50 interactions), our approach is significantly more efficient than pretraining from action-agnostic videos. Note that our approach is as scalable as existing video pretraining methods ( Seo et al., 2022 ; Mendonca et al., 2023 ; Wu et al., 2023 ; Agarwal et al., 2025 ; He et al., 2025 ; Yu et al., 2025 ) . To enhance the generalization ability of AdaWorld, we collect a large corpus of videos from thousands of environments through automated generation. The resulting dataset encompasses extensive interactive scenarios, spanning from ego perspectives and third-person views to virtual games and real-world activities. After action-aware pretraining at scale, we show that the adaptability of AdaWorld can seamlessly generalize to a wide variety of domains. In summary, we make the following contributions: • We present AdaWorld, an autoregressive world model that is highly adaptable across various environments. It can readily transfer actions to different contexts and