DiLA : Disentangled Latent Action World Models
Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang, Si Wu · 2026
Read DiLA : Disentangled Latent Action World Models by Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang, Si Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15725.txt.
Introduction
Tianqiu Zhang and four colleagues present DiLA, a disentangled latent action model. Latent action models learn models from unlabeled video by inferring actions between frames, facing a trade-off between abstraction and fidelity. Past work uses two-stage training with pretrained models or limits predictions to optical flow. DiLA splits layouts into a structure path and detail into a content path, driven by the bottleneck. The synergy yields a semantically structured action space with no loss of generative quality. DiLA tops rivals on generation quality, action transfer, planning, and interpretability.
Catalog record
| CREATOR | Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang, Si Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62813 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DiLA : Disentangled Latent Action World Models Tianqiu Zhang, Muyang Lyu, Yufan Zhang, Fang Fang, Si Wu Abstract Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA , a novel Di sentangled L atent A ction world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning. disentangled-latent-action-world-models.github.io 1 Introduction World models ( 14 ; 48 ; 21 ; 22 ) have emerged as a cornerstone for autonomous agents ( 41 ) , enabling planning ( 46 ) , simulation ( 26 ) , and policy learning ( 22 ; 24 ; 23 ) in complex environments. By learning to predict future states, these models implicitly capture the underlying latent dynamics of the physical world ( 16 ; 2 ; 65 ; 1 ) . However, traditional approaches rely heavily on action-labeled datasets, which are scarce and expensive to scale compared to the vast availability of unlabeled video data ( 52 ) . To bridge this gap, Latent Action Models (LAMs) ( 5 ; 44 ) have been introduced to infer latent actions directly from unlabeled videos, serving as surrogates for explicit control signals ( 62 ; 9 ; 6 ) . A LAM consists of two core components: an Inverse Dynamics Model (IDM), which extracts latent actions from consecutive frames, and a Forward Dynamics Model (FDM), which predicts future states by conditioning past observations on these inferred actions. Figure 1 : Co-evolving of latent actions and disentanglement. To resolve the "LAM Trade-off", DiLA jointly learns abstract latent actions and content-structure disentanglement. By imposing a restricted predictive bottleneck, the latent action model drives the disentanglement of spatial structures from content semantics. Conversely, this disentanglement provides structural layout inputs that facilitate the learning of highly abstract latent actions. Despite their promise, current LAMs face a fundamental dilemma, which we term the " LAM Trade-off " ( 15 ; 35 ) : the tension between action abstraction and generation quality. To ensure actions are transferable and abstract, models typically impose strong predictive bottlenecks, such as Vector Quantization ( 51 ; 5 ; 44 ; 62 ) or variational bottleneck ( 29 ; 15 ) . While these priors encourage abstraction, they often disrupt the intrinsic manifold of the latent action space, leading to over-simplification and degraded video generation fidelity ( 17 ) . Conversely, relaxing these bottlenecks improves generation but yields entangled representations where latent actions capture irrelevant visual details rather than pure dynamics ( 38 ) . Most existing approaches prioritize abstraction at the expense of generation quality ( 44 ; 62 ; 9 ) . They often adopt a disjoint, two-stage training paradigm, discarding the FDM in favor of a separate, pre-trained diffusion model for video generation ( 5 ; 15 ) . To learn more abstract latent actions, some recent works extract optical flow or depth maps as inputs or supervision targets ( 28 ; 38 ; 3 ) . Conceptually, these methods function as an explicit separation of motion-related spatial layouts from content details. This observation motivates our investigation into two pivotal questions: 1. Can disentanglement learning and latent action learning be co-optimized? 2. How can we simultaneously achieve abstract latent actions and high-fidelity prediction? In this paper, we argue that the key to resolving the "LAM Trade-off" lies in disentanglement . We propose DiLA , a novel D isentangled L atent A ction world model that learns to decouple video sequences into structure (dynamics-relevant spatial layout) and content (dynamics-irrelevant visual appearance and texture). The separation is functional: features needed to model latent dynamics under the prediction bottleneck are assigned to the structure pathway. As illustrated in Fig. 1 , instead of learning latent actions from entire visual features, DiLA forces the latent action to predict only the structural layout dynamics. This constraint facilitates the learning of content-invariant latent actions and further enforces disentanglement: to minimize structure prediction error, the model is incentivized to distill motion dynamics into the latent action while offloading static visual details to a separate content pathway. The structure pathway captures motion-related spatial information (e.g., positions and shapes), whereas the content pathway maintains visual details for high-fidelity generation. We show that while difficult individually, the co-evolution of disentanglement and latent action learning creates a synergistic loop that significantly enhances representation learning. To validate the capabilities of DiLA , we conducted experiments on a large-scale suite of datasets covering human activity, robot manipulation, and outdoor navigation. Our benchmarks demonstrate superior performance against leading methods, particularly in maintaining high generation quality while transferring actions across different embodiments. We also show that DiLA learns a structured, interpretable latent action space that effectively supports downstream visual planning. Collectively, these results confirm that disentangling structure from content is key to achieving both abstract action learning and high-fidelity generation. Our contributions are summarized as follows: • We present DiLA , the first disentangled latent action world model that reconciles the inherent trade-off between latent action abstraction and generation fidelity via content-structure disentanglement. • Co-evolution of latent action and disentanglement. The predictive bottleneck acts as a driving force to isolate spatial layouts from appearance. In turn, this separation facilitates the learning of abstract latent action. • We demonstrate DiLA ’s capabilities through extensive experiments, covering cross-embodiment action transfer, visual planning, and the interpretable analysis of latent manifolds on out-of-distribution datasets. 2 Related Works Figure 2 : Architecture of DiLA . Video features extracted via DINOv2 and a ST-Transformer are decoupled into two pathways. The structure pathway learns abstract latent actions to predict the next structural state \hat{\boldsymbol{s}}_{t+1} under an information bottleneck constraint. The content pathway processes features via Mamba to maintain historical context. A fusion decoder combines the predicted structure with content and initial-frame conditioning to reconstruct the target DINOv2 embedding, visualized using a pretrained RAE decoder. Latent action models. World models rely on explicit action labels to predict future states ( 21 ; 31 ; 1 ) . Latent Action Models (LAMs) mitigate this by inferring latent actions directly from videos ( 5 ; 44 ) . A challenge in LAMs is learning an abstract latent action space. Most approaches impose information bottlenecks to constrain these representations. For instance, Vector Quantization (VQ) is widely used to learn discrete latent spaces ( 5 ; 44 ; 62 ; 9 ; 7 ; 8 ; 43 ; 53 )