Latent Actions from Factorized Transition Effects under Agent Ambiguity
Heejeong Nam, Chandradithya S Jonnalagadda, Harshit Aggarwal, Eric Xu, Randall Balestriero · 2026
Read Latent Actions from Factorized Transition Effects under Agent Ambiguity by Heejeong Nam, Chandradithya S Jonnalagadda, Harshit Aggarwal, Eric Xu, Randall Balestriero (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30544.txt.
Introduction
Heejeong Nam and four colleagues learn action-like signals from videos where many things move at once. Latent action models infer actions from changes between frames. In busy scenes, distractors and camera motion pollute those changes. The true source of any change becomes ambiguous. Their Observed Transition Factorization splits each transition into primitives. A sparse set of reusable motion pieces describes what changed.
Those primitives become the interface for learning latent actions. One model wraps them in the standard inverse–forward dynamics setup. A second variant predicts future states in a frozen feature space. It needs no pixel decoder. The primitives transfer zero-shot across controlled changes. Carrier and body-shape shifts both preserve their usefulness. Downstream policy learning matches or beats the baselines. That holds under complex transition ambiguity. Structuring change into reusable effects is the paper's core bet. It pays off where raw latent actions confuse sources.
Catalog record
| CREATOR | Heejeong Nam, Chandradithya S Jonnalagadda, Harshit Aggarwal, Eric Xu, Randall Balestriero |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30544
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Actions from Factorized Transition Effects under Agent Ambiguity Heejeong Nam, Chandradithya S Jonnalagadda, Harshit Aggarwal, Eric Xu, Randall Balestriero Abstract Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed-transition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse–forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zero-shot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity. https://hazel-heejeong-nam.github.io/LAM/ 1 Introduction World models ( 12 ) for planning and control have been evaluated in settings where the actor and the action space are known, such as a robotic manipulator ( 14 ; 13 ) . In this regime, the model predicts the next state from the current state, and can therefore, separate the controlled source of variation from autonomous background dynamics or moving distractors at least implicitly. Figure 1: Motivation for agent ambiguity in observation-only latent action learning. Unlike action-conditioned world models, which receive the actor and action as part of the input, LAMs infer action-like latents only from visual transitions. Recently, motivated by the fact that most online videos do not provide frame-level action labels or proprioceptive signals, Latent Action Models (LAMs) ( 29 ; 39 ) have been proposed to learn action-like proxies directly from observation transitions. A typical LAM encodes an observation pair (x_{t},x_{t+1}) into a latent action and uses it to predict x_{t+1} from x_{t} . This formulation is appealing because it removes the need for action annotations, but it also changes what the model can identify. However, a LAM does not observe pure actions themselves. It observes the visual effects that actions, dynamics, rendering, and other transition sources produce. This distinction is often hidden in controlled settings where the agent is visually dominant and non-agent elements are static ( 42 ; 23 ) . In more general scenes, however, a single transition may mix agent motion with passive objects, articulated parts, camera/background dynamics, and distractors ( 26 ; 37 ; 1 ; 17 ) . From the model’s perspective, all of these changes arrive through the same channel: the pixels at time t and t+1 . Without additional supervision or inductive biases, the model has no direct basis for deciding which observed changes should be attributed to the controlled agent, or which changes should be selected as relevant to the desired action abstraction, as depicted in Figure 1 . A natural response to this ambiguity is to treat it as an object selection problem ( 37 ; 1 ; 17 ) , identifying the controllable entity in order to learn its latent action. However, relying on segmentation or object-centric priors limits scalability and imposes rigid assumptions. More importantly, objects are not always the appropriate unit of abstraction, as motion effects can manifest within, across, or independently of strict object boundaries. Recognizing these limitations, we step back from the standard operational view of LAMs as simple transition bottlenecks ( 29 ) and ask what can be identified from observation-only transitions, and what latent actions should represent under agent ambiguity. We interpret the pathway from action to observation at three levels as in Figure 2 : the true underlying action induces a physical or environment state change, which then produces visual transition effects in observation space. Under observation-only learning, the most reliable target is not necessarily the true action itself, but the structure of these observed transition effects. We therefore take a bottom-up approach to latent action learning under ambiguity. We first introduce Observed Transition Factorization (OTF) , which represents each transition as a sparse composition of reusable observed-transition primitives. Rather than compressing all changes into a monolithic embedding, OTF decomposes the transition into localized observed effects using a compositional VQ-style codebook. The resulting representation records which transition effects occur and where they are expressed. Using this factorized transition interface, we instantiate OTF-LAM , which forms state-aware action-like latents within the standard inverse–forward dynamics framework. A frozen OTF module first extracts reusable observed-transition primitives, and a state-aware aggregator abstracts these primitives into a compact latent action for prediction. We also introduce OTF-LAM-Dino , a decoder-free variant that predicts future states in a frozen DINO representation space, combining a reusable transition vocabulary with a reusable visual state representation. Empirically, OTF primitives transfer zero-shot across held-out visual carriers and cross-morphology shifts on Moving MNIST ( 32 ) and DCS ( 33 ) , indicating reusable transition structure rather than environment- or embodiment-specific templates. For downstream control, we map learned latent actions to true actions using a small action-labeled dataset. The resulting latent action interfaces match or outperform LAM baselines under complex transition ambiguity, demonstrating the effectiveness of factorized observed transitions for policy learning. OTF-LAM-Dino further improves over decoder-based OTF-LAM, showing that reusable transition vocabularies complement frozen visual state representations. 2 From Actions to Observed Effects LAMs trained from observation-only transitions do not recover actions directly, but representations of their visible consequences after being mediated by state, embodiment, dynamics, and rendering. As a result, the learned latent space is not necessarily action-equivalent. 2.1 The Forward Generative Chain Figure 2: From causal generation to inverse inference. Blue arrows denote the causal pathway from action to visual change, while red arrows denote the inverse problem faced by observation-only latent action learning. We view a transition as a chain of three stages: \underbrace{a_{t},\xi_{t}}_{\text{trainsition sources}}\;\xrightarrow{F_{\mathcal{E}}}\;\underbrace{\Delta s_{t}}_{\text{physical state change}}\;\xrightarrow{R_{\mathcal{E}}}\;\underbrace{\Delta x_{t}}_{\text{visual transition effect}} as in Figure 2 . Here, a_{t} denotes the controlled action, while \xi_{t} denotes uncontrolled transition sources such as camera motion, distractor motion, background dynamics, or passive environment changes. The environment \mathcal{E} includes the embodiment, dynamics, camera, and other environment-specific factors. Let F{\mathcal{E}} be the state transition function and R_{\mathcal{E}} be the observation or rendering map. The physical state change may be written as \Delta s_{t}=F_{\mathcal{E}}(s_{t},a_{t},\xi_{t}), while the visual transition effect may be written as \Delta x_{t}=R_{\mathcal{E}}(s_{t},\Delta s_{t},\xi_{t}). Equivalently, the observed transition is generated by the composition \Delta x_{t}=R_{\mathcal{E}}\bigl(s_{t},F_{\mathcal{E}}(s_{t},a_{t},\xi_{t}),\xi_{t}\bigr). (1) Intuition. This setup clarifies why a latent action cannot generally recover the true action. The observed transition is generated by Eq. 1 . For fixed observations, the map from a_{t} to \Delta x_{t} is generally not injective, and \Delta x_{t} may also contain changes not caused by a_{t} at all. Therefore, an observation-trained latent action can at best identify an equivalence class of causes and observed effects. 2.2 Observed Effects as a Transferable Intermediate Actions and physical state changes are often environment-specific. Actions depend on the control interface, such as torques, joint commands, or button inputs, while physical changes depend on state, embodiment, contact, and dynamics. Thus, neither level provides an obvious reusable primitive across heterogeneous embodiments. Observed visual effects provide a weaker but more transferable abstraction. They describe how change appears