Why Latent Actions Fail, and How to Prevent It

Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee · 2026

Read Why Latent Actions Fail, and How to Prevent It by Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20223.txt.

Introduction

Jung Min Lee and three co-workers explain latent action failures on wild videos. Latent action models learn action codes from video by compressing frame changes, but wild frames also carry outside state such as clutter, adding changes unrelated to actions. A linear model covering outside state shows the rebuild goal makes latent actions encode outside facts from future frames. Learning in a space focused on own parts curbs the noise, and helper goals such as action supervision keep latent actions steady across states.

Catalog record

CREATORJung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49193
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Latent Actions Fail, and How to Prevent It Jung Min Lee, Taehyun Cho, Li Zhao, Jungwoo Lee Abstract Latent action models (LAMs) aim to learn action-like representations from unlabeled videos by compressing frame-to-frame changes. The frames of in-the-wild videos, however, contain not only the agent’s own state but exogenous state such as background clutter. Since the exogenous state introduces changes unrelated to actions, it hinders reliable latent action learning. This paper investigates this problem analytically by extending a linear LAM framework to explicitly model exogenous state. Our analysis reveals two insights: (1) minimizing the standard reconstruction objective produces latent actions that encode exogenous information from future observation; and (2) learning in a representation space that focuses on endogenous components is a key to mitigating the interference of noise. We further show that previously proposed auxiliary objectives, such as action-supervision, provably encourage latent actions to be consistent across exogenous states. These findings are validated through experiments on both linear and nonlinear LAMs, providing a unified theoretical analysis of how exogenous state hinders latent action learning and why common remedies work. 1 Introduction Latent action models (LAMs) have emerged as a promising approach for learning action-relevant features from videos without action labels. This is particularly advantageous to real-world robotics, where collecting action-labeled datasets at scale is challenging [ 1 ] . Obtaining action annotations typically requires extensive human-expert teleoperation, careful curation, and repeated calibration, which creates a major bottleneck for dataset scaling. LAM mitigates this limitation by learning from unlabeled videos and producing pseudo-action labels that supervise downstream models, such as vision-language-action models (VLA) [ 2 , 3 , 4 , 5 ] and world models [ 6 , 7 ] . For pseudo-action supervision to be effective, latent actions must align with ground-truth actions even in the absence of action labels. This is nontrivial in in-the-wild videos, where observations often contain manipulation-irrelevant variation such as background motion and camera-induced changes. Prior works theoretically show that latent actions can spuriously encode such exogenous noise [ 8 , 9 ] . For instance, latent actions trained on egocentric data tend to encode camera motion rather than physical change, misleading downstream models. Therefore, to improve action alignment, reducing exogenous noise is necessary. Existing approaches reduce it by introducing additional training objectives, such as a small amount of action (or optical-flow) supervision, language descriptions, or multi-view datasets [ 3 , 5 , 10 , 11 ] . These methods augment the standard LAM objective with tractable auxiliary objectives to mitigate exogenous noise. However, why exogenous noise degrades latent actions and how auxiliary objectives mitigate it have not been analyzed in a unified manner. In this work, we provide a unified theoretical analysis of how exogenous variation degrades latent action learning and why auxiliary training objectives mitigate it, building on the linear LAM framework. We first extend linear LAM to Exogenous Block Markov Decision Process (Ex-BMDP) to explicitly model a source of exogenous noise. Under this formulation, we show that training a LAM with a reconstruction-based objective causes exogenous information from the future frame to leak into the latent action, reducing the capacity available for the action. We further analyze how the strength of exogenous noise behaves and explain why previously proposed auxiliary objectives improve action alignment. Our contributions are as follows. 1. We extend the linear LAM setting to Ex-BMDP, providing a framework that explicitly models the source of exogenous noise. Within this framework, we show that latent actions learned by reconstruction leak exogenous information from future frames, and that the energy of the resulting noise is governed by the sensitivity of the observation to exogenous changes. 2. We reformulate previously proposed auxiliary objectives under the Ex-BMDP. We then theoretically show that they mitigate exogenous noise by encouraging consistency across exogenous states. 3. We validate our analysis in controlled simulations with linear LAMs, and further show that the findings hold under general nonlinear LAM. 2 Related Work Latent Action Learning from Videos. Recent works have made substantial progress in learning latent actions from unlabeled videos. One line of research learns discrete latent actions using vector-quantized variational autoencoders (VQ-VAE) [ 12 ] . These discretized latent actions have been used to supervise VLA models [ 2 , 3 , 5 ] and world models [ 6 ] even when the ground-truth actions are not provided. In particular, latent actions have shown strong promise in robot manipulation, including cross-embodiment transfer [ 2 ] and the ability to scale VLA training with human videos in addition to robot datasets [ 3 , 13 ] . While these approaches offer effective ways to incorporate latent actions into downstream embodied AI systems, they offer a limited understanding of when and why latent actions align with ground-truth actions. Exogenous Noise in Latent Action Learning Exogenous noise causes frame transitions, but is independent of the agent’s state and action. For instance, the motion of other agents or background movements are representative examples of it. Prior works show that such noise hinders reliable latent action learning [ 8 , 9 , 10 ] . A theoretical analysis further shows that LAMs can fail when noise variation dominates the action-induced change [ 8 ] . However, their formulation models exogenous noise as an additive term in the observation transition, without explicitly characterizing its source in the environment. As a result, an in-depth analysis of exogenous noise is difficult under this formulation. To mitigate exogenous interference in practice, LAOM [ 10 ] and LAOF [ 11 ] supervise latent action learning with a small amount of action labels and optical flow, respectively. UniVLA [ 3 ] uses language descriptions to encode task-centric latent actions, while MVP-LAM [ 5 ] leverages multi-view datasets to encode consistent transitions across viewpoints, achieving higher mutual information with actions. Despite promising empirical results, the theoretical basis for why such auxiliary objectives alleviate exogenous interference remains an open question. Figure 1: Overview of Linear LAM. (Left) : Observation transitions contain a state-driven component q_{\xi}=\phi_{\xi}(a) and a \xi -driven exogenous component \varepsilon . (Right) : Architecture of linear LAM with a pretrained vision encoder f . Using a vision encoder f that is robust to \xi -variation, latent actions \tilde{z} from other exogenous state \tilde{\xi} , and an exogenous-robust target y improve latent action learning. 3 Setup In this section, we first discuss the Exogenous Block Markov decision process (Ex-BMDP) proposed by Efroni et al. 2022 . Ex-BMDP allows us to describe the general latent action learning in various video datasets. Then, we explain the standard latent action learning scenario in Ex-BMDP and extend the linear LAM setting to Ex-BMDP. 3.1 Exogenous Block Markov Decision Process For a given set \mathcal{U} , we use \Delta(\mathcal{U}) to denote the set of all possible probability distributions over \mathcal{U} . We first consider the Block Markov Decision Process (BMDP) system. BMDP is specified by \mathcal{M}=(\mathcal{O},\mathcal{X},\mathcal{A},T,q,R,H,\mu) (1) where \mathcal{O} is a set of observations, \mathcal{X} is a set of latent states, \mathcal{A} is a finite set of actions. We first start from initial distribution \mu\in\Delta(\mathcal{X}) , and the agent interacts with the environment by repeatedly generating H -step trajectories (x_{1},o_{1},a_{1},r_{1},\dots,x_{H},o_{H},a_{H},r_{H}) where the agent first receives observation o_{t}\sim q(\cdot\mid x_{t}) from emission function q:\mathcal{X}\rightarrow\Delta(\mathcal{O}) and chooses

Read the complete text