Latent Action Learning Requires Supervision in the Presence of Distractors
Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov · 2025
Read Latent Action Learning Requires Supervision in the Presence of Distractors by Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00379.txt.
Introduction
Alexander Nikulin and co-authors test a way to teach robots from plain video, in a 2025 paper. Latent action learning lets a model guess the hidden moves behind each video frame. It can then train on the huge stock of video on the web, with no labels at all. Past work used clean clips where frame changes come from the agent's own acts.
Real video is messier, with side events that move with the action and act as distractors. The team used a test suite full of such traps and found the base method, LAPO, falters there. Their fix, LAOM, made the learned actions eight times cleaner by one probe measure. Adding true action labels for even a tiny slice of the data lifted later scores more than fourfold. They urge builders to mix in some supervision from the start.
Catalog record
| CREATOR | Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00379
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Action Learning Requires Supervision in the Presence of Distractors Alexander Nikulin, Ilya Zisman, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov Abstract Recently, latent action learning, pioneered by Latent Action Policies (LAPO), have shown remarkable pre-training efficiency on observation-only data, offering potential for leveraging vast amounts of video available on the web for embodied AI. However, prior work has focused on distractor-free data, where changes between observations are primarily explained by ground-truth actions. Unfortunately, real-world videos contain action-correlated distractors that may hinder latent action learning. Using Distracting Control Suite (DCS) we empirically investigate the effect of distractors on latent action learning and demonstrate that LAPO struggle in such scenario. We propose LAOM, a simple LAPO modification that improves the quality of latent actions by 8x , as measured by linear probing. Importantly, we show that providing supervision with ground-truth actions, as few as 2.5% of the full dataset, during latent action learning improves downstream performance by 4.2x on average. Our findings suggest that integrating supervision during Latent Action Models (LAM) training is critical in the presence of distractors, challenging the conventional pipeline of first learning LAM and only then decoding from latent to ground-truth actions. 1 Introduction Figure 1 : We show that in the presence of distractors, LAPO struggles to learn latent actions useful for pre-training and that simple BC or IDM are more effective. We propose LAOM, a simple modification that doubles the performance but still underperforms. Thus, we propose to reuse available ground-truth action labels to supervise latent action learning, which significantly improves the performance, achieving normalized score of 0.44. It recovers almost half the performance of BC with access to the full action-labeled dataset, while having access to only 2.5%. Results are averaged over four environments from Distracting Control Suite, three random seeds each. We provide per-environment plots on Figure 8 . See Section 3 for the evaluation protocol, Sections 4 and 5 for method details. Figure 2 : Visualization of the environments from the Distracting Control Suite (DCS) used in our work. Top row: without any distractors, identical to the original DeepMind Control Suite. Bottom row: with distractors, which consists of dynamic background videos, agent color change and camera shaking. See Section 3 for additional details. Recently, a new wave of approaches based on latent action learning has emerged ( Edwards et al., 2019 ) , demonstrating superior pre-training efficiency on datasets without action labels in large-scale robotics ( Ye et al., 2024 ; Chen et al., 2024b ; Chen et al., 2024a ; Cui et al., 2024 ; Bruce et al., 2024 ) and reinforcement learning ( Schmidt & Jiang, 2023 ) . Latent Action Models (LAM) infer latent actions between successive observations, effectively compressing observed changes. Under certain conditions, latent actions can even rediscover the ground truth action space ( Schmidt & Jiang, 2023 ; Bruce et al., 2024 ) . After training, LAM can be utilized for imitation learning on latent actions to obtain useful behavioral priors. For example, LAPA ( Ye et al., 2024 ) showed that latent action learning can be used to pre-train large model on only human manipulation videos, and despite the huge cross-embodiment gap, still outperform OpenVLA ( Kim et al., 2024b ) , which was pre-trained on expert in-domain data with available action labels. Despite the initial success and the promise of unlocking vast amounts of video available on the web ( Schmidt & Jiang, 2023 ; Ye et al., 2024 ) , there is a critical shortcoming of previous work – it uses distractor-free data, where all changes between observations are mainly and most efficiently explained by ground truth actions only, such as robot manipulation on a static background ( Khazatsky et al., 2024 ) . Unfortunately, this is not true for real-world web-scale data, as it contains a lot of action-correlated noise ( Misra et al., 2024 ) , e.g. people moving in the background. Such noise may better explain video dynamics and thus lead to latent actions unrelated to real actions. The phenomenon of overfitting to task-irrelevant information is not new and has been studied in model-based ( Wang et al., 2024 ) and representation learning ( Lamb et al., 2022 ; Zhang et al., 2020 ; Zhou et al., 2023 ) . However, the effect of distractors on latent action learning, which we aim to address in this work, has not been similarly investigated. In this work we empirically investigate the effect of action-correlated distractors on latent action learning using Distracting Control Suite ( Stone et al., 2021 ) . We demonstrate that naive latent action learning based on quantization and reconstruction objectives, such as LAPO ( Schmidt & Jiang, 2023 ) , struggle in the precense of distractors (see Section 4 ). We propose LAOM, a simple LAPO modification that improve the quality of latent actions by 8x , as measured by linear probing, and double the downstream performance (see Figure 8 ). However, even after this, the resulting performance is only slightly better than simple Behavioral Cloning on available ground-truth actions. Thus, as our core contribution, we show that providing supervision with a small number, as little as 2.5 % of the complete dataset, of action labels during LAOM training improves the downstream performance by 4.3x on average (see Section 5 ), outperforming all baselines (see Figure 8 ). Our findings suggest that the pipeline used in most current work ( Ye et al., 2024 ; Cui et al., 2024 ; Chen et al., 2024b ) to first learn LAM and only then decode to ground-truth actions is suboptimal when distractors are present, as with supervision better result can be achieved using the same budget of actions labels. In addition, we show that latent action learning with supervision generalizes better in contrast to approaches based on inverse dynamics models ( Baker et al., 2022 ; Zhang et al., 2022a ; Zheng et al., 2023 ) but does not learn control-endogenous minimal state ( Lamb et al., 2022 ) . Figure 3 : Overview of the latent action learning pipeline. In the first stage, the Latent Action Model (LAM) is pre-trained to infer latent actions between consecutive observations. In the second stage, the LAM is used to relabel the entire dataset with latent actions, which are then used for behavioral cloning. Finally, a decoder is trained to map from latent to true actions using a small number of labelled trajectories. In our work, we do not modify this pipeline in any way; we only examine the LAM architecture itself (see Figure 4 ). 2 Preliminaries Learning from observations. Most methods in reinforcement learning require access to the dataset \mathcal{D}\coloneq\{\tau_{n}\}_{n=1}^{N} of N trajectories, where each \tau_{n}\coloneq\{(o_{i}^{n},a_{i}^{n},r_{i}^{n})\}_{i=1}^{\tau} contains observations, actions and rewards. Similarly, imitation learning requires access to trajectories \tau_{n}\coloneq\{(o_{i}^{n},a_{i}^{n})\}_{i=1}^{\tau} that contain actions. Unfortunately, most expert demonstrations in the real world, such as YouTube videos of some human activity ( Aytar et al., 2018 ; Baker et al., 2022 ; Zhang et al., 2022a ; Ghosh et al., 2023 ) , do not include rewards or action labels. Thus, researchers are actively exploring how to most effectively use the data \tau_{n}\coloneq\{(o_{i}^{n})\}_{i=1}^{\tau} without action labels to accelerate the learning of embodied