Vision-Language Models Unlock Task-Centric Latent Actions

Alexander Nikulin, Ilya Zisman, Albina Klepach, Denis Tarasov, Alexander Derevyagin, Andrei Polubarov, Lyubaykin Nikita, Vladislav Kurenkov · 2026

Read Vision-Language Models Unlock Task-Centric Latent Actions by Alexander Nikulin, Ilya Zisman, Albina Klepach, Denis Tarasov, Alexander Derevyagin, Andrei Polubarov, Lyubaykin Nikita, Vladislav Kurenkov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22714.txt.

Introduction

Alexander Nikulin and four co-authors use vision-language models to teach latent action models what counts as a real action. Latent action models learn action codes from video, and feed the training of robot-ready systems. Their 2026 preprint notes that they fail when the scene holds distractors that move with the action.

People split true motion from noise at a glance; the team give models that task sense. Cleaner action codes should yield robots that learn faster from video. The work links language-guided vision to robot skill. It fixes a subtle, common flaw.

Catalog record

CREATORAlexander Nikulin, Ilya Zisman, Albina Klepach, Denis Tarasov, Alexander Derevyagin, Andrei Polubarov, Lyubaykin Nikita, Vladislav Kurenkov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-Language Models Unlock Task-Centric Latent Actions Alexander Nikulin, Ilya Zisman, Albina Klepach, Denis Tarasov, Alexander Derevyagin, Andrei Polubarov, Lyubaykin Nikita, Vladislav Kurenkov Abstract Latent Action Models (LAMs) have rapidly gained traction as an important component in the pre-training pipelines of leading Vision-Language-Action models. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise in unsupervised way. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that more recent VLMs may perform worse than older ones. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld. † † affiliationnotice: \icmlWorkDoneAtAIRI 1 Introduction Figure 1 : Main result . Success rate on MetaWorld-10 benchmark for LAPO and proposed LAPO+VLM (Molmo), which uses promptable representations. We use three random seeds and report IQM and 95\% -CI based on stratified bootstrapping, following the Agarwal et al. 2021 . See Section 7 for full results. Latent action models (LAMs) ( Schmidt & Jiang 2023 ; Ye et al. 2024 ) have quickly become integral to the pre-training pipelines of leading Vision–Language–Action (VLA) models ( Bjorck et al. 2025 ; Bu et al. 2025a ; Zhong et al. 2025 ; Bruce et al. 2024 ; Jang et al. 2025 ) . By inferring compact, semantically meaningful latent action representations at scale, LAMs mitigate the scarcity of high-quality action-labeled data, removing the data bottleneck in embodied AI and robotics ( McCarthy et al. 2025 ) . Unfortunately, early LAMs that gained recognition ( Schmidt & Jiang 2023 ; Ye et al. 2024 ; Chen et al. 2024b ; Gao et al. 2025 ) were trained on relatively clean datasets, where changes between observations could be explained almost entirely by ground-truth actions, such as in static scenes with a single manipulator. In contrast, real-world data often contains numerous action-correlated distractors, including background human movement or other spurious correlations. However, as shown by Nikulin et al. 2025 ; Zhang et al. 2025 ; Klepach et al. 2025 , without explicit supervision, LAMs struggle to disentangle controllable changes from noise, completely failing to produce meaningful latent actions in the presence of action-correlated distractors. Humans, however, interpret the world through semantics rather than raw pixels, and with only a brief task description can easily separate task-relevant features from irrelevant details. Wouldn’t it also be convenient to simply ask LAM to focus on the relevant features, e.g. robotic arm, and ignore any other details? Figure 2 : Visualization of the task-relevant promptable representations extraction from the VLMs and their subsequent use as targets during latent action learning. Inspired by the work of Chen et al. 2024a on promptable representations, we propose to utilize the Vision-Language Models (VLMs) common-sense reasoning abilities as an unsupervised approach for effectively separating controllable changes from noise, thereby restoring the LAM’s ability to recover ground-truth actions even in the presence of distractors. Using a controlled Distracting MetaWorld ( Yu et al. 2020 ) setup, we show that representations produced by prompting VLMs to focus on task-specific details can serve as an effective target for LAMs in the presence of distractors, increasing the success rate six-fold (see Figure 1 ). As a motivation for our approach, we begin from a simple demonstration experiment, showing that limitations of LAM can be mitigated with the clean target ( Section 5 ). We then conduct large-scale benchmarking of different VLMs, comprising over 29 k+ experiments, to assess their effectiveness at providing promptable representations ( Section 6 ), revealing substantial variation in quality and robustness to hyperparameters. In addition, we show that VLMs significantly outperform self-supervised methods, such as DINOv2 ( Oquab et al. 2023 ) or CLIP ( Radford et al. 2021 ) , confirming our hypothesis that language based filtering is essential for performance. Finally, using the best setup found, we demonstrate that, without any supervision, promptable representations can significantly improve latent action quality and downstream performance ( Section 7 ), outperforming OTTER ( Huang et al. 2025 ) and UniVLA ( Bu et al. 2025b ) baselines. 2 Preliminaries Problem setting. We consider a setting of offline imitation learning from observation ( Liu et al. 2018 ; Torabi et al. 2019 ) . Our goal is to pre-train a policy \pi(o|a) , given a large dataset of expert trajectories \mathcal{D}\coloneq\{(o_{i}^{n})\}_{i=1}^{\tau} , containing observations but not actions (e.g. videos), and a limited number of real action labels. Ideally, the pre-trained agent should achieve maximum performance (e.g. success rate) in the environment after fine-tuning with a minimum amount of action-labeled data. The commonly considered ratio of labeled to unlabeled data is around 2-10\% in the existing work ( Zheng et al. 2023 ; Nikulin et al. 2025 ) , while in our experiments, we consider a ratio as low as <1\% . Promptable representations. We follow the Chen et al. 2024a and define promptable representations simply as a process of obtaining observation embeddings from the VLMs given a task-specific prompt and some extraction and aggregation strategy (see Figure 2 ). We obtain such representations from the last and next-to-last layers ( Chen et al. 2024a ) . In contrast to the Chen et al. 2024a ; Huang et al. 2025 we cannot learn pooling from the data to better predict true actions or obtain better reward. Thus, we experiment only with simple fixed strategies, such as taking the mean over all output embeddings or taking only the embedding of the last token from either prompt or the generated answer. Thus, the final promptable representations of an observation o_{t} is always just a single vector s_{t}\in R^{D} , where D is a VLMs hidden size. Latent action models. Given the dataset of observations \mathcal{D}\coloneq\{(o_{i}^{n})\}_{i=1}^{\tau} , LAMs ( Rybkin et al. 2018 ; Edwards et al. 2019 ; Schmidt & Jiang 2023 ) try to infer latent actions z_{t} such that they are maximally predictive of observed transitions (o_{t},o_{t+1}) while being minimal ( Schmidt & Jiang 2023 ) , i.e. describe changes only relevant to control. After pre-training, LAM is used to supply latent actions for behavioral cloning (BC) on unlabeled dataset to obtain useful behavioral priors. As a final stage, decoder is trained to map from latent to ground-truth actions on a small labeled dataset. Modern LAMs ( Bruce et al. 2024 ; Ye et al. 2024 ; Cui et al. 2024 ; Chen et al. 2024c ; Chen et al. 2024b ; Gao et al. 2025 ) mostly follow the same high-level architecture introduced by LAPO ( Schmidt & Jiang 2023 ) , which uses a combination of inverse (IDM) and forward (FDM) dynamics models to infer latent actions. Given a transition (o_{t},o_{t+1}) , IDM first infers latent action

Read the complete text