From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

Yihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, Jing Zhang · 2026

Read From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models by Yihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, Jing Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04678.txt.

Introduction

Yihan Lin and six co-authors systematically compare ways of supervising vision-language-action models, the systems that connect seeing and language to robot control, using latent actions as an intermediate representation. Latent actions promise consistent training across very different data sets, but supervision strategies have been fragmented and never compared head to head.

Under one unified baseline, they test four integration strategies split between image-based latent actions, which regularise the trajectory, and action-based ones, which unify the target space. Image-based supervision helped long-horizon reasoning and scene generalisation, while action-based supervision excelled at complex motor coordination. Directly supervising the model with discrete latent action tokens worked best overall.

Catalog record

CREATORYihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, Jing Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models Yihan Lin, Haoyang Li, Yang Li, Haitao Shen, Yihan Zhao, Chao Shao, Jing Zhang Abstract Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at GitHub . 1 Introduction Vision-Language-Action (VLA) models have emerged as a paradigm for learning generalist robotic policies ( 19 ; 3 ; 4 ; 38 ; 26 ) . By training on large-scale datasets, these methods achieve robust performance across diverse tasks and environments ( 16 ; 2 ) . However, the VLA training data are heterogeneous, spanning inconsistent action spaces from various robotic platforms and human videos ( 17 ; 12 ; 13 ) . This heterogeneity may lead to performance degradation due to mismatched action semantics across domains, motivating the use of suitable intermediate representations. Latent actions abstract motion from either visual transitions or raw actions, offering a unified supervision signal for training VLA models on heterogeneous data ( 27 ; 20 ) . Recent VLA models have started to leverage latent actions to improve generalization ( 1 ; 2 ) , but their integration strategies into VLA models remain fragmented. Existing approaches range from discrete token supervision ( 33 ; 6 ) to auxiliary learning objectives ( 10 ) , all targeting the VLM backbone; this diversity suggests different assumptions about the role of latent actions, leaving the optimal formulation unclear. In this work, rather than evaluating latent action models themselves, we study how different latent action supervision choices affect VLA policy learning under a unified baseline. Figure 1 : Overview of latent actions in VLA. Left: a unified VLA pipeline; right: our two perspectives and four integration strategies. We systematize integration strategies from two complementary perspectives, defined by the mapping direction between vision-language (VL) space and action spaces: (1) Regularizing the Trajectory. This forward mapping uses image-based latent actions as high-level visual plans to guide the VLM’s decision-making. (2) Unifying the Target Space. This reverse mapping unifies actions into action-based latent actions aligned with the VLM token space, bridging semantic mismatches across heterogeneous action spaces. Motivated by our two perspectives, we propose four distinct supervision strategies (as shown in Fig. 1 ). For trajectory regularization, we investigate Implicit Representation Alignment , Explicit Direct Decoding , and Explicit Conditional Decoding ; for target unification, we consider Action-to-Token Mapping . To evaluate these strategies under fair conditions, we implement them within a unified VLA baseline, specifically designed with a shared backbone and action head. We evaluate these strategies across benchmarks (LIBERO and RobotWin 2.0) and real-world experiments. We identify three empirical findings: (i) image-based latent actions are more effective for long-horizon reasoning and scene-level generalization, while action-based latent actions excel at motorically complex tasks; (ii) directly supervising VLMs to predict discrete latent action tokens yields the most effective performance; and (iii) latent action supervision consistently improves performance under joint training on mixed-task datasets. These results provide guidance for selecting latent action supervision strategies in VLA policy learning. Overall, the core contributions include: • We systematically study latent action supervision under a unified VLA baseline, isolating the effects of latent action formulation and architectural integration. • We categorize latent action supervision into two complementary perspectives and propose four integration strategies for VLA training. • We provide extensive empirical analysis that reveals (i) formulation-task correspondence, (ii) the comparative effectiveness of different strategies, and (iii) robustness under joint training on mixed-task datasets. 2 Related Work 2.1 Latent Action Representations Latent actions are compact representations used to structure action modeling, capturing either visual dynamics or control abstractions. Prior work broadly falls into two directions: image-based and action-based. On one hand, image-based latent actions learn a latent action model from visual transitions to capture structured changes between images, as explored in Genie ( 5 ) and LAPO ( 7 ) . Subsequent work further studies how more effectively capture task-relevant changes in visual transitions ( 25 ; 35 ) . On the other hand, action-based latent actions are learned from continuous action trajectories to induce compact and semantically structured action spaces. Some approaches focus on extracting recurring behavioral patterns from action sequences ( 24 ; 20 ; 32 ) . Others discretize actions and align the resulting representations to external semantic spaces, such as MLLM token spaces ( 29 ) or downstream VLA task spaces ( 36 ) . 2.2 VLA Models Integrating with Latent Actions Recent VLA models have increasingly incorporated latent actions to structure policy learning, exploring a variety of integration strategies. Methods using image-based latent actions guide low-level control through visual abstractions ( 33 ) . Some train the backbone to predict latent actions that are decoded into actions by a separate decoder (e.g., UniVLA ( 6 ) , IGOR ( 9 ) ), while others jointly generate latent actions and actions within the backbone (e.g., Moto-GPT ( 11 ) , villa-x ( 10 ) ). Conversely, methods using action-based latent actions compress continuous actions into a unified latent space to bridge heterogeneous action spaces ( 36 ) . However, prior methods entangle latent action formulation and integration with VLA architectural design. This work isolates their effects to study how latent action supervision influences policy learning. 3 Preliminaries 3.1 Vision-Language-Action Models We consider VLA models formulated as conditional sequence generation. Given visual observations o_{t}\in\mathcal{O} and a language instruction \ell\in\mathcal{L} at timestep t , the objective is to learn a policy \pi that predicts an action chunk \mathrm{a}_{t}=\langle a_{t},a_{t+1},\dots,a_{t+H-1}\rangle\in\mathcal{A}^{H\times m} , where H is the chunk size, m is action dimension. The model is trained on a set of demonstration trajectories \mathcal{D}=\{(o^{i}_{t},\ell,\mathrm{a}^{i}_{t})\}_{i=1,t=1}^{N,T_{i}} , where the model minimizes the reconstruction error between the predicted chunk and the ground truth action sequence. 3.2 Latent Action Models We categorize latent action models into two types based on their input modalities and modeling objectives. All latent action models are kept frozen during VLA training. 3.2.1 Image-based latent action models. Image-based latent action models learn compact action-relevant dynamics from visual transitions. Given a temporal horizon \delta , a start observation o_{t} , and an end observation o_{t+\delta} , the model encodes the visual transition into a discrete latent action z_{t}^{\mathrm{img}} . \left\{\begin{aligned} c_{t}^{\mathrm{img}}&=E_{\theta}\bigl([o_{t};\,o_{t+\delta}]\bigr),&c_{t}^{\mathrm{img}}\in\mathbb{R}^{d},\\ z_{t}^{\mathrm{img}}&=\mathrm{VQ}(c_{t}^{\mathrm{img}}),&z_{t}^{\mathrm{img}}\in\{1,\ldots,K_{img}\}^{P},\\ \hat{o}_{t+\delta}&=D_{\theta}\bigl([o_{t};\,e(z_{t}^{\mathrm{img}})]\bigr).\end{aligned}\right. (1) Here, E_{\theta} and D_{\theta} denote the encoder and decoder, \mathrm{VQ}(\cdot) is a vector-quantization operator with a learned codebook e(\cdot) , and c_{t}^{\mathrm{img}} denotes the continuous embedding, where d is the embedding dimension and each timestep corresponds to

Read the complete text