Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment

Theodor Wulff, Federico Tavella, Rahul Singh Maharjan, Manith Adikari, Angelo Cangelosi · 2026

Read Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment by Theodor Wulff, Federico Tavella, Rahul Singh Maharjan, Manith Adikari, Angelo Cangelosi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05614.txt.

Introduction

Wulff and four colleagues ground hierarchical robot vision-language-action models by aligning their words with their deeds. A contrastive model scores how well sub-task descriptions match visual observations and action paths, enabling preference learning that sharpens grounding. On the LanguageTable benchmark the method rivals fully supervised fine-tuning while cutting annotation costs.

Catalog record

CREATORTheodor Wulff, Federico Tavella, Rahul Singh Maharjan, Manith Adikari, Angelo Cangelosi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37240
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05614

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment Theodor Wulff, Federico Tavella, Rahul Singh Maharjan, Manith Adikari, Angelo Cangelosi Abstract Achieving robot transparency is a critical step toward effective human-robot collaboration. To be transparent, a robot’s natural language communication must be consistent with its actions and explicitly grounded in the task and environment. Existing hierarchical Vision-Language-Action (VLA) models can generate language (e.g., through chain-of-thought) and low-level actions. However, current work does not consider explicit alignment between these modalities during training. To address this crucial gap, we propose a novel training framework that explicitly grounds hierarchical VLA sub-task descriptions with respect to the visual observation and action space. Our framework uses a contrastive model to assess the alignment between generated language and corresponding action trajectories. This contrastive model enables direct ranking of different language-trajectory pairs based on their alignment, allowing us to refine the grounding of our hierarchical VLA through offline preference learning. We apply our framework to the LanguageTable dataset, a benchmark dataset of human language-annotated trajectories, and provide critical insights into multimodal grounding representations, all while establishing a strong baseline that achieves performance comparable to fully supervised fine-tuning and minimizing the need for costly data annotations. 1 Introduction Humans use natural language to express thoughts, observations, and actions, enabling us to collaborate across unseen scenarios and environments. During human-robot collaboration, humans expect robot behavior to be explained, like how humans provide explanations to other humans, offering brief explanations on why it failed and its behavior 16 . Thus, embodied AI systems need to be equipped with expressive communication abilities to pave the way for effective human-robot collaboration. The central challenge lies in ensuring that a robotic agent’s communication is not only coherent in language, but also faithfully grounded in its visual perception and behavior. Specifically, this requires solving the problem of symbol grounding 17 . In the context of robotics, grounding describes the ability to relate textual descriptions and visual features to spatial object relations and actions in the real world 19 . Robotic research is one of many disciplines that is being boosted by the rise of Large Language models (LLM) and Vision-Language models (VLM), affecting robotic tasks like plan generation or control policies for robot actions 29 . Current work leverages the effective processing capabilities of VLMs used in Vision-Language-Action models (VLA) for generative tasks that produce diverse outputs such as text, images, or continuous actions. VLAs benefit from the generalizability of the VLM across a diverse distribution of language and image inputs. In most cases, their task is to produce actions either by generating discrete tokens as if they represented word tokens 22 or by utilizing specific decoder heads for continuous action generation 8 ; 6 ; 38 . However, VLAs leverage powerful VLM backbones, providing an untapped capacity for rich, high-quality language output. Prompting LLMs and VLMs to think step-by-step, commonly referred to as chain-of-thought prompting 54 , has led to improved results in commonsense, arithmetic and symbolic reasoning capabilities. Alongside performance gains, this technique also benefits the human prompting the model, who often can follow the reasoning leading to an answer more easily 54 ; 60 . Similarly, hierarchical VLAs 4 ; 45 ; 7 take a high-level task as input and produce an executable subtask as intermediate output, much like chain-of-thought prompting in LLMs. This intermediate sub-task output provides model transparency, by semantic grounding and expression in natural language. However, both the generated sub-steps and the final result carry no certainty of being error-free 52 . Consequently, this introduces an additional layer of generation noise in hierarchical models, since the final output now depends on self-generated, decoupled intermediate outputs 45 . This intermediate sub-task output provides model transparency, by semantic grounding and expression in natural language; at the cost of the additional problem of learning to align the language and action modalities 43 . Learning objectives of current state-of-the-art VLAs are mainly concerned with the success rate of the model in a given task or the ability to generalize well to a large quantity of expert trajectories across embodiments. However, pure success rates fail to reveal how grounded the intermediate outputs are with respect to the actions and visual observation. One can only assume that the agent understood the instruction, but an evaluation metric that discloses possible misunderstanding is missing. To address this, we propose to train an action-conditioned grounding model on human-annotated language-action-vision data through contrastive learning to explicitly evaluate the grounding of language with respect to actions and visual observations. We test this approach in a benchmark task for VLAs, where a robot, provided with a high-level task description, must generate both sub-task descriptions and physical pushing actions to arrange objects. Inspired by recent work that uses learned rewards for preference optimization 58 , we propose a novel framework, called GPLA, for G rounded P reference-based L anguage-action A lignment, to improve the grounding and transparency of hierarchical VLAs. To circumvent the expensive annotation of intermediate outputs, our approach first trains a contrastive model to generate an explicit grounding score based on the alignment between generated natural language sub-goals, visual observations, and subsequent actions. We then use this learned score to generate preference pairs (i.e., more grounded vs. less grounded outputs) and employ preference-based fine-tuning to align the VLA towards producing more semantically correct intermediate steps. We demonstrate our framework on the LanguageTable manipulation benchmark 25 , providing critical insights into the quality and limitations of language grounding representations within hierarchical VLAs. To summarize, our contributions are the following: • We propose GPLA, a novel framework that uses preference learning to directly ground the intermediate language outputs of hierarchical VLAs with visual observations and actions, potentially eliminating the need for expensive annotation collection of intermediate outputs. • GPLA achieves comparable performance with respect to the generated trajectories on the LanguageTable manipulation benchmark 25 to fully supervised fine-tuning, while uniquely offering applicability to low-data regimes. • Visual analysis of the embedding space shows that pre-trained self-supervised models can be used to generate an explicit grounding score. However, they show clear distinctions between the visual observations and language inputs; our action-conditioned grounding model improves this separation by mapping action-vision and text inputs into overlapping embedding space. Figure 1 : Method Overview. We extend a regular VLA (left) into a hierarchical VLA by adding a high-level VLM module to break a high-level instruction down into executable low-level instructions (center), following recent trends on hierarchical VLAs 45 ; 4 . To align the intermediate low-level instruction and the generated trajectory, we invoke a separately trained ranking model, which ranks N sampled output pairs based on their grounding in the environment. Based on these scores, we select the chosen and rejected output pairs that serve as preference data to update our transparent VLA and increase the alignment of the multimodal outputs. 2 Related Work 2.1 Vision-Language-Action Models Foundation models are trained on large-scale data and have achieved outstanding performances across tasks in natural language processing, computer vision, and other domains 1 . The same paradigms have started to achieve generalizable results in the field of robotics, where huge collections of data 37 enabled the training of foundation models for action prediction 9 ; 61 ; 22

Read the complete text