From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
Irving Fang Juexiao Zhang ∗ Shengbang Tong Chen Feng New York University · 2025
Read From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models by Irving Fang Juexiao Zhang ∗ Shengbang Tong Chen Feng New York University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09930.txt.
Introduction
Irving Fang and co-authors probe, in a 2025 paper, where robot policies that join vision, language, and action stop generalizing. Such models promise versatile robots by borrowing the broad knowledge of vision-language models, but current tests are narrow and rarely separate what the borrowed brain contributes.
The team built a unified suite of fifty simulation tasks in ten families spanning instructions, vision, and objects. Leading architectures were run through it systematically. The pattern was clear: the vision-language backbone gives strong perception and high-level planning, good intentions, yet those intentions often fail to become precise physical execution. The suite exposes the intention-action gap.
Catalog record
| CREATOR | Irving Fang Juexiao Zhang ∗ Shengbang Tong Chen Feng New York University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models Irving Fang Juexiao Zhang ∗ Shengbang Tong Chen Feng New York University Abstract One promise that Vision-Language-Action (VLA) models hold over traditional imitation learning for robotics is to leverage the broad generalization capabilities of large Vision-Language Models (VLMs) to produce versatile, “generalist” robot policies. However, current evaluations of VLAs remain insufficient. Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions. Emerging benchmarks for VLAs that incorporate language often come with limited evaluation tasks and do not intend to investigate how much VLM pretraining truly contributes to the generalization capabilities of the downstream robotic policy. Meanwhile, much research relies on real-world robot setups designed in isolation by different institutions, which creates a barrier for reproducibility and accessibility. To address this gap, we introduce a unified probing suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects. We systematically evaluate several state-of-the-art VLA architectures on this suite to understand their generalization capability. Our results show that while VLM backbones endow VLAs with robust perceptual understanding and high-level planning, which we refer to as good intentions, this does not reliably translate into precise motor execution: when faced with out-of-distribution observations, policies often exhibit coherent intentions, but falter in action execution. Moreover, fine-tuning on action data can erode the original VLM’s generalist reasoning abilities. We release our task suite and evaluation code to serve as a standardized benchmark for future VLAs and to drive research on closing the perception-to-action gap. More information, including the source code, can be found at https://ai4ce.github.io/INT-ACT/ 1 Introduction Building on the significant progress of large Vision-Language Models (VLMs) [ 7 , 23 , 13 , 30 ] in computer vision and multi-modal tasks, Vision-Language-Action (VLA) [ 19 , 2 , 16 ] models are emerging to transfer these generalist skills to embodied agents and robotics. The ultimate goal is to create robots that can interpret natural language instructions, perceive and plan under complex and diverse environments, and execute versatile behaviors, all under few- or zero-shot scenarios. This goal calls for a strong generalization capability beyond the training and fine-tuning of robotics data. Early results are encouraging: VLAs can natively follow high-level language instructions in cluttered scenes and perform highly dexterous manipulation on challenging objects [ 19 , 2 , 16 ] . Moreover, since VLAs are typically pre-trained on a large-scale cross-embodiment dataset [ 9 ] , they can be fine-tuned on new tasks with novel configurations for quick adaptation, partially alleviating the data scarcity problem plaguing robotics research. Despite this progress, there is no consensus on how to systematically measure the generalization capability of VLAs, which is a supposedly strong point of leveraging the modern VLMs over previous perceptual systems. Notably, the current evaluation procedure for VLA generally relies on: 1. Recent simulation benchmarks such as CALVIN [ 27 ] , LIBERO [ 22 ] and SimplerEnv [ 20 ] are equipped with natural language instructions, enabling the evaluation of vision-language-action (VLA) models’ instruction-following capabilities, which differentiate them from the more traditional benchmarks such as RLBench [ 17 ] or RoboMimic [ 26 ] that focus primarily on low-level control and imitation, with limited or no support for diverse language-conditioned tasks. However, the variety of tasks is relatively limited, and VLA’s generalization capability is not targeted. 2. Real-world robot setup as seen in [ 19 , 2 , 16 ] . While this type of evaluation offers endless flexibility and opportunities to test the generalization of a VLA policy, it has considerably bigger barriers than any vision language benchmark due to the time and monetary commitment required. As a result, it remains unclear to many how well the state-of-the-art VLA models can generalize beyond the robotics datasets they train on by leveraging the remarkable generalization capability of their built-in VLM. To fill this void, we propose a unified probing suite comprising 50 simulation-based tasks organized into 10 categories that vary systematically along three categories: • Object diversity: out-of-distribution (OOD) object, appearance, and affordances unseen during embodiment-finetuning. • Language complexity: from templated commands (e.g., “ Put A on B ”) to compositional, knowledge and reasoning-intensive instructions. • Vision-language thinking : parsing various distractors that are challenging at the perception or planning level. In this work, we make two primary contributions: • We introduce and open-source INT-ACT, a comprehensive VLA generalization probing suite comprising 50 tasks across 3 major categories and 10 subcategories, substantially extending the scope of existing VLA benchmarks. • Through extensive benchmarking, we uncover two key failure modes in current state-of-the-art VLA models: – A persistent and pronounced Intention-Action Gap, where strong semantic understanding under distribution shift fails to translate into reliable execution. – Fragile multimodal generalization, particularly under language variations and compounding visual-language distribution shifts. 2 Related Work 2.1 Vision-Language-Action Models as Robotic Foundation Models The paradigm of foundation models —large, pre-trained architectures fine-tunable for diverse downstream tasks, has transformed natural language processing (NLP) and computer vision (CV). Notable examples include BERT [ 10 ] and GPT [ 5 ] for NLP, and CLIP [ 36 ] and DINOv2 [ 32 ] for vision, all showcasing strong zero- and few-shot generalization. This success has shifted research from specialized models towards unified architectures leveraging web-scale pre-training data. In robotics, Robotic Foundation Models aim to equip embodied agents with similar generalist abilities by training on diverse robotics datasets like Open-X Embodiment [ 9 ] . Early works, such as RT-1 [ 3 ] , Octo [ 29 ] , and RDT [ 25 ] , typically use separate pretrained encoders (e.g., T5 [ 37 ] for language, SigLIP [ 44 ] for vision) to encode multimodal inputs, and subsequently train imitation learning policies (e.g., diffusion policies [ 8 ] ) to map encoded features to robot actions. Following the success of VLMs [ 7 , 6 , 30 , 18 , 39 , 1 ] , Vision-Language-Action (VLA) models aim to integrate the powerful pretrained vision-language models (VLMs) directly, instead of leveraging only pretrained encoders. For instance, RT-2 [ 4 ] fine-tunes PaLM-E [ 11 ] and PaLI-X [ 6 ] on combined vision-language and robotics data via autoregressive token prediction, treating discretized robot actions as tokens. OpenVLA [ 19 ] finetunes Prismatic VLM [ 18 ] on open robotics datasets exclusively, while SpatialVLA [ 35 ] and FAST [ 34 ] further refine action tokenization strategies. Conversely, models like \pi_{0} [ 2 ] circumvent tokenization by coupling specialized transformers for actions with VLMs (e.g., PaliGemma [ 1 ] ) and applying flow-matching training [ 24 , 21 , 45 ] . These models can achieve zero-shot inference through retrieval if the tasks closely resemble training scenarios (e.g., robot, action representation, camera pose, etc). However, fine-tuning on domain-specific demonstration data is necessary for practical deployment. Our study examines VLAs, with the aim of determining whether the pretrained vision-language alignment inherent in VLM contributes to enhanced out-of-distribution generalization after domain-specific fine-tuning. 2.2 Benchmarks for Vision-Language-Action Models Benchmarking in robotics and embodied AI has traditionally relied on standardized simulation environments to systematically evaluate model performance. Early policy learning benchmarks such as RLBench [ 17 ] , RoboMimic [ 26 ] ,