Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions

Wei Zhao, Gongsheng Li, Zhefei Gong, Pengxiang Ding, Han Zhao, Donglin Wang · 2025

Read Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions by Wei Zhao, Gongsheng Li, Zhefei Gong, Pengxiang Ding, Han Zhao, Donglin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11214.txt.

Introduction

Wei Zhao and co-authors widen the ways people can give orders to a robot, in a 2025 paper. Modern robot models that see, read, and act take spoken or typed commands well. But a person might instead show a photo of the wanted object, point to words on a whiteboard, or act out the task in a video.

Their model, OE-VLA, accepts all such open-ended prompts through one network that turns sight and language into action. In tests it matched standard models that take only word commands. It also did well on four further prompt types, from image goals to shown behavior. The result points to home and work robots that take cues the way people naturally give them, not just in sentences.

Catalog record

CREATORWei Zhao, Gongsheng Li, Zhefei Gong, Pengxiang Ding, Han Zhao, Donglin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11214

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions Wei Zhao, Gongsheng Li, Zhefei Gong, Pengxiang Ding, Han Zhao, Donglin Wang Abstract Vision-Language-Action (VLA) models have recently become highly prominent in the field of robotics. Leveraging vision-language foundation models trained on large-scale internet data, the VLA model can generate robotic actions directly from visual observations and human instructions through a single end-to-end neural network. Despite their effectiveness, current VLA models usually accept only one form of human prompting, language instructions, which may constrain their applicability in open-ended human-robot interactions. For example, a user might expect the robot to retrieve an object shown in an image, follow an instruction written on the whiteboard, or imitate a behavior demonstrated in a video, rather than relying solely on language-based descriptions. To address this gap, we introduce OE-VLA, which explores the potential of VLA models for open-ended multimodal instructions. Extensive results demonstrate that our OE-VLA not only achieves comparable performance to traditional VLA models with linguistic input but also delivers impressive results across four additional categories of open-ended tasks. The proposed methodology could significantly expand the applications of VLA models across various everyday scenarios and facilitate human-robot interaction. 1 Introduction Large language models (LLMs), such as ChatGPT ( Ouyang et al., 2022 ) , LLaMA ( Touvron et al., 2023 ) , and Gemini ( Team et al., 2024 ) , have achieved significant success in various domains and greatly eased people’s daily lives. In particular, researchers have also conducted studies on multimodal large language models (MLLMs), such as vision language models (VLMs) ( Liu et al., 2023 ; Li et al., 2023a ; Awadalla et al., 2023 ; Karamcheti et al., 2024 ; Zhang et al., 2024 ) , to enable LLMs to understand human intentions using both linguistic and visual modalities. The aforementioned versatile models provide a robust foundation for the advancement of artificial general intelligence (AGI) within cyberspace. As these techniques undergo continuous improvement, a critical question emerges: How can we develop similar models capable of interacting with the physical world? One possible approach is the vision-language-action (VLA) model proposed in RT-2 ( Zitkovich et al., 2023 ) for robot manipulation. This approach leverages a pre-trained VLM as the base model and fine-tunes it with carefully curated robotic data, aiming to fully harness the knowledge acquired by the VLM. Specifically, numerous studies have been conducted to investigate the potential of the VLA model. RT-2 ( Zitkovich et al., 2023 ) , RT-X ( Collaboration et al., 2023 ) , and OpenVLA ( Kim et al., 2024 ) demonstrate that, compared to traditional policies ( Brohan et al., 2023 ; Lynch* and Sermanet*, 2021 ) , these end-to-end models can achieve superior generalization across diverse language commands and environments. Building on this starting point, numerous studies have been conducted to refine the VLA model into a robust and versatile general-purpose solution. One group of studies focuses on searching for better approaches to simulate the action space. For example, instead of discretizing robot actions into language tokens, these studies treat raw continuous actions as the predictive target. Therefore, these works ( Li et al., 2023b ; Li et al., 2024a ; Liu et al., 2025 ; Li et al., 2024b ; Wen et al., 2025a ; Wen et al., 2025b ) employ a separate policy head, using an RNN, Transformer ( Vaswani et al., 2017 ) , diffusion network ( Chi et al., 2023 ) , or the flow matching ( Black et al., 2024 ) technique, to enhance the model performance. Another group of studies ( Zhen et al., 2024 ; Barcellona et al., 2025 ) focuses on promoting the VLA by incorporating augmented robot observations, such as improving representations, utilizing historical information, or grounding robot actions in a world model. However, most previous studies place an emphasis on directly improving the performance (e.g., success rate) of the VLA, while overlooking the enhancement of the interactive process between humans and robots. Specifically, previous VLA models typically use pure linguistic instructions to guide the robot in performing a task, whereas in daily life, human instructions are not confined to a single form. For example, human guidance could take the form of optical instructions on a whiteboard, a combination of text and images, a one-shot video demonstration, or simply a goal image. To address this issue, we present OE-VLA, a novel approach capable of handling the aforementioned open-ended multimodal human instructions. Accompanied by our method, we also introduce two new benchmarks, OE-CALVIN base and OE-CALVIN hard , to ensure reproducibility and comparability. These two benchmarks are built upon the popular CALVIN test suite and feature varying levels of difficulty for open-ended instructions. Experimental results show that the proposed OE-VLA not only matches the performance of traditional VLA models in following language instructions but also demonstrates acceptable performance in tackling diverse free-form human prompts. In conclusion, our contributions are three-fold: • We introduce OE-VLA, a novel VLA model capable of processing diverse open-ended multimodal human instructions through a unified neural architecture. • We propose a general approach for constructing robotic datasets with free-form multimodal instructions by leveraging existing datasets. Meanwhile, a two-stage curriculum learning algorithm is proposed to fine-tune a vision language foundation model, capable of handling interleaved modalities, into a VLA model for open-ended instructions. • We introduce two robust benchmarks with diverse open-ended instructions, based on the CALVIN suite, to evaluate our method. Figure 1: The introduced OE-VLA with four open-ended robot manipulation tasks. 2 Related Work Previous studies, such as RT-2 ( Zitkovich et al., 2023 ) , RT-X ( Collaboration et al., 2023 ) , and OpenVLA ( Kim et al., 2024 ) , use a pretrained VLM as the foundation and fine-tune it into a VLA model with carefully curated robot demonstration datasets. Leveraging the vast knowledge contained within the VLM, these models outperform traditional robot policies, yielding remarkable results. RoboFlamingo ( Li et al., 2023b ) has made further efforts to incorporate historical visual information as input and predict continuous robot actions, rather than relying on discretized action tokens. Following the remarkable success of diffusion policy ( Chi et al., 2023 ) , numerous VLA studies, such as CogACT ( Li et al., 2024a ) , \pi_{0} ( Black et al., 2024 ) , and GR00T ( NVIDIA et al., 2025 ) , have integrated it as a separate output head and achieved SOTA performance. Numerous works have also attempted to improve VLA performance from other perspectives. 3D-VLA ( Zhen et al., 2024 ) leverages depth information, RT-H ( Belkhale et al., 2024 ) introduces action hierarchy techniques, and Embodied-COT ( Zawalski et al., 2024 ) integrates classic chain-of-thought reasoning into the VLA model. Moreover, there is also a line of work ( Wu et al., 2023 ; Barcellona et al., 2025 ) that leverages world models to further enhance performance. However, previous studies have predominantly concentrated on improving the success rate of robot manipulation using language modality instructions. Few studies have explored equipping VLA models with the capability to handle more open-ended task specifications. Perhaps the study that has the most similar motivation to ours is VIMA ( Jiang et al., 2023 ) , which is capable

Read the complete text