Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
Kaiyuan Chen Shuangyu Xie Zehan Ma Ken Goldberg · 2025
Read Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets by Kaiyuan Chen Shuangyu Xie Zehan Ma Ken Goldberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15517.txt.
Introduction
Vision-language models learn the world from internet pictures, but robot logs hold a richer record: poses, gripper states, and forces. Kaiyuan Chen and co-authors flip the pipeline in 2025 with Robo2VLM, a framework that turns real robot trajectories into question-answering tests for those models. It slices each trajectory into manipulation phases and derives ground truth from the non-visual sensors.
Their curated set holds 684,710 questions over 463 scenes and 3,396 tasks, drawn from 176,000 real trajectories. Questions probe spatial layout, goals, and physical interaction. Tests suggest the dataset can both grade and improve models' spatial and interaction reasoning, skills robots themselves will need.
Catalog record
| CREATOR | Kaiyuan Chen Shuangyu Xie Zehan Ma Ken Goldberg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15517
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets Kaiyuan Chen Shuangyu Xie Zehan Ma Ken Goldberg Abstract Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm — using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries – images with textural multiple-choice questions – based on spatial, goal-conditioned, and interaction reasoning question templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning. 1 Introduction Emerging Vision-Language Models (VLMs) radford2021learning ; qwen2.5 ; touvron2023Llama2 ; liu2023visual ; anthropic2024claude35 ; openai2024gpt4o ; google2025gemini25 can perform high-level reasoning and scene interpretation Chen_2024_CVPR ; karamcheti2024prismatic . Recent robotic manipulation systems that integrate VLMs demonstrate enhanced capabilities in semantic and long horizon task reasoning ( kim24openvla ; geminiroboticsteam2025geminiroboticsbringingai ; shi2025hirobotopenendedinstruction ) . Yet, the key challenge persists: the image-text corpora used for VLM pre-training high-quality lack fine-grained spatial information, which are prerequisites for robots to identify long-tail objects, complex scenes, reason about spatial relationships, and plan physical interactions. To address this challenge, some research islam2024eqamx ; yang2025embodiedbench ; li2024eai relies on data generation through simulation shridhar2020alfred ; szot2021habitat ; kolve2017ai2thor . However, such data has inherent limitations due to the sim-to-real gap, because simulator cannot accurately model visual properties such as noise, clutter, and lighting variations and physical properties such as contact dynamics, and interactions. Therefore, strong performance in simulation often fails to translate reliably to the physical world. Meanwhile, deriving spatial knowledge from real-world (“in-the-wild”) data typically requires extensive and costly human labeling sermanet2023robovqa ; ji2025robobrain . In contrast, teleoperated robot trajectories that are used to train visuomotor policies levine2016end , such as Vision-Language-Action(VLA) kim24openvla ; octo_2023 or diffusion policies chi2023diffusionpolicy , typically include precise, structured proprioceptive and kinematic information—joint angles, end-effector poses, gripper states, and force–torque readings—that implicitly encode 3D spatial information. We hypothesize that visual and textual data extracted from robot trajectories can improve VLM’s spatial reasoning capabilities. We present Robo2VLM, a multiple-choice Visual Question Answering (VQA) dataset generation framework for VLMs from real-world robot data. Given a human-teleoperated robot trajectory, Robo2VLM segments the trajectory into distinct manipulation phases, selects representative frames from each phase, and generates questions whose answers are supported by synchronized proprioceptive and kinematic ground truth. We apply Robo2VLM to 176k diverse, real-world trajectories from the Open X-Embodiment (OXE) dataset open_x_embodiment_rt_x_2023 , producing over 3 million VQA samples. Inspired by data optimization paradigms such as domain reweighting in natural language processing xie2023doremi and robot policy learning pmlr-v270-hejna25a , we curate Robo2VLM-1, a large-scale, in-the-wild VQA dataset with 684,710 questions covering 463 distinct scenes, 3,396 robotic manipulation tasks, and 149 manipulation skills. We evaluate 14 model configurations with state-of-the-art open source models (LLaVA, Llama and Qwen) and with different parameter sizes and prompting techniques. The results indicate that some VLMs can achieve near human performance in questions related to object reachability and interaction understanding. Evaluation also suggests a significant gap to human performance, especially in complex reasoning of fine-grained spatial relationship and interactions. Finetuning LLaVA liu2023visual with Robo2VLM-1 improves most of the spatial and interaction capabilities with increasing training dataset size, with a maximum 50% accuracy gain in state reasoning and task understanding. This paper makes the following contributions: (1) Robo2VLM, a VQA data generation framework from real robot trajectories. (2) Robo2VLM-1, an open VQA dataset with 684,710 questions covering diverse and realistic evaluation scenarios for manipulation. (3) Extensive evaluation data on state-of-the-art and fine-tuned VLMs. Figure 1 : Robo2VLM-1 dataset overview . The middle colorbar traces a typical manipulation episode—from pre-grasp through immobilization, contact, detach, and into post-grasp. Surrounding panels give example questions for each VQA category. Dashed arrows connect every category to the phase(s) in which its questions are sampled. Icons beneath each panel list the key sensing modalities (RGB, stereo depth, wrist/side cameras, gripper state, end-effector pose, language instructions) needed to derive ground-truth answers. 2 Related Work Large-Scale Robotics Datasets Recent large-scale robotics datasets, such as Open - X - Embodiment open_x_embodiment_rt_x_2023 and DROID droid , provide extensive teleoperated demonstrations of complex manipulation skills. These datasets are foundational for training modern generalist robot policies—including Octo octo_2023 , RT-1 brohan2023rt1roboticstransformerrealworld , RT-2 brohan2023rt2visionlanguageactionmodelstransfer , OpenVLA kim24openvla , Gemini Robotics geminiroboticsteam2025geminiroboticsbringingai , \pi_{0} black2024pi0 , and Hi Robot shi2025hirobotopenendedinstruction —enabling them to learn diverse skills and understand nuanced physical interactions from broad data. Crucially for grounding VLMs, robotics datasets from Open - X - Embodiment contains rich sensory-modal including RGB video, proprioceptive khazatsky2024droid ; brohan2023rt1roboticstransformerrealworld ; rosete2022tacorl ; mees23hulc2 ; lee2019icra ; BerkeleyUR5Website ; liu2022robot ; Radosavovic2022 ; pari2021surprising ; fanuc_manipulation2023 ; zhou2023modularity ; zhou2023learning ; zhu2022viola ; zhu2022bottom ; haldar2023watch , depth data khazatsky2024droid ; brohan2023rt1roboticstransformerrealworld ; rosete2022tacorl ; mees23hulc2 ; BerkeleyUR5Website , and force-torque Radosavovic2022 ; fanuc_manipulation2023 ; zhou2023modularity ; zhou2023learning , that reflect the dynamics of interaction. These information presents an opportunity to bridge robotics data with VLMs. VQA Benchmarks for Robotics and Embodied AI VQA offers a powerful paradigm for evaluating the visual reasoning capabilities of VLMs antol2015vqa ; goyal2017making ; lee2022what . Recently, VQA benchmarks have been developed for robotic tasks such as visual navigation in long-horizon planning das2018embodiedqa ; anderson2018vision . Simulation-based approaches islam2024eqamx ; yang2025embodiedbench ; li2024eai (often utilizing environments like shridhar2020alfred ; szot2021habitat ; kolve2017ai2thor ) generate large-scale VQA dataset, but face the persistent sim-to-real domain gap, where the result may not hold in reality due to factors like noise, clutter, and lighting variations. Real-world data benchmark, such as RoboVQA sermanet2023robovqa (human-verified Q/A), improve generalization to real world setting but often involve significant manual annotation effort. These methods typically do not fully automate VQA generation by exploiting the rich spectrum of non-visual modalities (e.g., force, torque, proprioception), limiting their ability to support questions grounded in concepts such as grasp stability or multi-view spatial alignment. In contrast, Robo2VLM reduces the need for manual annotation and enables interaction and physical properties reasoning that are underexplored in previous VQA benchmarks, such as gripper states, grasping stability, task goal, and spatial information focus on the robot and target objects. 3 Robo2VLM Robo2VLM generates five-way multiple-choice question answering (MCQ) from real robot teleoperated trajectories. Robo2VLM offers the following key features: (1) High-quality and representative keyframe selection from long-horizon, in-the-wild, multi-modal robot trajectories, ensuring semantic diversity and relevance; (2) Manipulation-centric question generation encompassing spatial, goal-conditioned, and interaction reasoning, each aligned with specific manipulation phases and grounded in corresponding sensor modalities. We begin by defining a