UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces

Baining Zhao *, Jianjie Fang *, Zichao Dai *, Ziyou Wang, Jirong Zha, Dataset, Code · 2025

Read UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces by Baining Zhao *, Jianjie Fang *, Zichao Dai *, Ziyou Wang, Jirong Zha, Dataset, Code (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06157.txt.

Introduction

Zhao and a large team introduce in twenty twenty-five a test set for judging how well video AI models handle city spaces from a moving, first-person view. Real embodied skill means recalling, seeing, reasoning, and navigating while in motion. The team flew drones by hand in real cities and in simulated settings to gather a large set of short clips.

From the clips they built several thousand multiple-choice questions. They tested many leading video models and found clear limits in city-scale embodied thinking. Skill at causal reasoning tracked closely with recall, perception, and navigation. Two other reasoning types tracked poorly with the rest. Tuning on simulated data also showed promise for transfer to real city tasks.

Catalog record

CREATORBaining Zhao *, Jianjie Fang *, Zichao Dai *, Ziyou Wang, Jirong Zha, Dataset, Code
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces Baining Zhao *, Jianjie Fang *, Zichao Dai *, Ziyou Wang, Jirong Zha, Dataset, Code Abstract Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban 3D space remain to be explored. We introduce a benchmark to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. We have manually control drones to collect 3D embodied motion video data from real-world cities and simulated environments, resulting in 1.5k video clips. Then we design a pipeline to generate 5.2k multiple-choice questions. Evaluations of 17 widely-used Video-LLMs reveal current limitations in urban embodied cognition. Correlation analysis provides insight into the relationships between different tasks, showing that causal reasoning has a strong correlation with recall, perception, and navigation, while the abilities for counterfactual and associative reasoning exhibit lower correlation with other tasks. We also validate the potential for Sim-to-Real transfer in urban embodiment through fine-tuning. 1 Introduction Figure 1: Example of video-language multiple choice question-answering. This figure presents three video examples corresponding to three data sources: real cities in Guangdong Province, China; the simulator EmbodiedCity constructed based on the real city of Beijing, China; and the simulator AerialVLN built on virtual cities. To ensure logical consistency in the movement trajectories within the videos, all video clips consist of continuous perceptual observations generated from ongoing or completed vision-language navigation tasks. For each video clip, we designed different task types to evaluate the embodied cognitive intelligence of Video-LLMs. Table 1: Task set overview. Embodied cognition in motion is divided into four abilities, each of which is manifested in several tasks. Each task is provided with corresponding handcrafted question prototypes. Recall Trajectory Captioning Summarize the agent’s movement path using visual cues / landmarks. Sequence Recall What is the agent’s next step after [changing direction to the left over the intersection]? Object Recall what is [located next to] the [Central ALL-STAR cafe]? Scene Recall Describe scene the agent observes [during the descent to a lower height near the destination]. Start/End Position Where are the starting point and final destination of the agent’s movement? Perception Proximity How does the distance between the agent and the [rooftop with solar panels in the residential area] change after the [agent descends to street level]? Duration Which takes longer, [the agent’s movement through the skyscraper alley] or [its descent to the balcony on the 9th floor]? Landmark Position Given [navigation goal/route] initially, what is agent’s current position relative to [landmark]? Goal Detection Given [navigation goal] at starting location, is the target currently visible in the field of view, and if so, what is its position within the view? Cognitive Map Summarize historical movement observations into a cognitive map. Reasoning Causal Why did the agent [perform a descent after ascending alongside the cylindrical building]? Counterfactual Instead of [flying over the elevated highway intersection], if the agent chooses to [fly around the cylindrical building], can it complete the task, and how is the alternative route? Association Given [navigation goal] at starting location, are there any relevant urban elements or objects in sight when the navigation goal is not visible? Navigation Progress Evaluation (Route-oriented vision-language navigation) Given [navigation route] at starting location, analyze which step the navigation is currently perform. High-level Planning (Goal-oriented vision-language navigation) Given [navigation goal] at starting location, make next plan from the current location. Action Generation Given [navigation goal/route] initially, generate the next control action from the current location. Humans can process continuous first-person visual observations, enabling them to discern direction, judge distance, and navigate in the three-dimensional space of the real world Richardson et al. (2010) ; Burigat et al. (2017) ; Grauman et al. (2022) ; Song et al. (2023) . This refers to embodied cognition in motion, which highlights that cognitive processes are deeply rooted in the body’s interactions with the world Shapiro (2019) ; Newcombe et al. (2023) . Naturally, endowing agents with this embodied cognition capability has been a long-term goal in the field of embodied intelligence Fan et al. (2022) ; Singh et al. (2023) . In recent years, large multimodal models (LMMs) Li (2023) ; Wang et al. (2024d) ; Xu et al. (2025) have emerged as a promising approach to achieve this goal. Typically, video-large language models (Video-LLMs) are evaluated on capabilities such as video summarization Samel et al. (2024) ; Hua et al. (2024) , event question answering Wang et al. (2024a) , and goal localization Yu et al. (2023) . However, the benchmarks used to assess these capabilities are often limited to disembodied third-person video clips, where the agent itself is static Wu et al. (2024b) ; Song et al. (2024) ; Fang et al. (2024) . Besides, existing embodied video understanding research Suglia et al. (2024) ; Cheng et al. (2024) mainly focuses on robotic arm manipulation Nair et al. (2022) or indoor/ground-level movement Marcu et al. (2024) ; Yang et al. (2024) . However, the embodied cognitive abilities required for three-dimensional motion in urban open-ended spaces have not been well-defined or assessed. The first-person perspective visual continuous observations generated in this scenario possess the following characteristics: • Complex Scene and Rich Semantic Information: Urban areas are vast, containing diverse elements like skyscrapers, bridges, and tunnels that provide rich semantic information and pose comprehension and navigation challenges, while dynamic elements like pedestrians and vehicles require real-time adaptation Yao et al. (2024) ; Xu et al. (2023) . • Unique Aerial Motion: Aerial navigation involves vertical mobility and a first-person perspective, adding complexity by requiring enhanced embodied cognition for processing diverse motion and observation angles, necessitating advanced spatial awareness and decision-making Gao et al. (2024) ; Lee et al. (2024) . From these characteristics, we infer that embodied cognition in urban open spaces poses new challenges, and assessing LMMs’ embodied cognitive abilities offers insights for future urban applications Wang et al. (2024c) . We can use drones to capture motion video in urban spaces as they navigate buildings and dynamic elements. Establishing a benchmark presents challenges: 1) Creating a task set to evaluate embodied capabilities in urban spaces. 2) Obtaining video data: Unlike most high-altitude aerial views Li et al. (2016) ; Zhu et al. (2021) ; Wen et al. (2021) , our goal is to record drones maneuvering among urban structures with flexible movement and camera angles. Drones face issues like signal loss, limited range, and crashes due to obstructions and interference, making data collection difficult and costly. 3) Designing logical and purposive motion routes to ensure coherent visual observations. Accordingly, we introduce a benchmark, UrbanVideo-Bench , designed for embodied motion cognition from embodied videos in urban airspace. Firstly, we propose a novel task set comprising 16 tasks characterized by urban spatiotemporal features, as shown in Figure 1 and Table 1 . Secondly, we manually operate drones to collect embodied video data from 1) the real cities in Guangdong Province, China, 2) a simulator EmbodiedCity Gao et al. (2024) built on the real city Beijing, China, and 3) a simulator AerialVLN Liu et al. (2023) built on virtual cities. Using both real devices and simulators helps to rapidly increase the

Read the complete text