VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, [0.2em] Aditya Nagachandra Di Chang Dongdong Chen Xin Eric Wang Achuta Kadambi, [0.4em] UCLA Microsoft UCSC USC[0.3em] https://vlm4d.github.io/ · 2025
Read VLM4D: Towards Spatiotemporal Awareness in Vision Language Models by Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, [0.2em] Aditya Nagachandra Di Chang Dongdong Chen Xin Eric Wang Achuta Kadambi, [0.4em] UCLA Microsoft UCSC USC[0.3em] https://vlm4d.github.io/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02095.txt.
Introduction
Shijie Zhou and a team of co-authors wrote a 2025 paper introducing VLM4D, a benchmark for testing whether vision language models understand motion in video. Such models read images and text well, but tracking movement, rotation, and viewpoint change over time is a different skill. Humans do it without effort, and current models largely cannot.
The benchmark mixes real and synthetic videos with curated question answer pairs. Questions stress sideways and spinning motion, perspective awareness, and whether movement stays continuous. The team ran leading open and commercial models through it and compared them with human scores. Large gaps appeared, with models failing above all when several visual cues must be combined. The authors release the benchmark to push the field.
Catalog record
| CREATOR | Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, [0.2em] Aditya Nagachandra Di Chang Dongdong Chen Xin Eric Wang Achuta Kadambi, [0.4em] UCLA Microsoft UCSC USC[0.3em] https://vlm4d.github.io/ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, [0.2em] Aditya Nagachandra Di Chang Dongdong Chen Xin Eric Wang Achuta Kadambi, [0.4em] UCLA Microsoft UCSC USC[0.3em] https://vlm4d.github.io/ Abstract Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemporal interactions. Humans effortlessly track and reason about object movements, rotations, and perspective shifts-abilities essential for robust dynamic real-world understanding yet notably lacking in current VLMs. In this paper, we introduce VLM4D , the first benchmark specifically designed to evaluate the spatiotemporal reasoning capabilities of VLMs. Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs emphasizing translational and rotational motions, perspective awareness, and motion continuity. Through comprehensive evaluations of state-of-the-art open and closed-source VLMs, we identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models. Extensive analysis reveals that VLMs struggle particularly with integrating multiple visual cues and maintaining temporal coherence. We further explore promising directions, such as leveraging 4D feature field reconstruction and targeted spatiotemporal supervised fine-tuning, demonstrating their effectiveness in enhancing spatiotemporal comprehension. Our work aims to encourage deeper exploration into improving VLMs’ spatial and temporal grounding, paving the way towards more capable and reliable visual intelligence for dynamic environments. Figure 1 : Spatiotemporal (4D) Awareness . Humans intuitively reason in 4D (3D space + time), effortlessly reconstructing the dynamic spatial trajectory of moving objects from any perspective. In contrast, current Vision Language Models (VLMs) typically rely on aggregating 2D visual features across time, leading to incorrect predictions when motion understanding and interpretation requires deeper spatiotemporal reasoning. In this example, humans correctly perceive the car moving to the right, while the VLM (GPT-4o) inaccurately predicts leftward movement, suggesting VLMs struggle to perform spatiotemporal reasoning. 1 Introduction Humans possess an innate ability to perceive, track, and interpret motion as well as spatial and temporal changes, enabling rich interpretations of complex dynamic events from both egocentric and allocentric perspectives [ 21 , 59 , 10 ] . When observing an object move, we can inherently process any changes such as lateral shifts, rotational directions, and periodic or repeated actions unfolding along a specific trajectory [ 10 ] . These sophisticated perceptual abilities are a product of our spatiotemporal cognition [ 28 ] , and form an essential foundation that allows us to comprehend and reason about physical phenomena, object interactions, and causal relationships within our environment [ 42 , 36 ] . Vision language models (VLMs), which also have the potential to perceive motion and spatiotemporal changes in videos, represent a prominent class of methods aimed at emulating or surpassing human capabilities in integrated visual and linguistic reasoning [ 41 , 23 ] . While previous work on VLMs has primarily focused on static visual understanding—through large-scale training on paired language and image data [ 68 ] —or explored video understanding tasks such as captioning [ 57 ] and scene understanding [ 13 ] , we find that their strong performance in these tasks does not naturally translate into robust spatiotemporal reasoning. This limitation is particularly striking given that state-of-the-art VLMs are typically trained on datasets comprising hundreds of billions of tokens [ 53 ] . In contrast, human infants naturally develop robust spatiotemporal cognition within the first few months of life [ 71 ] . Another key challenge that limits VLM performance on spatiotemporal tasks is the need to implicitly or explicitly reconstruct a four-dimensional (4D) representation—3D space + time—of dynamic scenes, and subsequently reason over this reconstruction [ 80 ] . As illustrated in Fig. 1 , the car is advancing forwards and turning to the left in its own frame of reference. However, from the camera’s perspective, its motion appears as a combination of heading to the right and receding into the distance despite the car being in the center of the frame due to camera rotation. Human observers can seamlessly disentangle these complex dynamics, accurately interpreting trajectories by synthesizing diverse visual cues including camera rotation compensation, stationary scene landmarks, prior knowledge of 3D and 4D environmental structures, and perspective projections [ 59 , 10 , 42 , 28 ] . The inability of current VLMs to integrate spatial, temporal, and semantic cues in a human-like manner stems from their fundamentally different encoding paradigms for visual and language information. This discrepancy highlights a significant gap between human and machine spatiotemporal understanding, suggesting that future VLMs may benefit from insights in cognitive science and neuroscience to develop more advanced mechanisms for perceiving, integrating, reconstructing, and reasoning over dynamic scenes. Figure 2 : Distribution of Dataset Sources and Annotations. Overview of the dataset composition, illustrating the proportions of real third-person (exo-centric) videos (DAVIS [ 66 ] , YouTube-VOS [ 88 ] ), real first-person (ego-centric) videos (Ego4D [ 32 ] ), and synthetic videos (Cosmos [ 4 ] ). The real video data is further categorized by annotation types, including translational, rotational, action, counting, and false positive queries (targeting nonexistent events to assess critical reasoning). Figure 3 : Dataset Generation and Annotation Pipeline. Our dataset was constructed by collecting real videos and generating synthetic data, followed by human-in-the-loop quality reviews to address ambiguous videos and annotations. After temporal alignment and quality assurance, human-annotated questions and ground-truth answers were created, complemented by multiple-choice (MC) answers generated by large language models (LLMs). The final dataset includes real and synthetic video data with comprehensive VLM scoring metrics. In order to effectively characterize and challenge the existing spatiotemporal reasoning abilities of VLMs, we introduce VLM4D , a rigorous benchmark specifically designed to probe the spatiotemporal grounding capabilities of current vision language models. Through this contribution, we aim to catalyze research that addresses the critical gap in spatiotemporal understanding and reasoning within VLMs and provide a foundational analysis highlighting key deficiencies in existing models. We summarize our main contributions as follows: 1. We propose the first benchmark VLM4D explicitly designed to rigorously evaluate the spatiotemporal (4D) reasoning capabilities of Vision Language Models (VLMs). 2. We introduce a novel, meticulously curated dataset consisting of diverse real-world and synthetic video sequences paired with carefully crafted spatiotemporal question-answer (QA) annotations. 3. We analyze the critical limitations of contemporary VLMs in spatiotemporal reasoning and conduct experiments exploring potential solutions, highlighting fundamental challenges and outlining clear directions for impactful future research. 2 Related Work Spatiotemporal Understanding in Vision Language Models Early methods for video understanding before the advent of large vision language models (VLMs) leveraged trajectory-based representations and separate modeling of spatial and motion cues for tasks like action and motion recognition [ 40 , 77 , 70 , 11 ] . Recently, VLMs have evolved rapidly by fully leveraging the significant achievements of Large Language Models (LLMs) [ 9 , 22 , 83 , 7 , 74 , 67 ] and large-scale visual instruction tuning datasets [ 52 , 103 , 20 ] . While VLMs [ 31 , 52 , 103 , 3 , 35 , 44 , 73 , 79 ] exhibit transformative potential for applications such as embodied AI [ 72 , 24 , 38 ] , robotics [ 76 , 64 ] , scene generation [ 62