VISTA: Video Interaction Spatio-Temporal Analysis Benchmark

Alejandro Aparcedo, Akash Kumar, Aaryan Garg, Dalton Pham, Wen-Kai Chen, Anirudh Bharadwaj, Aman Chadha, Yogesh Rawat · 2026

Read VISTA: Video Interaction Spatio-Temporal Analysis Benchmark by Alejandro Aparcedo, Akash Kumar, Aaryan Garg, Dalton Pham, Wen-Kai Chen, Anirudh Bharadwaj, Aman Chadha, Yogesh Rawat (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01391.txt.

Introduction

Alejandro Aparcedo and seven co-authors introduce VISTA, a benchmark for testing how well vision-language models follow who does what in video. Older tests lean on single-action clips, closed lists of traits, and narrow types of things, so they miss messy real scenes with many actors.

VISTA breaks each clip into entities, their actions, and the relations between them, which allows checks along spatial, temporal, and relational lines. It merges several data sets into one interaction-based scheme and holds about 12,000 curated video-question pairs of varied scene types and difficulty. The team runs 11 leading models through it and maps where each one fails. The fine breakdown shows biases and weak spots that a single score would hide, giving builders a clearer guide for design and training.

Catalog record

CREATORAlejandro Aparcedo, Akash Kumar, Aaryan Garg, Dalton Pham, Wen-Kai Chen, Anirudh Bharadwaj, Aman Chadha, Yogesh Rawat
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01391

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VISTA: Video Interaction Spatio-Temporal Analysis Benchmark Alejandro Aparcedo, Akash Kumar, Aaryan Garg, Dalton Pham, Wen-Kai Chen, Anirudh Bharadwaj, Aman Chadha, Yogesh Rawat Abstract Existing benchmarks for Vision-Language Models (VLMs) primarily evaluate spatio-temporal understanding on simple single-action videos, closed attribute sets and restricted entity types, failing to capture the freeform, multi-action interactions between diverse entities which characterize real-world video understanding. Furthermore, the lack of a systematic framework for analyzing model failures across complementary spatio-temporal axes hinders comprehensive evaluation. To address these gaps, we introduce VISTA , a V ideo I nteraction S patio- T emporal A nalysis benchmark designed for open-set, multi-entity and multi-action spatio-temporal understanding in VLMs. VISTA decomposes videos into interpretable entities, their associated actions, and relational dynamics, enabling multi-axis diagnostics and unified assessment of relational, spatial, and temporal understanding. Our benchmark integrates multiple datasets into a single interaction-aware taxonomy and comprises ~12K curated video-query pairs spanning diverse scenes and complexities. We systematically evaluate 11 state-of-the-art VLMs on VISTA, and break down aggregate performance across our taxonomy to reveal shortcomings and pronounced spatio-temporal biases obscured by traditional metrics. By providing detailed, taxonomy-driven diagnostics on a challenging dataset, VISTA offers a nuanced framework to guide advances in model design, pretraining strategies, and evaluation protocols. Overall, VISTA is the first large-scale, interaction-aware diagnostic benchmark for spatio-temporal understanding in VLMs. 1 Introduction Real-world video understanding requires reasoning about complex interactions among entities over time. From pedestrian–vehicle dynamics in autonomous driving to human–human and human–object interactions in surveillance. To achieve this, intelligent visual systems must determine which entities interact, how they interact, and where and when these interactions occur. This capability, broadly referred to as spatio-temporal understanding 45 ; 54 , extends beyond traditional object detection and motion analysis, requiring the joint modeling of spatial structure, temporal evolution, and inter-entity relationships. Figure 1 : VISTA vs. Existing Spatio-Temporal Benchmarks. Existing benchmarks focus on coarse, single-step spatio-temporal understanding without localization. VISTA utilizes grounded evaluation and enables detailed analysis of multi-entity, multi-action dynamics through coarse-to-fine categorization. Vision-Language Models (VLMs) 38 ; 42 ; 6 ; 12 ; 44 ; 37 ; 2 have significantly advanced spatio-temporal understanding by scaling architectures capable of jointly modeling visual and linguistic information. Early evaluation of these models relied on high-level VQA-style benchmarks, which were instrumental in measuring general capabilities. However, subsequent analyses 22 ; 26 ; 55 ; 27 ; 9 indicate that performance on such benchmarks can be confounded by linguistic priors limiting their ability to faithfully assess visual understanding. In response, the community has shifted toward grounded benchmarks that validate visual understanding through localization. Early efforts centered on tasks such as object tracking and action recognition, while more recent works 75 ; 29 ; 1 ; 64 ; 67 have introduced increasingly complex tasks involving multi-entity tracking and 4D reasoning reflecting a growing emphasis on capturing the relational dynamics underlying real-world spatio-temporal understanding. Despite this rapid progress, key limitations remain: existing benchmarks largely reduce performance to aggregate metrics, providing little insight into where and why models fail. Moreover, as model families expand, the lack of a structured evaluation framework renders consistent, fine-grained cross-model analysis increasingly intractable. To address these limitations, we introduce interaction as a unifying lens for structured evaluation. Through a systematic dataset aggregation and annotation pipeline, VISTA transforms video-query pairs into a coarse-to-fine interaction-centric representation, factorized into involved entities, spatio-temporal type, and fine-grained interaction type. An overview of the differences between previous work and ours is presented in Figure 1 . Our interaction-centric framework enables three diagnostic capabilities: (a) exposing hidden failure modes, as interaction-level evaluation surfaces systematic limitations masked by aggregate metrics; (b) characterizing generalization patterns, revealing how model behavior stratifies across interaction types, entity configurations, and query formulations; and (c) uncovering directional biases and tendencies, identifying consistent spatial, temporal, and semantic preferences embedded in modern VLMs. In summary, VISTA provides the first large-scale, interaction-focused diagnostic benchmark for spatio-temporal understanding in VLMs. Our contributions are threefold: 1. Interaction-centric diagnostic framework: We introduce a unified coarse-to-fine evaluation taxonomy that decomposes spatio-temporal grounding into interpretable interaction types enabling principled diagnostics across ~12K video-query pairs and 11 diverse models. 2. Systematic cross-model analysis: By aggregating and reorganizing multiple datasets under a common interaction-aware structure, we reveal consistent stratification patterns across model families, exposing how architecture, pretraining breadth, and instruction tuning shape understanding. 3. Bias and failure-mode characterization: Our analysis uncovers prominent failure modes - same-entity disambiguation, linguistic template preferences, and semantic-intent inflation - offering the first interaction-grounded view of systematic reasoning failures in modern VLMs. 2 Related Work VLM Benchmarks: The rapid progress of VLMs has been paralleled by increasingly sophisticated benchmarks designed to probe spatio-temporal understanding 54 ; 45 . From early datasets that evaluate general visual understanding 49 ; 28 ; 15 ; 26 ; 50 ; 69 , to fine-grained spatio-temporal localization tasks 59 ; 51 ; 52 ; 32 ; 48 ; 63 ; 68 ; 3 ; 24 . Unlike general video understanding benchmarks 5 ; 35 ; 74 ; 23 ; 46 that assess abstract comprehension, spatio-temporal benchmarks emphasize grounded reasoning. Within the segmentation community, MOSE 19 introduced crowded, heavily occluded scenes where targets frequently disappear and reappear, revealing that state-of-the-art VOS methods are brittle under such conditions. Its successor MOSEv2 21 extends this further with adverse weather, low-light environments, camouflaged objects, and non-physical targets. On the language-guided side, MeViSv2 20 shifts the focus from static-attribute referring expressions to motion -based descriptions that require genuine temporal reasoning across frames, supporting multi-target and no-target expressions. In the detection-style grounding setting, Spatio-Temporal Video Grounding (STVG) 73 ; 60 requires joint localization of entities across space and time from freeform relational queries, with recent efforts 75 ; 29 ; 1 ; 64 ; 67 further broadening this toward 4D reasoning, multi-object grounding, and grounded captioning. Yet across both settings, benchmarks largely reduce performance to aggregate metrics, providing little insight into where and why models fail. While prior diagnostic efforts 39 ; 22 shed light on performance across coarse-grained spatial and temporal categories, they neglect the intricate interaction semantics that critically influence spatio-temporal behavior. VISTA complements segmentation benchmarks by adopting STVG as its diagnostic probe, enabling structured evaluation of how and why models fail across diverse interaction types, entity configurations, and query formulations—dimensions that mask-based benchmarks do not directly expose. Spatio-temporal understanding in VLMs: Early spatio-temporal understanding modeled space and time independently under closed-set conditions - spatial models 53 ; 10 handled object detection within fixed categories while temporal models 58 ; 11 targeted action recognition under constrained label settings. Subsequent work advanced vision-language alignment across both spatial and temporal dimensions - through OVD 68 ; 47 and REC 16 , culminating in strong detectors such as GLIP 36 and Grounding-DINO 42 , while parallel progress in Moment Localization 3 ; 24 enabled language-guided temporal understanding 2 . The integration of LLMs into VLMs 34 ; 41 ; 6 further strengthened multimodal grounding, and video-centric extensions 5 ; 4 ; 70 ; 37 introduced joint spatial-temporal understanding. Despite evaluating increasingly complex spatio-temporal tasks, existing benchmarks reduce performance of VLMs to aggregate metrics, leaving the failure modes, biases, and systematic tendencies of modern VLMs largely undiagnosed. 3 The VISTA Benchmark Approach Encoders VISTA Spatio-Temp. Entity Image

Read the complete text