VideoLLM Benchmarks and Evaluation: A Survey
Yogesh Kumar · 2025
Read VideoLLM Benchmarks and Evaluation: A Survey by Yogesh Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03829.txt.
Introduction
Yogesh Kumar surveys how video language models are tested and scored in a 2025 paper. Models that watch and answer about video have raced ahead, yet judging them fairly lags behind. The survey maps the test sets in use and the rules that grade them.
Closed tests with fixed answers, open tests with free replies, and special tests for timing and space each get a chapter. Score trends for leading models are charted across the sets. Flaws include narrow content, shaky metrics, and little focus on why models answer as they do. The author urges richer, more open tests for the next wave.
Catalog record
| CREATOR | Yogesh Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoLLM Benchmarks and Evaluation: A Survey Yogesh Kumar Abstract The rapid development of Large Language Models (LLMs) has catalyzed significant advancements in video understanding technologies. This survey provides a comprehensive analysis of benchmarks and evaluation methodologies specifically designed or used for Video Large Language Models (VideoLLMs). We examine the current landscape of video understanding benchmarks, discussing their characteristics, evaluation protocols, and limitations. The paper analyzes various evaluation methodologies, including closed-set, open-set, and specialized evaluations for temporal and spatiotemporal understanding tasks. We highlight the performance trends of state-of-the-art VideoLLMs across these benchmarks and identify key challenges in current evaluation frameworks. Additionally, we propose future research directions to enhance benchmark design, evaluation metrics, and protocols, including the need for more diverse, multimodal, and interpretability-focused benchmarks. This survey aims to equip researchers with a structured understanding of how to effectively evaluate VideoLLMs and identify promising avenues for advancing the field of video understanding with large language models. I Introduction The advent of large language models (LLMs) has revolutionized the field of video understanding, establishing a new paradigm that transcends traditional computer vision approaches. Videos, as dynamic sequences of visual information often accompanied by audio and text, present unique challenges compared to static images due to their temporal nature, complex motion patterns, and higher-dimensional information space [ 1 ] . The integration of LLMs with video understanding capabilities has led to the development of VideoLLMs, a specialized class of models that leverage the reasoning abilities of LLMs to interpret and analyze video content. The evolution of video understanding methods has progressed through several distinct phases, beginning with conventional methods that relied on handcrafted features such as SIFT [ 2 ] , SURF [ 3 ] , and HOG [ 4 ] . This was followed by the emergence of early neural video models like DeepVideo [ 5 ] and two-stream networks [ 6 ] , which demonstrated the potential of deep learning for video analysis. The field then advanced to self-supervised video pretraining approaches such as VideoBERT [ 7 ] and VideoMAE [ 8 ] , which leveraged large-scale pretraining to improve performance on downstream tasks. The most recent phase involves the application of LLMs to video understanding, exemplified by models such as Video-LLaMA [ 9 ] , VideoChat [ 10 ] , and Video-ChatGPT [ 11 ] . These VideoLLMs demonstrate enhanced abilities in open-ended multi-granularity reasoning combined with commonsense knowledge, marking a significant advancement in the field. As Tang et al. [ 12 ] note in their comprehensive survey, VideoLLMs can be categorized based on their architecture and the role of the LLM, with different approaches offering unique advantages for various video understanding tasks. Evaluating VideoLLMs presents unique challenges due to the complexity of video content and the diverse capabilities these models aim to demonstrate. Traditional evaluation metrics designed for specific video understanding tasks may not fully capture the nuanced abilities of VideoLLMs, particularly their reasoning capabilities and contextual understanding. Therefore, specialized benchmarks and evaluation methodologies are essential for comprehensively assessing VideoLLM performance. This survey provides a focused examination of VideoLLM benchmarks and evaluation methodologies, building upon the broader survey by Tang et al. [ 12 ] . We analyze existing benchmarks, discussing their characteristics, strengths, and limitations. We explore various evaluation approaches, including closed-set evaluations that use predefined answers, open-set evaluations that allow for more flexible responses, and specialized evaluations for temporal and spatiotemporal understanding tasks. Furthermore, we highlight current limitations in VideoLLM evaluation and propose future research directions to advance the field. II VideoLLM Benchmarks Benchmarks serve as standardized platforms for evaluating and comparing the performance of VideoLLMs. This section provides a comprehensive overview of the most prominent benchmarks used in the field, analyzing their characteristics, compositions, and evaluation protocols. II-A Benchmark Characteristics and Composition VideoLLM benchmarks vary significantly in terms of size, content diversity, task focus, and evaluation protocols. Table I presents a comparative analysis of various benchmarks, highlighting key attributes such as the number of videos, clips, average video duration, and question-answer pairs. MSRVTT-QA [ 13 ] contains 2,990 videos with an average duration of 15.2 seconds and 72,821 question-answer pairs. The questions cover multiple aspects of video content, including objects, actions, and attributes. Similarly, MSVD-QA [ 13 ] comprises 504 videos with an average duration of 9.8 seconds and 13,157 question-answer pairs. Both benchmarks primarily focus on factual understanding of video content. TGIF-QA [ 14 ] features 9,575 short animated GIFs with an average duration of 3 seconds and 8,506 question-answer pairs. The questions in this benchmark are designed to test repetition counting, state transition recognition, and frame-based question answering, making it particularly suitable for assessing fine-grained action understanding. ActivityNet-QA [ 15 ] consists of 800 videos with significantly longer durations (average 111.4 seconds) and 8,000 question-answer pairs. This benchmark challenges models to understand complex activities and extended temporal sequences, requiring more sophisticated temporal reasoning capabilities. TVQA [ 16 ] incorporates 2,179 videos from TV shows, segmented into 15,253 clips with an average duration of 11.2 seconds, each accompanied by a question-answer pair. This benchmark introduces additional complexity by including dialogue and plot understanding. More recent benchmarks such as MVBench [ 17 ] , Video-Bench [ 18 ] , and VideoVista [ 19 ] contain videos with longer durations and a greater variety of domains, testing the generalization abilities of VideoLLMs across different contexts. For instance, VideoVista includes 3,400 clips from 894 videos with an average duration of 131 seconds and 25,000 question-answer pairs covering diverse topics and domains. CinePile [ 20 ] represents one of the largest benchmarks with 9,396 videos and 303,828 question-answer pairs, specifically designed to evaluate understanding of cinematic content. InfiniBench [ 21 ] features 1,219 videos with extremely long durations (average 4,460.4 seconds) and 108,200 question-answer pairs, testing VideoLLMs’ ability to handle long-form content. Specialized benchmarks like TempCompass [ 22 ] focus on specific aspects of video understanding, such as temporal reasoning, with 500 clips from 410 videos and 7,540 question-answer pairs designed to evaluate temporal cognition capabilities. This trend toward more specialized benchmarks reflects the growing recognition of the need to assess specific aspects of video understanding in greater depth. TABLE I: Comparison of VideoLLM benchmarks highlighting the total number of videos, clips, average video duration, and question-answer pairs. Benchmarks #QA Pairs #Clips #Videos Len.(s) TVQA [ 16 ] 15,253 15,253 2,179 11.2 How2QA [ 23 ] 2,852 2,852 1,166 15.3 STAR [ 24 ] 7,098 7,098 914 11.9 MSRVTT-QA [ 13 ] 72,821 2,990 2,990 15.2 MSVD-QA [ 13 ] 13,157 504 504 9.8 TGIF-QA [ 14 ] 8,506 9,575 9,575 3.0 ActivityNet-QA [ 15 ] 8,000 800 800 111.4 EgoSchema [ 25 ] 5,063 5,063 5,063 180.0 AutoEval-Video [ 26 ] 327 327 327 14.6 TempCompass [ 22 ] 7,540 500 410 11.4 Video-MME [ 27 ] 2,700 900 900 1,017.9 VideoVista [ 19 ] 25,000 3,400 894 131.0 CinePile [ 20 ] 303,828 9,396 9,396 160 SOK-Bench [ 28 ] 44,000 10,000 10,000 - SFD [ 29 ] 4,885 1,078 1,078 780 InfiniBench [ 21 ] 108,200 1,219 1,219 4,460.4 MMWorld [ 31 ] 1,599 1,910 1,910 102.3 VELOCITI [ 32 ] - 900 900 10.0 NExT-QA [ 33 ] 8,564 1,000 1,000 39.5 MVBench [ 17 ]