IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs

M-A-P ByteDance Inc · 2025

Read IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs by M-A-P ByteDance Inc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15415.txt.

Introduction

David Ma and co-authors present the first benchmark that tests whether video models can use a given image as context for what happens on screen, in a 2025 paper. Most tests grade image reasoning or plain video understanding, skipping the common case where a picture anchors the question. Their benchmark holds nine hundred sixty-seven videos and over two and a half thousand carefully annotated image-text queries, across thirteen tasks in five categories, split between perception and reasoning.

Leading open and closed models were graded, and the best reached only about twenty-nine percent accuracy. How a model takes its input, the number of frames, and the resolution all moved the score. A simple data synthesis trick did not close the gap, so the weakness is not mere format mismatch. The authors release code and data, and frame the task as a real frontier for multimodal models.

Catalog record

CREATORM-A-P ByteDance Inc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15415

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs M-A-P ByteDance Inc Abstract Existing evaluation frameworks for Multimodal Large Language Models (MLLMs) primarily focus on image reasoning or general video understanding tasks, largely overlooking the significant role of image context in video comprehension. To bridge this gap, we propose IV-Bench , the first comprehensive benchmark for evaluating Image-Grounded Video Perception and Reasoning . IV-Bench consists of 967 videos paired with 2,585 meticulously annotated image-text queries across 13 tasks (7 perception and 6 reasoning tasks) and 5 representative categories. Extensive evaluations of state-of-the-art open-source (e.g., InternVL2.5, Qwen2.5-VL) and closed-source (e.g., GPT-4o, Gemini2-Flash and Gemini2-Pro) MLLMs demonstrate that current models substantially underperform in image-grounded video Perception and Reasoning, merely achieving at most 28.9% accuracy. Further analysis reveals key factors influencing model performance on IV-Bench, including inference pattern, frame number, and resolution. Additionally, through a simple data synthesis approach, we demonstratethe challenges of IV - Bench extend beyond merely aligning the data format in the training proecss. These findings collectively provide valuable insights for future research. Our codes and data are released in https://github.com/multimodal-art-projection/IV-Bench . Figure 1 : (a) Video Categories. IV-Bench includes videos spanning five representative categories, ensuring diverse topical coverage. (b) Task distribution in IV-Bench. IV-Bench consists of a total of 13 tasks, which are categorized into two main types: 6 reasoning tasks and 7 perception tasks. (c) Model Performance on IV-Bench. All evaluated MLLMs exhibit limited performance on IV-Bench. Even on the best-performing task (Natural Language Inference), the highest achieved accuracy is merely 64.7%, with other tasks resulting in substantially lower scores. 1 Introduction Building upon the remarkable success of Large Language Models (LLMs) across various AI tasks [ Young et al., 2024 , Zhang et al., 2024b ] , Multimodal Large Language Models (MLLMs) have shown impressive capabilities in integrating and interpreting information from multiple modalities, such as text, images, and videos [ Liu et al., 2023 , Chen et al., 2024b , Guo et al., 2024 ] . Consequently, diverse benchmarks [ Yue et al., 2024 , Zhang et al., 2024a , Hu et al., 2025 , Cheng et al., 2025 , Wu et al., 2024a , Wu et al., 2024b , Zhu et al., 2024 ] have emerged to systematically evaluate their multimodal integration and task-solving capabilities. Images capture subtle details, such as facial features, that are difficult to accurately describe in text, especially when a person’s identity is unknown. For instance, if we lack a name, text descriptions may resort to vague clues like clothing color or hair color, which might not be sufficient to uniquely identify someone. In contrast, images offer complete visual cues that clearly depict these details. However, existing video benchmarks typically focus on general video comprehension with purely text-based queries, neglecting the critical scenario where static images provide essential context for video understanding [ Fang et al., 2025 , Li et al., 2024c ] . Image-grounded video perception and reasoning—the ability to leverage a image as critical contextual information to locate and interpret video content—is fundamental for numerous real-world applications, such as accurate scene interpretation, object recognition, and event retrieval. Despite its importance, there is currently no benchmark specifically designed to evaluate this capability. To address this critical research gap, we introduce IV-Bench , the first comprehensive benchmark for evaluating MLLMs in image-grounded video perception and reasoning tasks . IV-Bench comprises 967 videos paired with 2,585 meticulously annotated image-text queries, spanning 13 distinct tasks (7 perception and 6 reasoning tasks) across five representative categories (Knowledge, Film & Television, Sports Competition, Artistic Performance, and Life Record), as shown in Figure 1 (a). Notably, the images used in IV-Bench are sourced externally, not extracted from the videos, ensuring the generalizability and robustness of the benchmark. We conduct extensive evaluations on 23 state-of-the-art open-source models (e.g., InternVL2.5 [ Chen et al., 2024a ] , Qwen2.5-VL series [ Bai et al., 2025 ] ) and 4 closed-source models (e.g., Doubao-1.5-vison-pro [ Doubao Team, 2025 ] , GPT-4o [ OpenAi, 2024 ] , Gemini-2 Flash [ Team et al., 2024 ] , Gemini-2 Pro [ Team et al., 2024 ] ). Results demonstrate that existing models substantially struggle with image-grounded video perception and reasoning tasks, with the top-performing MLLMs achieving only 28.9% accuracy. Performance deteriorates further on complex reasoning tasks such as Temporal Reasoning, underscoring significant limitations and highlighting a critical research gap. Furthermore, ablation studies comparing models with and without image contexts reveal that smaller models show minimal benefit from incorporating images, whereas larger models significantly benefit from image contexts, particularly when images follow the video. Additional analysis identify other key factors influencing model performance, providing valuable insights for future research. To investigate whether the lack of video image formatted training data contributes to the performance gap on IV - Bench, we employ a simple synthetic data pipeline that automatically generates supervised fine - tuning examples from existing video QA datasets. Although this automated augmentation yields slight improvements, the gains remain minimal—suggesting that the challenges of IV-Bench arise from deeper image - grounded video understanding ability demands rather than mere data format alignment. We hope these results will motivate the development of more advanced methods for tackling image - grounded video perception and reasoning. In summary, our work has three-fold contributions: • IV-Bench . We present IV-Bench, the first comprehensive benchmark for image-grounded video perception and reasoning in MLLMs. IV-Bench comprises 967 videos paired with 2,585 meticulously annotated image-text queries, where the images, collected from external sources rather than extracted from the videos themselves, provide the essential context required to accurately answer the queries. The dataset spans 5 major categories and covers 13 distinct tasks (7 perception and 6 reasoning tasks), ensuring substantial diversity across various scenarios and task types. Moreover, two round quality control—one ensuring clarity, accuracy, and category labeling, and another confirming that both image and video are required to answer correctly, ensuring the high quality of IV-Bench. • Comprehensive Evaluation of MLLMs . We evaluate 27 state-of-the-art MLLMs, including the latest closed-source models (e.g., GPT-4o, Gemini-2-Flash and Gemini-2-Pro) and open-source models (e.g., InternVL2.5 and Qwen2.5-VL series). Our experiments reveal that current models perform sub-optimally on image-grounded video perception and reasoning, with the best model achieving only 28.9% overall accuracy and just 24.9% on reasoning tasks—clearly indicating an urgent need for enhanced image-grounded video perception and reasoning capabilities in MLLMs. • Insights for Future Research . Our analysis provides key insights to guide future research. Ablation studies indicate that increasing frame number and video resolution positively affect performance. Moreover, larger models significantly benefit from image contexts presented after the video, while smaller models show minimal improvements. Through a simple data synthesis approach, we demonstrate that the challenges of IV-Bench do not arise from a lack of video–image format alignment in training data, which underscoring the need for more advanced methods beyond mere format alignment. 2 IV-Bench Figure 2 : Representative examples from IV-Bench. Each sample consists of a video paired with an image-text query, comprising a query image and corresponding query text. The correct answer is marked in green , with relevant video frames also highlighted in green. 2.1 Overview IV-Bench is designed to evaluate image-grounded video perception and reasoning, aiming

Read the complete text