An Empirical Study on How Video-LLMs Answer Video Questions

Chenhui Gou, Ziyu Ma, Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, Hamid Rezatofighi · 2025

Read An Empirical Study on How Video-LLMs Answer Video Questions by Chenhui Gou, Ziyu Ma, Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, Hamid Rezatofighi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15360.txt.

Introduction

Chenhui Gou, Ziyu Ma, and colleagues probe how video language models answer questions about footage in a 2025 study. Such models score well on benchmarks, yet most research chases higher scores rather than asking what the model uses inside. Without that understanding, failures are hard to predict or fix.

Attention knockouts, which silence chosen parts of the input, serve as the main tool. Variants block temporal information, spatial information, or other channels one at a time. Watching scores fall reveals which cues the model truly leans on. The study offers a template for auditing video models before trusting them in classrooms or clinics.

Catalog record

CREATORChenhui Gou, Ziyu Ma, Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, Hamid Rezatofighi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15360

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study on How Video-LLMs Answer Video Questions Chenhui Gou, Ziyu Ma, Zicheng Duan, Haoyu He, Feng Chen, Akide Liu, Bohan Zhuang, Jianfei Cai, Hamid Rezatofighi Abstract Taking advantage of large-scale data and pretrained language models, Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions. However, most existing efforts focus on improving performance, with limited attention to understanding their internal mechanisms. This paper aims to bridge this gap through a systematic empirical study. To interpret existing VideoLLMs, we adopt attention knockouts as our primary analytical tool and design three variants: Video Temporal Knockout, Video Spatial Knockout, and Language-to-Video Knockout. Then, we apply these three knockouts on different numbers of layers (window of layers). By carefully controlling the window of layers and types of knockouts, we provide two settings: a global setting and a fine-grained setting. Our study reveals three key findings: (1) Global setting indicates Video information extraction primarily occurs in early layers, forming a clear two-stage process—lower layers focus on perceptual encoding, while higher layers handle abstract reasoning; (2) In the fine-grained setting, certain intermediate layers exert an outsized impact on video question answering, acting as critical outliers, whereas most other layers contribute minimally; (3) In both settings, we observe that spatial-temporal modeling relies more on language-guided retrieval than on intra- and inter-frame self-attention among video tokens, despite the latter’s high computational cost. Finally, we demonstrate that these insights can be leveraged to reduce attention computation in Video-LLMs. To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content, offering interpretability and efficiency perspectives for future research. Introduction Figure 1: Typical architecture design for Video-LLMs. Video Large Language Models (Video-LLMs) have recently shows their strong ability to understand video content and answer various types of questions ( Zhang et al. 2024c ; Zhang et al. 2024d ; Yao et al. 2024 ; Bai et al. 2025 ; Chen et al. 2024b ; Cheng et al. 2024 ) . Most recent research on Video-LLMs focuses on improving model performance, e.g., enhancing the quality and scale of video instruction datasets ( Zhang et al. 2024d ; Yao et al. 2024 ; Bai et al. 2025 ; Chen et al. 2024b ; Cheng et al. 2024 ) , extending the input frame length ( Zhang et al. 2024a ; Xue et al. 2024 ; Liu et al. 2024b ) , and refining positional encoding for video tokens ( Liu et al. 2025 ; Ge et al. 2024 ; Wei et al. 2025 ) . However, there remains limited understanding of their internal mechanisms—specifically, how they process and reason about video content. Gaining deeper insights into these mechanisms is crucial for enhancing explainability, improving model efficiency, and facilitating future model developments. In the image domain, many researchers have attempted to increase the interpretability of these large multimodal models to avoid purely black-box usage. These works study how the internal states of MLLMs correspond to their external behaviors ( Zhang et al. 2025 ; Kaduri, Bagon, and Dekel 2024 ; Basu et al. 2024 ; Zhao et al. 2024 ; Zhang et al. 2024b ) . This includes analyses of the information flow from the image to the formation of different stage patterns ( Zhang et al. 2025 ; Chen et al. 2024a ; Lin et al. 2025 ) , undesirable content generation patterns in logit distributions ( Zhao et al. 2024 ) , two-stage pattern and safety mechanism fine grainedization ( Xu et al. 2024 ) , grounding and evolution of object-related visual cues ( Neo et al. 2024 ; Schwettmann et al. 2023 ; Ma et al. 2024 ) , information storage in model parameters ( Basu et al. 2024 ) , and reduction of redundant visual tokens ( Zhang et al. 2024b ) . Compared to the abundant studies on the interpretability of MLLMs in the image domain, the high-dimensional video domain remains largely unexplored. A recent work ( Xiao et al. 2025 ) conducts comprehensive experiments to analyze the behavior of various Video-LLMs and reports several interesting observations: these models excel at VideoQA but falter in temporal grounding, and they are sensitive to language variations while being less sensitive to video perturbations. While this work primarily focuses on analyzing the models’ external behaviors, their internal transparency remains largely unexplored. Our work takes a step toward uncovering the internal patterns of existing Video-LLMs and understanding how these patterns relate to their strong performance in video question answering. Modern Video-LLMs ( Zhang et al. 2024a ; Wang et al. 2025 ; Li et al. 2024a ; Zhang et al. 2024c ) typically follow a similar architecture: a pre-trained visual encoder that converts video into tokens, a projection layer that maps these tokens into the language space, and a large language model (LLM) that takes both video and language tokens to generate responses. Each video token is processed by every layer of the LLM and interacts with other video tokens and question tokens through attention mechanisms. This facilitates information extraction and propagation across layers, ultimately contributing to the final answer. Within each layer, the attention mechanism can be decomposed into three types: temporal attention (across video frames), spatial attention (within each frame), and language-to-video attention. We design three corresponding attention knockouts that selectively disable one specific type of attention, allowing us to analyze its individual impact. To gain insights at different granularities, we investigate the internal patterns of Video-LLMs under two settings: a global setting and a fine-grained setting by carefully controlling the knockout layer range and the type of attention knockout. In the global setting, we explore two questions: (1) Do Video-LLMs exhibit a clear two-stage behavior similar to image-level VLMs—i.e., perceptual encoding in early layers and semantic reasoning in later layers—or do they follow a distinct paradigm? (2) Globally, how does each attention type contribute to video QA performance? In the fine-grained setting , we explore: (3) How does each attention type impact VideoQA across different layers? We analyze a range of representative Video-LLMs, including LongVA ( Zhang et al. 2024a ) , InternVideo2.5 ( Wang et al. 2025 ) , LLaVA-OneVision ( Li et al. 2024a ) , and LLaVA-Video ( Zhang et al. 2024c ) , on mainstream VideoQA benchmarks with diverse task types and video lengths. These include short-duration multi-task video QA: MVBench ( Li et al. 2024b ) , medium-duration egocentric VideoQA: EgoSchema ( Mangalam, Akshulakov, and Malik 2023 ) , and long-duration open-domain VideoQA: Video-MME ( Fu et al. 2024 ) . Through extensive experiments comprising over 300 data points, we summarize our key findings: (1) Observation under global setting: Applying a language-to-video attention knockout starting from a certain proportion of layers (e.g., 60% of the entire model) does not show a significant impact on performance across various benchmarks and models, as shown in (2) Observation under global setting: Applying a full language-to-video attention knockout leads to a significant performance drop, which is substantially larger than that caused by temporal or spatial knockouts, as shown in fig. 5 . (3) Observation under fine-grained setting: For most individual layers, language-to-video attention knockouts have a stronger impact than temporal or spatial attention knockouts, as shown

Read the complete text