ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, Dongyan Zhao · 2025

Read ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models by Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, Dongyan Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09313.txt.

Introduction

Yueqian Wang and co-authors published this study in 2025. It benchmarks video language models on proactive interaction: speaking up at the right moment, not just when asked. In proactive chat, a system watching a video must choose its own response times. That timing choice breaks classic grading, which reads only the words. The team built ProactiveVideoQA, the first bench for this skill, and PAUC, a score that folds response timing into quality.

Many baseline systems were run on the bench, and a user study checked human taste. PAUC agreed with people's preferences better than text-only metrics. The authors argue timing is part of helpfulness, not a side detail. A model that answers late can be right and useless at once. Their tools give this young research area a shared yardstick.

Catalog record

CREATORYueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, Dongyan Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09313

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, Dongyan Zhao Abstract With the growing research focus on multimodal dialogue systems, the capability for proactive interaction is gradually gaining recognition. As an alternative to conventional turn-by-turn dialogue, users increasingly expect multimodal systems to be more initiative, for example, by autonomously determining the timing of multi-turn responses in real time during video playback. To facilitate progress in this emerging area, we introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system’s ability to engage in proactive interaction. Since model responses are generated at varying timestamps, we further propose PAUC, the first metric that accounts for the temporal dynamics of model responses. This enables a more accurate evaluation of systems operating in proactive settings. Through extensive benchmarking of various baseline systems on ProactiveVideoQA and a user study of human preferences, we show that PAUC is in better agreement with human preferences than traditional evaluation metrics, which typically only consider the textual content of responses. These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios. 1 1 1 Project homepage: https://github.com/yellow-binary-tree/ProactiveVideoQA 1 Introduction Figure 1: A demonstration of offline, online and proactive interaction. Figure 2: An example of user journey map. The content in the figure is fictional only for demonstration purpose. Recently, video multimodal large language models (Video MLLMs) have undergone rapid development. With increasingly powerful video understanding capabilities and support for diverse input modalities Li et al. 2024a ; Zhang et al. 2024a ; Bai et al. 2025 ; Chen et al. 2024b ; Zhang et al. 2024b ; Xu et al. 2025 , MLLMs are being deployed across a growing range of real-world scenarios. Beyond advancements in model architecture and training paradigms, there is also a surge of interest in exploring novel interaction paradigms between users and models. A comparative illustration of these interaction methods is presented in Fig. 2 . In offline interaction, users must upload the entire video before posing any questions, allowing the model to generate responses after consuming the whole video. In contrast, online interaction allows users to query the model in real time as the video plays, with the model required to respond immediately using only the information observed up to that point. In addition to offline and online paradigms, proactive interaction has emerged as a promising and increasingly studied direction in video-text MLLMs Chen et al. 2024a ; Wang et al. 2024 ; Qian et al. 2025 ; Yao et al. 2025 . The defining characteristic of proactive interaction is that the model autonomously determines when to respond during video playback, rather than replies solely in response to user-initiated queries. This capability necessitates continuous monitoring of evolving visual and textual cues, real-time detection of salient moments, and timely, contextually appropriate responses. Proactive video MLLMs hold significant potential for real-time scenarios, including live stream understanding, intelligent surveillance, egocentric assistants, and socially interactive AI agents. While several models claim to have proactive response capabilities, their evaluations are often conducted on benchmarks that do not actually require such novel interaction. For example, most experiments are performed in offline settings where models are not required to autonomously determine when to respond, and are evaluated using multiple-choice questions rather than open-ended dialogue, which significantly differs from real-world application scenarios. Although some benchmarks do include tasks that require response timing decisions Lin et al. 2024 ; Wang et al. 2025 , they still suffer from critical limitations. Specifically: (1) the questions and answers are overly simplistic (e.g., “Inform me when [event] happens”), thereby evaluating only the timing of the response while ignoring the quality of its content; or (2) they have only a single round of response, rather than supporting multi-turn, context-aware interactions. A more fundamental issue is the lack of appropriate evaluation metrics. Unlike in offline or online interactions, proactive interaction allows the model autonomously generating a sequence of responses at varying points in time, with the output evolving continuously. As such, evaluation methods must capture this temporal progression and take the models’ timing strategy into account, rather than relying solely on static snapshots or isolated response instances. To address this gap and advance research on proactive interaction, we introduce ProactiveVideoQA, the first comprehensive benchmark specifically designed to evaluate the proactive interaction capabilities of MLLMs. ProactiveVideoQA specifically targets the task of question answering: given a question presented at the very beginning of the video, the system must proactively detect when relevant information appears in one or more video segments and initiate responses accordingly as the video progresses. Compared to more open-ended interactive tasks, this question-answering setup offers relatively objective evaluation criteria (e.g., the availability of ground-truth answers), making it particularly suitable for academic research. By constructing a suite of diverse tasks and collecting videos from a wide range of sources, ProactiveVideoQA encompasses several representative application scenarios for proactive interaction. It includes a broad spectrum of video topics, integrates multiple modalities, and supports multi-turn outputs to reflect realistic and varied use cases. In addition, we propose PAUC (Proactive Area Under Curve), a novel evaluation metric tailored to better capture the performance of proactive interaction systems. The design of PAUC is inspired by the user journey map Huo et al. 2023 , a widely used visualization tool in human-computer interaction research. An example of user journey map is shown in Fig. 2 . A user journey map typically employs a line graph to illustrate how users interact with a system over time, capturing the temporal dynamics of their experiences, including shifts in emotion, engagement, and pain points—rather than relying on static snapshots. Analogously, PAUC employs a line graph to track how the quality of a model’s responses evolves throughout the video, thereby highlighting the inherently dynamic nature of proactive interaction. This temporal perspective sets PAUC apart from traditional metrics designed for offline interactions, which evaluate only static textual outputs. To summarize, this work presents three key contributions: (1) Benchmark with Multi-turn, fully open-ended answers. Unlike most existing video question answering benchmarks that predominantly adopt multiple-choice formats, ProactiveVideoQA features fully open-ended questions requiring multi-turn, free-form textual responses. While this design introduces additional challenges for evaluation, it offers a more realistic and comprehensive assessment of a model’s interactive capabilities and is closer to real-world application scenarios. (2) Benchmark with diverse topics and multimodal inputs. ProactiveVideoQA includes videos covering a wide array of topics that are highly relevant to proactive interaction, such as web videos, egocentric recordings, television series, and surveillance footage. It also incorporates multiple input modalities including text, video, and speech, reflecting the complexity and richness of real-world use cases. (3) A reply time-aware metric for proactive evaluation. We introduce PAUC, a novel evaluation metric that captures the evolving nature of model responses in proactive interaction settings. By explicitly modeling how response quality changes over time, PAUC provides stronger alignment with human judgments and better reflects the user experience. 2 Related Works 2.1 Video Understanding Benchmarks Recent years have witnessed a surge in video understanding and question-answering benchmarks. Li et al. 2023b ; Fu et al. 2024 ; Cai et al. 2024 ; Li et al. 2023c ; Li et al. 2023a ; Li

Read the complete text