Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models

Martin Q. Ma, Willis Guo, Aditya Agrawal, Ankit Gupta, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency · 2026

Read Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models by Martin Q. Ma, Willis Guo, Aditya Agrawal, Ankit Gupta, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01662.txt.

Introduction

Martin Q Ma and six co-authors answer long-video questions by choosing frames the way an active perceiver would: fetch what defies expectation. Uniform sampling of a long clip is costly and stalls in quality, so the frame budget needs aim.

Video Active Perception needs no training. A light text-conditioned video generator stands in for the model's prior knowledge of how scenes unfold. Frames that depart from that prior carry the news, and those are the frames the vision-language model is shown. Across five long-form and reasoning video sets, the method posts state-of-the-art zero-shot scores. It reaches them with far fewer frames per question than flagship baselines need, up to several times the frame efficiency. The picks also read as sensible to a human eye, landing on the moments the question is really about.

Catalog record

CREATORMartin Q. Ma, Willis Guo, Aditya Agrawal, Ankit Gupta, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01662

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models Martin Q. Ma, Willis Guo, Aditya Agrawal, Ankit Gupta, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency Abstract Large vision-language models (VLMs) have advanced multimodal tasks such as video question answering (QA). However, VLMs face the challenge of selecting frames effectively and efficiently, as standard uniform sampling is expensive and performance may plateau. Inspired by active perception theory, which posits that models gain information by acquiring data that differs from their expectations, we introduce Video Active Perception ( VAP ), a training-free method to enhance long-form video QA using VLMs. Our approach treats keyframe selection as data acquisition in active perception and leverages a lightweight text-conditioned video generation model to represent prior world knowledge. Empirically, VAP achieves state-of-the-art zero-shot results on long-form or reasoning video QA datasets such as EgoSchema, NExT-QA, ActivityNet-QA, IntentQA, and CLEVRER, achieving an increase of up to 5.6 × 5.6\times frame efficiency by frames per question over standard GPT-4o, Gemini 1.5 Pro, and LLaVA-OV. Moreover, VAP shows stronger reasoning abilities than previous methods and effectively selects keyframes relevant to questions. These findings highlight the potential of leveraging active perception to improve the frame effectiveness and efficiency of long-form video QA. 1 Introduction Multimodal foundational models, particularly large vision-language models (VLMs) [ 1 , 38 ] , have achieved remarkable results in tasks such as image captioning, text-to-image generation, and video question answering [ 25 ] . Long-form video question answering [ 49 , 30 ] stands out as a challenging and intriguing problem. It requires models to reason over complex dynamics, intricate scenes, and subtle visual details across extended time frames. Developing effective solutions for this task is practically important and poses compelling scientific challenges. However, long videos produce two challenges during VLM inference. The first is frame effectiveness : standard VLMs use one-frame-per-second (fps) sampling of frames, which is suboptimal as some video regions can be irrelevant while others can be information-dense and task-relevant [ 47 ] . In Table 7 , we show the performance of uniform sampling plateaus on the NeXT-QA dataset, suggesting that the uniformly sampled frames may not be the most effective. The second challenge is frame efficiency : processing a single hour of video can produce nearly four million tokens. The prohibitive token amounts hinders the practical deployment of VLMs, where real-world video applications such as autonomous driving, content summarization, and patient monitoring with thousands of hours of video data. Figure 1 : VAP outperforms standard uniform sampling GPT-4o, Gemini 1.5 and LLaVA-OV on EgoSchema, NExT-QA, ActivityNet-QA, IntentQA, and CLEVRER with up to 5.6\times more efficient on frame per question. VAP also achieves better latency under the same performance with GPT-4o mini compared to standard sampling due to using fewer frames. In this paper, we introduce Video Active Perception ( VAP ) , a training-free method that improves both the frame effectiveness and efficiency for long-form video question answering, by drawing inspiration from “active perception” [ 4 , 2 , 3 , 45 , 31 , 6 , 40 , 48 ] , which posits that intelligent agents should guide data acquisition through a priori knowledge of the world. Concretely, the data acquisition corresponds to selecting key video frames in videos for VLM inference, while the a priori knowledge is embodied by a lightweight, pre-trained text-conditional video generation model that encodes complex prior visual knowledge. Initial uniform sampled frames, along with questions and possible answers, are the conditional signals to the diffusion model. The latents from the diffusion model are then compared against a separate set of latents encoded from all real frames. The real frames whose latent are the most distinct from the diffusion latents are selected for VLM inference. Unlike previous frame selection methods [ 46 , 7 , 47 ] , VAP does not require a captioning model and operates in a single selection round rather than through multi-round selection or complex data structures, providing a simplified, unified approach. VAP demonstrates substantial empirical improvements in both performance and frame efficiency across long-form video QA datasets such as EgoSchema [ 30 ] , NExT-QA [ 49 ] , ActivityNet-QA [ 59 ] , IntentQA [ 23 ] , and the reasoning dataset CLEVRER [ 55 ] . Performance-wise, by selectively focusing on frames that diverge from prior knowledge, VAP obtains state-of-the-art zero-shot results with 68.1\% on EgoSchema, 81.4\% on NExT-QA, 64.6\% on ActivityNet-QA, 72.2\% on IntentQA, and 40.5\% on CLEVRER (Table 1 ). It also outperforms standard uniform sampling baselines with GPT-4o, Gemini 1.5 Pro, and LLaVA-OV-72B [ 22 ] (Figure 1 ). For frame efficiency, VAP achieves up to a 5.6\times increase in efficiency by frames per question with similar or better performance (Figure 1 ). Furthermore, compared to the standard GPT-4o mini, VAP achieves better efficiency through a lower latency (Figure 1 and Table 4 ) with the same performance due to the use of fewer frames. VAP is both VLM-agnostic and task-agnostic, which makes it applicable in different vision-language models and tasks. Our quantitative analysis reveals that VAP exhibits stronger visual reasoning capabilities than previous caption-based models on challenging temporal and causal reasoning tasks (Section 4.5 ). Qualitative results illustrate that VAP effectively selects unseen pivotal frames relevant to the questions for explanatory and counterfactual reasoning (Section 4.6 ). These findings demonstrate the effectiveness of leveraging prior world knowledge from a generation model to enhance both the frame effectiveness and efficiency of long-form video question answering, highlighting the potential of a more intelligent data acquisition strategy for VLM inference. 2 Related work Active perception, active feature and input acquisition. Active perception [ 45 , 4 , 2 , 36 , 3 , 41 , 60 , 62 ] refers to the theory in which an agent actively acquires its sensory input to optimize perception based on feedback from an a priori knowledge model. Similar ideas include active sensing [ 15 , 58 , 52 , 56 ] , active feature acquisition [ 39 , 43 , 20 ] , and active data acquisition [ 18 , 19 ] . In this work, we leverage the active perception framework by using a video generation model to represent a priori knowledge and selecting keyframes as a data acquisition process for VLM inference. Frame selection methods for inference-time video question-answering. Long-form video question answering presents significant challenges for large vision-language models (VLMs) due to the computational cost. Several approaches rely on LLMs iteratively reasoning over captions of densely sampled video frames [ 26 , 28 , 9 , 61 , 47 ] . Recently, agent-based methods [ 53 , 42 , 7 , 46 ] have also been proposed, though many of these agents still rely on video frame captions. In addition to video frame captions, other methods leverage external tools like object detection to build a memory module for RAG [ 27 , 14 , 32 ] Elsewhere, other methods represent videos as graphs [ 12 ] , image grids [ 17 ] , or perform spatiotemporally pooled visual features [ 50 ] . Among all approaches, many methods rely on video frame captions. However, caption-based methods are sensitive to captioner quality [ 61 ] . Furthermore,

Read the complete text