Agentic Keyframe Search for Video Question Answering
Sunqi Fan, Meng-Hao Guo, Shuojin Yang · 2025
Read Agentic Keyframe Search for Video Question Answering by Sunqi Fan, Meng-Hao Guo, Shuojin Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16032.txt.
Introduction
Sunqi Fan, Meng-Hao Guo, and Shuojin Yang let a language agent choose which video frames deserve attention. Their 2025 paper tackles video question answering, where watching everything is costly and most frames are filler.
Their method, AKeyS, chops the video into a tree of segments. A language agent then directs a classic search over that tree, estimating which branches promise answers and what expanding them costs, and stops once enough keyframes are gathered. On two standard benchmarks the system beat all earlier methods on accuracy per frame watched. In one test it scored higher than a leading rival while viewing under half its frames. The authors present agent-guided search as the efficient path to video understanding.
Catalog record
| CREATOR | Sunqi Fan, Meng-Hao Guo, Shuojin Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentic Keyframe Search for Video Question Answering Sunqi Fan, Meng-Hao Guo, Shuojin Yang Abstract Video question answering (VideoQA) enables machines to extract and comprehend key information from videos through natural language interaction, which is a critical step towards achieving intelligence. However, the demand for a thorough understanding of videos and high computational costs still limit the widespread applications of VideoQA. To address it, we propose Agentic Keyframe Search ( AKeyS ), a simple yet powerful algorithm for identifying keyframes in the VideoQA task. It can effectively distinguish key information from redundant, irrelevant content by leveraging modern language agents to direct classical search algorithms. Specifically, we first segment the video and organize it as a tree structure. Then, AKeyS uses a language agent to estimate heuristics and movement costs while dynamically expanding nodes. Finally, the agent determines if sufficient keyframes have been collected based on termination conditions and provides answers. Extensive experiments on the EgoSchema and NExT-QA datasets show that AKeyS outperforms all previous methods with the highest keyframe searching efficiency, which means it can accurately identify key information and conduct effective visual reasoning with minimal computational overhead. For example, on the EgoSchema subset, it achieves 1.8% higher accuracy while processing only 43.5% of the frames compared to VideoTree. We believe that AKeyS represents a significant step towards building intelligent agents for video understanding. The code is publicly available at https://github.com/fansunqi/AKeyS . 1 Introduction The rapid advancement of image-based Multimodal Large Language Models (MLLMs) [ 24 , 34 ] has significantly simplified image understanding tasks in daily life. Users can easily upload images to OpenAI’s GPT-4V or Google Gemini, ask questions about them, and receive responses via natural language interaction. However, video understanding presents greater challenges, and the development of Video Large Language Models (Video-LLMs) has notably lagged behind image-based MLLMs. Existing Video-LLMs often struggle to capture details in videos and lack a holistic understanding of video content [ 19 ] . Moreover, the computational overhead of Video-LLMs is substantially higher than that of Large Language Models (LLMs) and image-based MLLMs, hindering their commercial deployment. To address video understanding tasks in daily life more effectively, This paper focuses on the efficient extraction of keyframes, and analyzing them using image-based MLLMs for video understanding. \begin{overpic}[fig1_3.pdf] \put(72.0,25.7){LLoVi~\cite[cite]{[\@@bibref{Number}{zhang2024simplellmframeworklongrange}{}{}]}} \put(72.0,21.6){VideoAgent~\cite[cite]{[\@@bibref{Number}{wang2024videoagentlongformvideounderstanding}{}{}]}} \put(72.0,17.5){VideoTree~\cite[cite]{[\@@bibref{Number}{wang2024videotreeadaptivetreebasedvideo}{}{}]}} \put(72.0,13.7){AKeyS (Ours)} \end{overpic} Figure 1 : Demonstration of AKeyS ’s high frame efficiency. When processing the same number of video frames with the same (M)LLM, AKeyS achieves higher QA accuracy. At the same accuracy level (66%), AKeyS uses only about 1/4 of the frames required by VideoTree. Moreover, VideoTree clusters features of all frames during preprocessing, whereas AKeyS only has access to visible frames and does not utilize information from the rest. This experiment is conducted on EgoSchema [ 21 ] subset. Figure 2 : Comparison of three methods for analyzing a travel vlog: (1) Video-LLM can generate correct answers but is highly token-intensive; (2) The method of uniform frame sampling may introduce irrelevant content, leading MLLM to incorrect predictions; (3) The method of keyframe sampling for MLLM achieves both accuracy and efficiency. The keyframes relevant to the given question are highlighted in the figure. However, one key advantage of keyframe extraction is the ability to significantly reduce computational overhead while preserving essential information. Figure 2 presents three approaches to solving the VideoQA task. Among them, only the keyframe sampling based method achieves both accuracy and efficiency, highlighting the importance of keyframes in VideoQA task. However, a major challenge is how to effectively identify keyframes that contain the essential information needed to answer specific questions. This challenge becomes more pronounced in the context of long-form video understanding [ 21 , 43 ] , where the abundance of irrelevant information necessitates precise temporal localization of key content based on the question at hand. Addressing efficiency and accuracy keyframe location challenge is crucial in video understanding tasks. In this paper, we propose an efficient algorithm named AKeyS to tackle the video understanding and analysis problems, exemplified by VideoQA tasks. Drawing inspiration from both traditional search algorithms and modern language agents, our approach harnesses the cognitive capabilities of language agents, such as reasoning, planning, summarization and reflection, to guide and provide feedback to traditional search algorithms. This methodology effectively extracts key content from redundant information, similar to sifting wheat from chaff. Specifically, given an input video, AKeyS divides it into segments and extracts textual information from the representative frame of each segment using a Vision-Language Model (VLM), such as an image captioner. Then, it employs language agents to perform temporal comparisons and identify key content in an iterative, deepening process until reaching the termination condition. This leads to a tree-like search through the video until sufficient key information is found to answer the question. In our experiments, AKeyS achieves 63.1% accuracy on EgoSchema fullset [ 21 ] (surpassing the best baseline by 2.0%) and 77.4% average accuracy on NExT-QA [ 45 ] (surpassing the best baseline by 1.8%). In Figure 1 , we compare the frame efficiency of AKeyS with two baselines, highlighting its ability to effectively identify key information. In summary, AKeyS not only achieves state-of-the-art accuracy, but also exhibits the highest frame searching efficiency, making it a highly promising approach for real-world video analysis tasks. 2 Related Work Video Question Answering VideoQA is a typical subtask of video understanding, involving the comprehension, analysis, and responding to questions about video content. It comprehensively tests various capabilities of multimodal QA systems [ 45 , 51 , 23 ] , while benchmarks and datasets for VideoQA have been progressively focused on longer videos and more complex reasoning scenarios [ 21 , 7 , 44 , 52 ] . Early approaches for VideoQA typically employ neural networks (e.g., ResNet [ 11 ] , 3D convolutional neural networks [ 35 , 2 , 10 ] ) to extract visual features, while language models were used to process the questions. These two components are then aggregated to produce answers. With the advent of LLMs, the common practice is to use a pre-trained visual encoder to extract visual features, a projection layer to map visual representations into the text latent space of LLMs, and a pretrained LLM for response generation [ 49 , 39 , 18 , 17 ] . These Video-LLMs, with their extensive parameters, can model long contexts [ 30 , 42 , 32 ] and, through instruction tuning and alignment, better address VideoQA tasks, serving as foundational models in the video domain. Another popular method is based on (M)LLMs or agents [ 46 , 33 ] . Many works have achieved significant success by leveraging a range of agentic techniques, including prompting [ 48 , 50 ] , memory [ 6 , 13 , 40 ] , tools [ 5 , 9 , 47 ] , and planning [ 12 , 20 , 22 , 27 ] . These advancements have progressively contributed to developing an intelligent and powerful video agent. A detailed discussion of these works can be found in the Appendix Section A. Keyframe Extraction Another line of research focuses on extracting keyframes from videos, which is also highly relevant to our work. LVNet [ 25 ] selects keyframes using a small network that is specifically trained