FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering

Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekaterina Eroshenko, Ekaterina Derevyanka, Dmitry Yudin · 2026

Read FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering by Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekaterina Eroshenko, Ekaterina Derevyanka, Dmitry Yudin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04349.txt.

Introduction

Zemskova and colleagues pick keyframes for long egocentric video questions. More frames slow multimodal models and cut accuracy, so frame choice is key. FocusGraph finds query clips with a light Scene-Graph selector reading compact graph captions. Keyframes then come from an offline program evolved sparse flow method with no learned parameters at inference. It leads on FindingDory and HourVideo while cutting question time cost.

Catalog record

CREATORTatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekaterina Eroshenko, Ekaterina Derevyanka, Dmitry Yudin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44642
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekaterina Eroshenko, Ekaterina Derevyanka, Dmitry Yudin Abstract The ability to understand long videos is vital for embodied intelligent agents, because their effectiveness depends on how well they can accumulate, organize, and leverage long-horizon perceptual memories. Recently, multimodal LLMs have been gaining popularity for solving the long video understanding task due to their general ability to understand natural language and to leverage world knowledge. However, as the number of frames provided to an MLLM increases, the quality of its responses tends to degrade, and inference time grows. Therefore, when using MLLMs for long video understanding, a crucial step is selecting key frames from the video to answer user queries. In this work, we develop FocusGraph, a framework for keyframe selection for question answering over long egocentric videos. It leverages a lightweight trainable Scene-Caption LLM Selector that selects query-relevant clips based on their graph-based captions, and a training-free method for selecting keyframes from these clips. Unlike existing methods, the proposed Scene-Caption LLM Selector does not rely on the original sequence of low-resolution frames; instead, it operates on a compact textual representation of the scene. We then design a training-free Patch-wise Sparse-Flow Retention (PSFR) method to select keyframes from the resulting sequence of clips, which are fed into an MLLM to produce the final answer. Together, these components enable FocusGraph to achieve state-of-the-art results on challenging egocentric long-video question answering benchmarks, including FindingDory and HourVideo, while significantly reducing inference time relative to baseline approaches. 1 Introduction Understanding long videos is a fundamental task for embodied intelligent agents operating in real-world environments [ 25 , 1 ] . Such agents must continuously perceive their surroundings, interact with objects, and reason over extended temporal horizons [ 5 , 8 ] . Their performance therefore critically depends on the ability to accumulate, organize, and leverage long-term perceptual memories [ 46 , 16 ] . This challenge has recently gained increasing attention in long-video question answering (LVQA), where an agent answers natural language queries over potentially hours-long video streams [ 41 , 4 , 48 ] . Figure 1 : We propose FocusGraph, a modular method that addresses navigation and question-answering tasks on long videos using hierarchical textual scene graphs and a training-free fast key frame selection method called PSFR. Embodied videos, typically captured from an egocentric or agent-centric perspective, present unique challenges compared to conventional internet videos [ 6 , 10 ] . They exhibit frequent camera motion, occlusions, viewpoint changes, and visually repetitive content [ 10 ] , while the information required to answer a question may be sparsely distributed across the entire video [ 44 ] . For example, to solve the task «Navigate to the receptacle that you placed an object on right before you started interacting with the can» [ 44 ] , the agent must answer the question: «What is the receptacle that you placed an object on right before you started interacting with the can?». To answer this question, the agent must be able to recognize actions, identify objects by their semantic category, and perform temporal reasoning. Beyond the challenges inherent to question answering in embodied videos, such videos also provide additional information: during recording, the camera moves through a continuous 3D space. This makes it possible to exploit the spatio-temporal coherence of observations, reconstruct the agent’s trajectory, and establish stable spatial relationships between objects and scenes over time. As a result, LVQA for embodied videos represents a particularly promising research direction that calls for the development of specialized methods. Recent progress in multimodal large language models (MLLMs) has shown promising results for video understanding tasks, leveraging powerful natural language reasoning capabilities and world knowledge [ 51 , 39 , 2 ] . However, directly applying MLLMs to long videos introduces significant challenges, including sharply increased inference cost [ 37 ] and degraded answer quality as number of input frames grows for open-source models [ 40 , 44 ] . These limitations arise from constrained context capacity and attention «dilution» effect, making naive scaling to long videos impractical. Existing approaches address this problem mainly in two ways: either by compressing the visual representation of individual frames to reduce the number of tokens [ 13 , 21 , 22 , 52 , 32 ] , or by selecting a subset of query-relevant frames [ 53 , 35 , 19 , 20 , 49 , 34 ] . However, visual feature compression inevitably leads to information loss, while frame selection requires a well-designed retrieval strategy, particularly in scenarios where answering a question depends on the full long-term context of the video. As a result, two-stage methods appear especially promising: in the first stage, a coarse frame selection is performed using an MLLM operating on a compressed scene representation, followed by a more detailed analysis of the selected segments in the second stage. Nevertheless, existing two-stage approaches [ 47 , 43 , 38 ] treat video as a sequence of images constraining the maximum number of frames and events that an MLLM can process as input, while additionally reducing computational efficiency and overall inference speed. In this work, we address these challenges by introducing an efficient, query-aware framework for long-video question answering in embodied settings. Motivated by the strong redundancy in long embodied videos and the localization of relevant information to a few semantically meaningful moments, we decouple long-video understanding into two complementary stages: (1) lightweight, query-relevant clip selection trainable model, and (2) training-free identification of key frames in the selected clips. Unlike prior frame selection approaches, our frame selector operates on a compact scene caption representation rather than raw low-resolution frame sequences. This representation encodes objects, interactions, and their temporal relationships, enabling efficient long-horizon reasoning with reduced computational cost. The selected clips are converted into a sequence of frames, some of which contain redundant information. To extract key frames from these clips, we introduce a fast, training-free algorithm for key frame selection based on tracking changes in the image sequence. By combining graph-based temporal abstraction with query-conditioned selection and optical flow for key frame extraction, our method achieves both accuracy and efficiency. We evaluate the proposed approach on challenging egocentric LVQA benchmarks, including FindingDory and HourVideo, where questions require long-horizon reasoning and precise temporal grounding. Our method achieves state-of-the-art performance on these benchmarks while substantially reducing inference time compared dense frame processing baselines. In summary, our contributions are the following: • We propose FocusGraph, a novel framework for embodied videos that combines query-conditioned clip selection with training-free key frames idnetification based on Patchwise Sparse-Flow Retention algorithm (PSFR). • We introduce a hierarchical textual graph-based scene representation for efficient long-horizon reasoning, enabling lightweight and scalable frame selection independent of raw frame sequences. • We demonstrate state-of-the-art performance and improved inference efficiency on challenging egocentric long-video question answering benchmarks. Our results highlight the importance of selective temporal abstraction and structured representations for effective long-video question answering in embodied environments. 2 Related works MLLMs for Long Video Understanding. Long-form video understanding has emerged as a central challenge in multimodal reasoning, as increasing video length leads to higher inference costs and degraded performance in visual language models due

Read the complete text