Object-centric Video Question Answering with Visual Grounding and Referring

Haochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, [3pt] Xiaolong Jiang, Yao Hu, Weidi Xie, Stratis Gavves, [3pt] University of Amsterdam SAI, Shanghai Jiao Tong University Xiaohongshu Inc · 2025

Read Object-centric Video Question Answering with Visual Grounding and Referring by Haochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, [3pt] Xiaolong Jiang, Yao Hu, Weidi Xie, Stratis Gavves, [3pt] University of Amsterdam SAI, Shanghai Jiao Tong University Xiaohongshu Inc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19599.txt.

Introduction

Haochen Wang and seven co-authors present an object centric video question answering model with visual grounding and referring. Video language models grasp broad content yet answer only in text, which limits multi round interaction about specific objects. Their model accepts visual prompts as input and grounds objects in its output, in both directions.

A spatial temporal overlay module propagates a visual prompt marked at any single frame to all other frames of the video. A hand curated instruction dataset supplies question answer pairs that require object level reasoning. Across twelve benchmarks and six tasks, the model beats baselines on question answering and referring segmentation alike. The design supports richer human video interaction for analysis tools.

Catalog record

CREATORHaochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, [3pt] Xiaolong Jiang, Yao Hu, Weidi Xie, Stratis Gavves, [3pt] University of Amsterdam SAI, Shanghai Jiao Tong University Xiaohongshu Inc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19599

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Object-centric Video Question Answering with Visual Grounding and Referring Haochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, [3pt] Xiaolong Jiang, Yao Hu, Weidi Xie, Stratis Gavves, [3pt] University of Amsterdam SAI, Shanghai Jiao Tong University Xiaohongshu Inc Abstract Video Large Language Models (VideoLLMs) have recently demonstrated remarkable progress in general video understanding. However, existing models primarily focus on high-level comprehension and are limited to text-only responses, restricting the flexibility for object-centric, multi-round interactions. In this paper, we make three contributions: (i) we address these limitations by introducing a VideoLLM model, capable of performing both object referring for input and grounding for output in video reasoning tasks, i.e. , allowing users to interact with videos using both textual and visual prompts; (ii) we propose STOM (Spatial-Temporal Overlay Module), a novel approach that propagates arbitrary visual prompts input at any single timestamp to the remaining frames within a video; (iii) we present VideoInfer , a manually curated object-centric video instruction dataset featuring question-answering pairs that require reasoning. We conduct comprehensive experiments on VideoInfer and other existing benchmarks across video question answering and referring object segmentation. The results on 12 benchmarks of 6 tasks show that our proposed model consistently outperforms baselines in both video question answering and segmentation, underscoring its robustness in multimodal, object-centric video and image understanding. Project page: https://qirui-chen.github.io/RGA3-release/ . 1 1 footnotetext: Both authors contribute equally to this work. 2 2 footnotetext: Corresponding author. 1 Introduction Figure 1 : RGA3 enables both referring and grounding capabilities for object-centric video understanding. In contrast to previous VideoLLMs, which are limited to text-only responses with mask reference, RGA3 can process arbitrary visual prompts and generate both textual answers and segmentation masks . Video question answering (VideoQA) refers to the task that enables machines to understand and reason about videos. As a natural way for both humans and embodied agents ( e.g. , robots or virtual assistants) to perceive the world, videos capture rich, continuous streams of data with complex spatial-temporal information, such as dynamic scenes, evolving interactions, and object behaviors. Despite the tremendous progress, existing Video Large Language Models (VideoLLMs) [ 13 , 30 , 61 , 71 , 33 , 14 , 9 ] have primarily focused on holistic scene comprehension, such as answering high-level questions about videos, rather than addressing fine-grained spatial-temporal queries on objects, limiting their ability to resolve detailed object-centric reasoning in complex, dynamic video content. This paper proposes to advance the task of multi-modal, object-centric video understanding by introducing RGA3 ( R efer and G round A nything A nytime at A ny Granularity), a unified framework designed to ground objects of interest and answer detailed questions about them in videos. In detail, the resulting model enables: (i) integrating arbitrary visual prompts—such as masks, bounding boxes, arrows, points, or scribbles—with textual inputs to perform complex reasoning tasks about specific objects in videos, and (ii) processing video and textual queries to generate segmentation masks for objects referred to in the query. By combining these capabilities, RGA3 facilitates object-centric reasoning, enabling more precise and interactive video understanding. Our core innovation of RGA3 lies in two components that enable object-centric video reasoning. The first is the Spatial-Temporal Overlay Module (STOM), which allows the model to process arbitrary visual prompts—such as masks, bounding boxes, arrows, points, or scribbles—at any timestamp within a video. This enhances the model’s interactivity and flexibility in handling spatial-temporal reasoning tasks. The second is to unify the object-centric question-answering and segmentation tasks for both images and videos based on the multimodal large language models (MLLMs) and SAM2 [ 49 ] , which enables the generation of segmentation masks for the referred objects in the query, while simultaneously reasoning over visually grounded questions. By integrating these two designs, RGA3 provides a unified framework for fine-grained object grounding and reasoning in videos. In addition to the development of model architecture, we introduce an object-centric video instruction dataset with manually crafted questions and reasoning process in the answers, namely VideoInfer . Unlike existing datasets [ 69 ] , which construct questions through automated pipelines that largely focus on object descriptions, resulting in a large proportion of perception questions, our proposed dataset emphasizes reasoning and long-term temporal understanding. VideoInfer contains around 29K question-answer (QA) pairs, and significantly surpasses existing datasets in question complexity and reasoning depth. We conduct extensive evaluations of RGA3 on VideoInfer as well as 11 existing benchmarks, including referring visual question-answering and referring/reasoning object segmentation tasks at both image and video levels. The results demonstrate that RGA3 achieves state-of-the-art performance across both referring and grounding tasks. The rest of the paper is organized as follows: In Sec. 2 , we provide an overview of recent Multi-modal Large Language Models (MLLMs) and existing approaches for referring and grounding tasks; Sec. 3 details our proposed RGA3 framework and the VideoInfer dataset collection process; Sec. 4 presents the implementation details and reports extensive experimental results and analysis on VideoInfer as well as 11 existing benchmarks. 2 Related Work Multimodal Large Language Models (MLLMs). Recent research in multimodal models has focused on visual instruction tuning, where pre-trained LLMs are fine-tuned for image and video understanding tasks. Models such as LLaVA [ 36 ] , MiniGPT-4 [ 76 ] , mPLUG-Owl [ 64 ] , Otter [ 27 ] , and InstructBLIP [ 36 ] have demonstrated significant progress by aligning vision-language representations through instruction tuning. Beyond text generation, recent works such as [ 24 , 52 , 25 , 37 ] have further extended the capabilities of MLLMs to retrieval and generation tasks, showcasing the growing versatility of MLLMs. Video Large Language Models (VideoLLMs). Extending image-level MLLMs to the video domain presents unique challenges, since videos comprise dynamic sequences that encode temporal dependencies, requiring models to process both spatial and temporal information. Models such as Video-LLaVA [ 33 ] , Video-ChatGPT [ 61 ] , and InternVideo [ 57 , 58 ] have shown strong performance in tasks like video captioning, temporal reasoning, and commonsense understanding, enabling text-based interaction with video content. Despite these advancements, existing VideoLLMs largely focus on holistic scene understanding, generating high-level textual descriptions of videos, while failing to capture fine-grained object interactions or nuanced temporal dependencies [ 29 , 74 , 28 ] . This limitation restricts their ability for object-centric reasoning, which is critical for detailed video analysis and practical real-world applications. Figure 2 : The proposed RGA3 architecture overview. (a) The S patial- T emporal O verlay M odule (STOM) is introduced to process arbitrary visual prompts ( e.g. , scribble , ellipse , arrow , etc.) at any timestamp and propagate to all frames, allowing for interactive object-centric reasoning and continual visual attention. (b) A visual encoder is employed to extract video representations of overlaid frames processed through STOM. (c) A Large Language Model (LLM) takes the concatenated sequence of visual and text tokens as input and generates responses. (d) To facilitate reasoning-based video object segmentation, a SAM2 decoder is incorporated for generating segmentation masks when prompted with a [SEG] token, extending RGA3’s capabilities beyond text-only responses. Object-centric MLLMs. In the recent literature, the tasks of object-centric referring and grounding have been well-studied in the image domain, with models like GPT4ROI [ 73 ] ,

Read the complete text