Towards Fine-Grained Video Question Answering

Wei Dai, Alan Luo, Zane Durante, Debadutta Dash, Arnold Milstein, Kevin Schulman · 2025

Read Towards Fine-Grained Video Question Answering by Wei Dai, Alan Luo, Zane Durante, Debadutta Dash, Arnold Milstein, Kevin Schulman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06820.txt.

Introduction

Wei Dai and co-authors present a 2025 data set and model for fine-grained video question answering. Video tests often miss detail in time and space. Models then pass while still weak at pinning when things happen, where actors stand, and how objects relate. The new data set was built to close those gaps.

It holds questions tied to scene graphs and to marked time spans, with a stress on actors and objects. The paired model guesses the scene graph, pulls the right frames, and answers with a large language model. Tests on the new set and on public sets put the model ahead of prior work. The authors frame it as a new bar for video tests.

Catalog record

CREATORWei Dai, Alan Luo, Zane Durante, Debadutta Dash, Arnold Milstein, Kevin Schulman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06820

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Fine-Grained Video Question Answering Wei Dai, Alan Luo, Zane Durante, Debadutta Dash, Arnold Milstein, Kevin Schulman Abstract In the rapidly evolving domain of video understanding, Video Question Answering (VideoQA) remains a focal point. However, existing datasets exhibit gaps in temporal and spatial granularity, which consequently limits the capabilities of existing VideoQA methods. This paper introduces the Multi-Object Multi-Actor Question Answering (MOMA-QA) dataset, which is designed to address these shortcomings by emphasizing temporal localization, spatial relationship reasoning, and entity-centric queries. With ground truth scene graphs and temporal interval annotations, MOMA-QA is ideal for developing models for fine-grained video understanding. Furthermore, we present a novel video-language model, SGVLM, which incorporates a scene graph predictor, an efficient frame retriever, and a pre-trained large language model for temporal localization and fine-grained relationship understanding. Evaluations on MOMA-QA and other public datasets demonstrate the superior performance of our model, setting new benchmarks for VideoQA. 1 Introduction In the current era of abundant digital video content, video understanding has become a key focus in computer vision research, with significant implications in various fields such as entertainment Tapaswi et al. 2016b ; Huang et al. 2020b ; Rao et al. 2020 , healthcare Singh et al. 2020 ; Rajavel et al. 2022 ; Khan et al. 2020 ; Shorfuzzaman et al. 2021 , and surveillance Sreenu and Durai 2019 ; Ullah et al. 2021 . Among the numerous aspects of video comprehension, Video Question Answering (VideoQA) has garnered a significant amount of attention, since it requires models to answer questions regarding a specific video segment, which necessitates a thorough grasp of the scene, relationships, and temporal changes depicted in the video Zhong et al. 2022 ; Jang et al. 2017 ; Xiao et al. 2021 . Grounding in video understanding—specifically, temporal and spatial grounding—plays a pivotal role in bridging the gap between low-level video features and high-level semantic interpretations Wang et al. 2023b . Temporal grounding ensures that events or actions within videos are associated with specific time intervals Soldan et al. 2021 , while spatial grounding offers localized regions within video frames that correspond to certain entities or objects Tan et al. 2021 . A dataset incorporating both temporal and spatial dimensions can offer rich contextual cues and pave the way for more detailed and accurate video comprehension tasks. Figure 1 : Visualizations of Sample Questions from MOMA-QA. We illustrate the three distinct types of questions in our dataset, each representing a different category for video question answering. All questions in our dataset are generated from a human-annotated spatio-temporal scene graph (shown on the right). The node of interest for the relationship and motion questions is colored red in the scene graph and outlined in the video. While many datasets exist in the realm of video understanding, a deeper dive into their contents uncovers notable gaps in their representations of spatial relationships. For instance, in the ActivityNet-QA dataset Yu et al. 2019 , only 10% of its questions revolve around questions with spatial dimensions. This restricts the range and depth of inquiries a model can proficiently address. Another concern is the absence of fine-grained spatial relationship annotations. While TVQA+ Lei et al. 2020a offers object-level details via bounding boxes, it fails to provide relationships between these objects. STAR Wu et al. 2021 provides relationship annotations for its videos, but the automated nature of these annotations significantly restricts their precision and applicability. Category Question Format Answer Format Example Question Example Answer Relationship When [C_{i}] , what is [V_{s}] [E_{i}] ? [V_{t}] When the players are passing the basketball for the 3rd time, Who is the outlined person looking at? Basketball Player Motion When [C_{i}] , is [V_{t}] standing, walking or running? [Att( V_{t} )] When the players are passing the basketball for the 3rd time, is the outlined person standing, walking, or running? Running Description When [C_{i}] , how many [Identity] are in the scene? [Id. Count] When the players are passing the basketball for the 3rd time, how many basketball players are in the scene? 7 Table 1: General Structure of Generated Questions. C_{i} denotes the description of a particular sub-activity. Additionally, V_{s} and V_{t} denotes the name of a source and a target node from the sub-activity. E_{i} represents the description of a relationship connecting V_{s} and V_{t} . Temporal grounding, too, has its share of challenges in existing VideoQA datasets. Foundational datasets such as MSRVTT-QA Xu et al. 2017 often neglect the importance of temporal localization. Approximately 33.4% of its questions can be distilled to a generic format: “What is [someone] doing” Such questions steer models predominantly toward video classification objectives, bypassing the need to anchor responses to specific moments or sequences within a video. Recent datasets like TVQA+ and TGIF-QA Lei et al. 2020a ; Jang et al. 2017 have shifted focus towards temporal reasoning within videos. However, they lack ground truth annotations for temporal localizations, thus there is no definitive means to ascertain whether a model has accurately localized the correct frames. Lastly, understanding and identifying the actions of individuals in crowded settings is challenging, and few datasets tackle this Luo et al. 2022 . Addressing entity-specific queries in group situations is crucial for advanced video comprehension. Given the gaps observed in current datasets, we introduce the Multi-Object Multi-Actor Question Answering (MOMA-QA) dataset. Stemming from the foundation of the Multi-Object Multi-Actor (MOMA) Luo et al. 2022 dataset, MOMA-QA brings unique attributes designed to challenge and improve the current generation of video question-answering models. Firstly, as shown in Figure 1 , every question within MOMA-QA requires temporal localization and is accompanied by ground truth temporal interval annotations to provide a means to validate models’ temporal localization abilities. Secondly, 71.6% of the questions in the dataset require spatial relationship understanding, which MOMA-QA intensively assesses models on interpreting spatial connections among video entities. Each frame features ground truth scene graph annotations, laying a foundation for the evolution of more sophisticated spatially-aware models. Lastly, understanding the challenge of discerning specific individuals in crowded settings, we visually demarcate specific actors via frame-level bounding boxes on a subset of questions, thereby testing the model’s proficiency in entity-specific reasoning. As current datasets lack fine-grained annotations, existing VideoQA models struggle with nuanced understanding due to their linear approach of directly processing video frames and questions to produce answers. This limits their interpretability, a gap that becomes more apparent with the rise of visual language models in this domain. To address this issue, we introduce SGVLM , a video-language model with enhanced retrieval and relationship understanding abilities. Our model encapsulates three main features. First, the vision encoder has been restructured and augmented with a Motif-based scene graph generator Zellers et al. 2018 . The scene graph generator provides robust grounding for the spatial relationships depicted in videos and also provides an interpretable understanding of the model’s decision-making pathway and elucidating how it arrives at its final predictions. Second, we devise an efficient frame retriever that identifies frames relevant to posed questions by leveraging both video and scene graph features, providing greater accuracy, especially for tasks on discerning relationships. Lastly, SGVLM harnesses the power of pre-trained large language models, empowering it to tackle intricate reasoning tasks. In summary, our work has the

Read the complete text