FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering

Ju-Young Oh, Ho-Joong Kim, Seong-Whan Lee · 2025

Read FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering by Ju-Young Oh, Ho-Joong Kim, Seong-Whan Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12816.txt.

Introduction

Ju-Young Oh, Ho-Joong Kim, and Seong-Whan Lee present a study from twenty twenty-five on video question answering with richer training facts. Past training pairs tie queries to events in a clip. Event facts alone leave out object types, layouts, and traits of the scene. Models then learn a shard of the scene and reason weakly. Their frame, FIQ, crafts basic query and answer pairs from clip descriptions.

Those pairs teach the core context of each clip. A module named VQ-CAlign binds query codes to visual traits so key facts are not lost. Tests on a traffic clip set hit state of the art scores. The team credits the basic pairs for gains in reach and reason.

Catalog record

CREATORJu-Young Oh, Ho-Joong Kim, Seong-Whan Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12816

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering Ju-Young Oh, Ho-Joong Kim, Seong-Whan Lee Abstract Video question answering (VQA) is a multimodal task that requires the interpretation of a video to answer a given question. Existing VQA methods primarily utilize question and answer (Q&A) pairs to learn the spatio-temporal characteristics of video content. However, these annotations are typically event-centric, which is not enough to capture the broader context of each video. The absence of essential details such as object types, spatial layouts, and descriptive attributes restricts the model to learning only a fragmented scene representation. This issue limits the model’s capacity for generalization and higher-level reasoning. In this paper, we propose a fundamental question generation with the integration of question embeddings for video question answering (FIQ), a novel approach designed to strengthen the reasoning ability of the model by enhancing the fundamental understanding of videos. FIQ generates Q&A pairs based on descriptions extracted from videos, enriching the training data with fundamental scene information. Generated Q&A pairs enable the model to understand the primary context, leading to enhanced generalizability and reasoning ability. Furthermore, we incorporate a VQ-CAlign module that assists task-specific question embeddings with visual features, ensuring that essential domain-specific details are preserved to increase the adaptability of downstream tasks. Experiments on SUTD-TrafficQA demonstrate that our FIQ achieves state-of-the-art performance compared to existing baseline methods. I INTRODUCTION Video question answering (VQA) is a multimodal task [ 1 ] that combines computer vision and natural language processing. It requires the model to answer given questions based on the understanding of dynamic events in a video. VQA has acquired significant attention for its importance of tasks and broad applications in various fields, including education, health care, and surveillance systems [ 2 ] . Despite the substantial advancements of existing works in recent years and the wide applicability of the task, the alignment of natural languages and visual features still remains as a challenge. Recent studies have demonstrated significant advancements in this area, with various works [ 3 , 4 , 5 ] achieving notable results by aligning both modalities. Figure 1: The existing dataset only focuses on event-centric information of video, but not on fundamental information of video such as shape, color, and direction of objects. The existing VQA methods employ CLIP-based encoders to leverage their image-text alignment capability from their pretrained knowledge of large-scale data. Even though video-based encoders [ 6 , 7 , 8 , 9 ] which are specialized for video data exist, visual-text alignment requires both pretrained visual and text encoders, and CLIP provides both of them. FrozenBiLM [ 10 ] introduces a lightweight module that connects the frozen image encoder from CLIP and the frozen bidirectional language model for effective multimodal reasoning through masked language modeling. ViLA [ 11 ] proposes QFormer-Distiller, a module to enhance the alignment of both modalities by teaching Q-Former from BLIP [ 12 ] . While CLIP offers strong cross-modal features, it is pretrained on static images and thus relies heavily on textual annotations to supply spatio-temporal context. However, the current VQA dataset usually provides event-centric textual annotations, frequently omitting the fundamental scene attributes such as object identity, shape, or color. Although event-centric annotations already provide semantic cues, they serve only partial scene representations, thereby limiting a model to acquire only a fragmentary understanding of each scene. Fig. 1 shows an example in which a VQA model exclusively trained on event-centric annotations easily focuses on partial scenes. The model focuses on partial scenes where the collision occurs while preceding and subsequent frames are largely ignored. This behavior occurs because the event-centric annotations alone provide insufficient information to answer the corresponding questions. Such reliance on event-centric scenes restricts the ability of the model to both generalization and higher-level reasoning, which requires a broader understanding of the video context [ 13 , 14 ] . As shown in Fig. 1 , the model recognizes that an accident occurs, but fails to identify which vehicles are involved or what events happened before or after the accident. This partial understanding of video from the model is not enough for the model to establish causal or temporal relations between frames, leading to weaker generalization and limited higher-level reasoning. To alleviate this issue, enriching the fundamental attributes such as shape, color, and the direction of objects is significant in enhancing the reasoning ability of the model. These attributes are crucial for enabling the model to track other attributes over time, thereby facilitating a broader understanding of the video content, rather than limiting the comprehension to specific event-related frames. We propose a fundamental question generation with the integration of question embeddings for video question answering (FIQ), which integrates the general Q&A pairs into the original dataset to enhance the fundamental video understanding. Additionally, we incorporate question embeddings as task-specific information through a VQ-CAlign module. Our approach integrates Q&A pairs that focus on the fundamental understanding of video such as object types and shapes in specific scenes to the original dataset, enabling the model to enhance the fundamental understanding of video for more advanced reasoning. Furthermore, we introduce the VQ-CAlign module to integrate the task-specific information using question embeddings. This module prevents the model from missing the necessary task-specific information to interpret the direction of the task after the integration of general information. Our contributions are summarized as follows: • We propose FIQ which generates Q&A pairs to provide a fundamental understanding of the visual information, enhancing the generalizability of the model. • We construct the VQ-CAlign module to provide question embeddings to reflect task-specific information. • Our method achieves the state-of-the-art result on SUTD-TrafficQA dataset compared to other baseline models. II RELATED WORKS II-A Video Question Answering VQA is the task of determining the semantic information of a video to generate an answer to a given query. There are two approaches designed to achieve a lower computational cost for the VQA task, adapter-based methods, and utilizing the text-based representations. Adapter-based methods [ 15 , 16 , 17 ] are proposed to reduce the computational cost of VQA tasks, thereby enabling large language models (LLM) to be adapted to downstream tasks without the need of finetuning the entire model. Tem-Adapter [ 18 ] presents a novel alignment method that utilizes auto-regression to integrate semantic and temporal information from the video domain to the image domain. While various works adapt textual information to enhance the understanding of spatial and temporal features from video, achieving a competitive understanding of video data solely through textual representations for VQA is not common. Vamos [ 19 ] is a text-based video understanding framework that achieves superior performance without relying on visual features by generating task-agnostic texts, emphasizing utilizing only textual data leads to enhanced performance. Similarly, ColPro [ 20 ] integrates three distinct task-specific prompts to prevent catastrophic forgetting during the training process. Despite the differences in objective, both methods demonstrate remarkable performance without relying on visual embeddings. Our approach aligns with these two previous methods by leveraging textual prompts to enhance the interpretability of video data, reflecting its spatial and temporal properties. Furthermore, we employ both a language model (LM) and a LLM to generate task-agnostic Q&A pair to

Read the complete text