FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos

Zhengqian Wu, Ruizhe Li, Zijun Xu, Zhongyuan Wang, Chunxia Xiao, Chao Liang · 2024

Read FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos by Zhengqian Wu, Ruizhe Li, Zijun Xu, Zhongyuan Wang, Chunxia Xiao, Chao Liang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17022.txt.

Introduction

Zhengqian Wu and co-authors release FriendsQA, a new 2024 dataset for deep understanding of story videos. Video question answering works well on simple factual clips but still struggles with stories. Stories hinge on characters, actions, and locations that evolve and interact over long spans of time, which shallow models tend to miss.

The dataset adds fine-grained topic categorization for story videos, organizing questions and answers around those core story topics. It is large-scale and aimed squarely at the deep video understanding task rather than simple fact lookup. A harder, better-labeled benchmark gives model builders a fair test of whether their systems truly follow a plot from start to finish.

Catalog record

CREATORZhengqian Wu, Ruizhe Li, Zijun Xu, Zhongyuan Wang, Chunxia Xiao, Chao Liang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos Zhengqian Wu, Ruizhe Li, Zijun Xu, Zhongyuan Wang, Chunxia Xiao, Chao Liang Abstract Video question answering (VideoQA) aims to answer natural language questions according to the given videos. Although existing models perform well in the factoid VideoQA task, they still face challenges in deep video understanding (DVU) task, which focuses on story videos. Compared to factoid videos, the most significant feature of story videos is storylines, which are composed of complex interactions and long-range evolvement of core story topics including characters, actions and locations. Understanding these topics requires models to possess DVU capability. However, existing DVU datasets rarely organize questions according to these story topics, making them difficult to comprehensively assess VideoQA models’ DVU capability of complex storylines. Additionally, the question quantity and video length of these dataset are limited by high labor costs of handcrafted dataset building method. In this paper, we devise a large language model based multi-agent collaboration framework, StoryMind, to automatically generate a new large-scale DVU dataset. The dataset, FriendsQA, derived from the renowned sitcom Friends with an average episode length of 1,358 seconds, contains 44.6K questions evenly distributed across 14 fine-grained topics. Finally, We conduct comprehensive experiments on 10 state-of-the-art VideoQA models using the FriendsQA dataset. 1 School of Computer Science, Wuhan University, China 2 School of Cyber Science and Engineering, Wuhan University, China 3 National Engineering Research Center for Multimedia Software, Wuhan University, China 4 Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China {2023202110068, 2020300004016, 2019300003083, cxxiao, cliang}@whu.edu.cn, [email protected] Code and Datasets — https://github.com/nercms-mmap/FriendsQA Introduction Video question answering (VideoQA) aims to answer natural language questions based on given videos, supporting advanced applications like video grounding ( Ren et al. 2024 ) and video chatbots ( Jin et al. 2024 ) . Early researches on factoid VideoQA focus on questions about actions or objects in videos ( Yu et al. 2019 ) . With the popularity of video language model (VLM) and multimodal large language model (MLLM) technologies, significant progress has been made in recent years ( Yu et al. 2023 ; Zhong et al. 2022 ) . Figure 1: Comparisons of factoid VideoQA and DVU. Figure 2: The distribution of 5 datasets questions across the 14 fine-grained topics, based on 7 topics (character (C), action (A), location (L) and their combinations) and 2 cognitive attributions, i.e., perception (P) and inference (I). Dataset Venue Fine-grained topic distribution Question Scale Difficulty Measure Cross Episodes # Fin. Top. Gin. Ent. # Que. Vid. Len. (s) # Que. \times Vid. Len. (Ks) MovieQA CVPR’16 6 0.819 2.713 14.9K 202.7 3,020.2 ✗ ✗ TVQA EMNLP’18 8 0.821 2.873 144.9K 76.2 11,041.4 ✗ ✗ TVQA+ ACL’20 5 0.789 2.660 29.4K 61.5 1,808.1 ✗ ✗ HLVU (DVU 22&23) ICMR’20 6 0.773 2.548 455 106 / 4,907 1,010.5 ✗ ✗ DramaQA AAAI’21 - - - 17.9K 3.6 / 91.8 429.9 ✓ ✗ DeepMaven EACL’23 - - - 1K 3,102 3,102.0 ✗ ✗ CinePile CVPRW’24 - - - 200K 160 32,000.0 ✗ ✗ MovieChat-1K CVPR’24 4 0.701 2.203 19.0K 564 10,716.0 ✗ ✗ FriendsQA (ours) 14 0.927 3.794 44.6K 1,358 / 5,390 98,874.8 ✓ ✓ Table 1: Comparisons of existing DVU datasets. Fine-grained topic distribution consider the number of fine-grained topics exceeding 5% of the dataset (# Fin. Top.) and the balance degree of topic distribution. The Gini index (Gin.) and entropy (Ent.) are employed to measure the distribution’s balance. Question scale is compared by analyzing the number of questions (# Que.), the average video length (Vid. Len.), and their product. The proposed FriendsQA dataset offers additional features, including difficulty measure and cross episodes questioning. The figures around the “/” represent two distinct video input lengths. However, VideoQA models’ performance significantly declines on deep video understanding (DVU) task. For instance, VideoChat2 achieves 61.70% accuracy on the factoid NExT-QA dataset but drops to 44.05% on our DVU FriendsQA dataset. As illustrated in Figure 1 , the key reasons include: First, regarding question attributes, in addition to perception questions that inquire about visual cues, DVU also includes inference questions that assess the understanding of storylines, offering more diverse questions. Second, for the same perception questions, factoid VideoQA involves short-range shot-level perception in factoid video depicting simple events without complex narrative interactions, whereas DVU involves long-range scene-level ( Liang et al. 2009 ; Xu, Wei, and Wu 2023 ) perception in story videos encompass intricate storylines. In addition, characters and locations in DVU have specific identities ( Sang et al. 2011 ) , i.e., Ross and Monica’s apartment. Due to these factors, the DVU task is more challenging. The core of DVU task lies in understanding storylines with long-range evolvement, composed of story topics ( Guo, Liang, and Wang 2023 ) , i.e. characters, actions, locations and their combinations. Although story topics are essential to analyzing storylines, the most of existing DVU datasets ( Lei et al. 2018 ; Tapaswi et al. 2016 ; Song et al. 2024 ) do not organize questions based on them. We use Gemini 1.5 Pro 1 1 1 https://aistudio.google.com/ to categorize questions from 5 classic DVU datasets (Please refer to Appendix A for more details.) into 14 fine-grained topics we refer to as in this paper, i.e., 7 story topics, character, action, location and their combinations attached to 2 question attributions, perception and inference ( Ammanabrolu et al. 2021 ) . The results are shown in Figure 2 . We find that these datasets tend to concentrate on a limited type of fine-grained topics. This situation make it difficult to provide a comprehensive evaluation of VideoQA models’ DVU capability. Furthermore, complex storylines involve long-range evolvement, necessitating both numerous questions to cover the entire storyline and sufficiently video length to convey the storyline clearly. However, as shown in Table 1 , existing DVU datasets either pair a large number of questions with short videos, such as TVQA ( Lei et al. 2018 ) , or feature longer videos with fewer questions, such as DeepMaven ( Fung et al. 2023 ) . This limitation prevents them from evaluating the model’s understanding of storylines in both breadth and depth simultaneously. To address the above two challenges, we propose StoryMind, a large language model (LLM) based multi-agent collaboration framework, comprising a generator to generate questions with 14 fine-grained topics and two reviewers to remove low-quality ones. We provide the generator with detailed topic explanations and manually constructed examples to guide question generation. To ensure balanced topic coverage, generator will iteratively generate questions until the number of questions for each fine-grained topic reaches the predetermined threshold. To ensure dataset quality, only the question and its answer unanimously deemed reasonable and accurate by both reviewers will be retained. Moreover, as illustrated in Table 1 , we perform difficulty measure for each question to accurately evaluate the DVU capability of various VideoQA models, and introduce cross-episode questions to augment the challenge of the dataset by covering more complex storyline. We apply StoryMind to the popular sitcom Friends , comprising 234 episodes with average video length of 1,358s across ten seasons. This results in a new large-scale DVU dataset, FriendsQA, with the most diverse and balanced fine-grained topics. As

Read the complete text