Outside Knowledge Conversational Video (OKCV) Dataset - Dialoguing over Videos

Benjamin Reichman, Constantin Patsch, Jack Truxal, Atishay Jain, Larry Heck · 2025

Read Outside Knowledge Conversational Video (OKCV) Dataset - Dialoguing over Videos by Benjamin Reichman, Constantin Patsch, Jack Truxal, Atishay Jain, Larry Heck (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09953.txt.

Introduction

Benjamin Reichman and co-authors release a data set for dialogue about videos that needs outside knowledge, in work from twenty twenty-five. In such dialogue, a model must spot the right video moments and add facts not shown on screen. Conversation history also shapes each reply.

The set holds more than two thousand videos and near six thousand human written dialogues. Those dialogues span more than forty thousand turns, each tied to video segments. Questions often ask for facts beyond the footage itself. Baseline models were run to show the task's hard spots. The data set is public, with code links given. The authors frame it as a test bed for grounded, knowledge rich chat.

Catalog record

CREATORBenjamin Reichman, Constantin Patsch, Jack Truxal, Atishay Jain, Larry Heck
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Outside Knowledge Conversational Video (OKCV) Dataset - Dialoguing over Videos Benjamin Reichman, Constantin Patsch, Jack Truxal, Atishay Jain, Larry Heck Abstract In outside knowledge visual question answering (OK-VQA), the model must identify relevant visual information within an image and incorporate external knowledge to accurately respond to a question. Extending this task to a visually grounded dialogue setting based on videos, a conversational model must both recognize pertinent visual details over time and answer questions where the required information is not necessarily present in the visual information. Moreover, the context of the overall conversation must be considered for the subsequent dialogue. To explore this task, we introduce a dataset comprised of 2,017 2,017 videos with 5,986 5,986 human-annotated dialogues consisting of 40,954 40,954 interleaved dialogue turns. While the dialogue context is visually grounded in specific video segments, the questions further require external knowledge that is not visually present. Thus, the model not only has to identify relevant video parts but also leverage external knowledge to converse within the dialogue. We further provide several baselines evaluated on our dataset and show future challenges associated with this task. The dataset is made publicly available here: https://github.com/c-patsch/OKCV 1 Introduction Much of the human experience is mediated by continuous visual perception. Consequently, understanding videos—analogous to this continuous perception—is a crucial task for artificial intelligence. Intelligent comprehension of the world includes the ability to filter relevant aspects of visual information, engage in dialogue about it, and link it to knowledge that extends beyond immediate perceptual content. These are skills that are beyond current large vision foundation models. In Visual Question Answering (VQA), models are tasked with answering questions about the content of an image. Outside knowledge-based VQA not only relates the question to visual cues but rather also extends it to knowledge that is not present within the image Marino et al. 2019 ; Reichman et al. 2023 . Thus, current approaches utilize external knowledge bases like Wikidata or Wikipedia Wang et al. 2015 ; Gao et al. 2022 ; Gui et al. 2021 ; Lin et al. 2022 to answer questions that require information that goes beyond the visual content of the image. Due to the promising reasoning capabilities of LLMs, recent approaches leverage models like GPT-3 and GPT-4 Brown et al. 2020 ; OpenAI 2023 to further improve answers with respect to the knowledge inherently present in the pretraining datasets. These approaches often Shao et al. 2023 ; Yang et al. 2022 use image captioning models to transfer the visual information present in the image to the text domain. A further extension of the VQA is the Video QA task. As humans generally perceive temporally evolving visual information in the form of videos, Video QA Fu et al. 2021 ; Zellers et al. 2021 provides an analog to this by visually grounding questions in terms of video data. VQA dialogue systems depending on such visual grounding, are then capable of having a conversation about the image or video content Nie et al. 2019 ; Engin et al. 2021 . Figure 1: An example conversation from the OKCV dataset. Our dataset utilizes videos for visual grounding, requiring models to process information both spatially and temporally. This necessitates that models discern which parts of a video are pertinent to answering a question. With each video, we provide corresponding dialogues requiring the model to process the visual information present in the video, retrieve and reason over external knowledge, and be able to interact with users within a dialogue setting. Thus, compared to previous VQA and outside knowledge VQA datasets, our dataset tests a model’s capability to understand visual scenes and reason about it with additional information that is not present in the visual information. Providing dialogues instead of one-off QA like previous outside knowledge VQA datasets enables testing a model’s ability for extended reasoning capabilities over a video and the ability to mimic natural conversational scenarios. Thus, a model has to consider visual cues, external knowledge not present in the video, and the overall context of dialogue. To address these challenges, we introduce a new dataset, OKCV, which consists of 2,017 videos for visual grounding, accompanied by 5,986 human annotated dialogues consisting of 40,954 interleaved dialogue turns. These conversations evolve around the visual context but require external knowledge to effectively complete a dialogue turn. Consequently, our dataset challenges the capabilities of video-language models in terms of their spatiotemporal, factual, and commonsense reasoning as well as their dialogue management skills. Thus, resulting dialogue systems should be able to have natural comprehensive conversations about videos extending beyond their internal knowledge bases. Dataset VQA-Type # Dial. Vis. Text Temp. Know. Conv. Human Annot. OK-VQA Marino et al. 2019 ImageKB - ✓ ✓ ✗ ✓ ✗ ✓ Visual Dialogue Das et al. 2017 Image-Conv 140k ✓ ✓ ✗ ✗ ✓ ✓ TVQA Lei et al. 2018 Video - ✓ ✓ ✓ ✗ ✗ ✓ TVQA+ Lei et al. 2019 Video - ✓ ✓ ✓ ✗ ✗ ✓ AVSD Alamri et al. 2019 Video-Conv 11.8k ✓ ✓ ✓ ✗ ✓ ✓ KnowIT-VQA Garcia et al. 2020 VideoKB - ✓ ✓ ✓ ✓ ✗ ✓ VideoChat Li et al. 2023 Video-Conv 4k ✓ ✓ ✓ ✗ ✓ ✗ OKCV (ours) VideoKB-Conv 5.7k ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparision of the dataset characteristics number of dialogues (# Dial.), visual grounding (Vis.), text (Text), temporal information in the form of videos (Temp.), required external knowledge (Know.), conversational structure (Conv.) and human-annotated data (Human Annot.). VQA type indicates the type of visual grounding, required external knowledge and dialogue data structure. 2 Related Work 2.1 Visual Question Answering (VQA) In VQA, models must align text and vision representations to reason about the visual content of an image. VQA datasets generally contain question-answer pairs that are assigned to specific images Antol et al. 2015 ; Gao et al. 2015 ; Goyal et al. 2017 . In this task, the model has to reason about the visual content of the image and be able to answer related questions. Knowledge-based VQA expands on this by requiring models to consider facts beyond the visual content of an image Wang et al. 2017 ; Shah et al. 2019 ; Marino et al. 2019 ; Reichman et al. 2023 . VideoQA datasets take this further by grounding the model in video content rather than static images Tapaswi et al. 2016 ; Lei et al. 2019 ; Garcia et al. 2020 . Thus, tasking the model with extracting not only spatially relevant information from frames but also temporally important information throughout the video duration. There are several existing datasets in the VQA and Video QA area that require external knowledge to answer the questions contained in them. In the image modality, datasets such as OK-VQAv2, A-OKVQA, and KVQA feature a wide array of questions that necessitate this additional knowledge Reichman et al. 2023 ; Marino et al. 2019 ; Schwenk et al. 2022 ; Sanket Shah and Talukdar 2019 . In the video modality, there is the KnowIT VQA and KnowIT-X VQA datasets Garcia et al. 2020 ; kno introduce human-generated question-answer pairs, that integrate visual, textual, and temporal reasoning with knowledge-based questions, where the required knowledge-base is restricted to the

Read the complete text