ViTED : Video Temporal Evidence Distillation

Yujie Lu, Yale Song, William Wang, Lorenzo Torresani, Tushar Nagarajan · 2025

Read ViTED : Video Temporal Evidence Distillation by Yujie Lu, Yale Song, William Wang, Lorenzo Torresani, Tushar Nagarajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12855.txt.

Introduction

Yujie Lu, Yale Song, William Wang and colleagues teach video question answering systems to show their evidence in 2025. Hard questions about long videos need facts from several moments, but most models sample frames evenly and can miss the key seconds.

Their framework, ViTED, first builds chains of evidence automatically: it searches a video for the time spans and visual details that best support an answer. The model is then trained to produce such chains itself, so it both locates the evidence and reasons across it step by step. On long-video benchmarks, the evidence-trained model beat strong systems that answer without explicit evidence, pointing to a more checkable style of video understanding.

Catalog record

CREATORYujie Lu, Yale Song, William Wang, Lorenzo Torresani, Tushar Nagarajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12855

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ViTED : Video Temporal Evidence Distillation Yujie Lu, Yale Song, William Wang, Lorenzo Torresani, Tushar Nagarajan Abstract We investigate complex video question answering via chain-of-evidence reasoning — identifying sequences of temporal spans from multiple relevant parts of the video, together with visual evidence within them. Existing models struggle with multi-step reasoning as they uniformly sample a fixed number of frames, which can miss critical evidence distributed nonuniformly throughout the video. Moreover, they lack the ability to temporally localize such evidence in the broader context of the full video, which is required for answering complex questions. We propose a framework to enhance existing VideoQA datasets with evidence reasoning chains, automatically constructed by searching for optimal intervals of interest in the video with supporting evidence, that maximizes the likelihood of answering a given question. We train our model ( ViTED ) to generate these evidence chains directly, enabling it to both localize evidence windows as well as perform multi-step reasoning across them in long-form video content. We show the value of our evidence-distilled models on a suite of long video QA benchmarks where we outperform state-of-the-art approaches that lack evidence reasoning capabilities. 1 Introduction Video Question Answering (VideoQA) is a critical task towards general-purpose video understanding, for which large vision-language models (VLMs) [ 55 , 44 , 16 ] have recently demonstrated strong performance on a variety of benchmarks [ 48 , 49 , 47 , 30 , 17 , 32 ] . Despite their popularity, these models typically excel at questions for which information is readily available throughout the video (e.g., high-level actions, colors/counts of objects, etc.), but struggle on long-form videos [ 30 , 26 , 9 , 32 ] and on questions that require gathering and aggregating evidence from the full video [ 49 ] . For instance, answering a question like, “Why does the baby put their hand in their mouth at the beginning of the video?” requires more than just locating the action — it involves an intricate approach that gathers contextual clues both before and after the action to infer the underlying reason. For instance, examining what occurred just before or after the baby put their hand in their mouth and observing how they performed the action (e.g., slowly or quickly) can offer valuable insights into their intent. Figure 1 : Main idea. We produce multiple, temporally localized pieces of evidence (the “evidence chain”) to support complex reasoning in VideoQA. Our ViTED model is trained to generate this evidence chain to enable temporally-grounded chain-of-thought reasoning in video. Current models face limitations in effectively processing the temporal relationships among visual frames, as well as in bridging the gap between a question and the evidence needed to answer it. On the one hand, current models do not temporally ground their responses, and rather sample a fixed number of video frames at regular intervals, potentially missing important moments from the video (e.g., failing to identify the short window when the girl waves at dog in Fig. 1 , left). Consequently, these models struggle to answer questions requiring complex temporal understanding , e.g., “what did the girl do right before interacting with the dog?” in a video where she approaches, picks up a ball, then engages with the dog; and multi-hop reasoning , e.g., “why did the baby put his hand in his mouth?” in a video where the mother feeds him with a spoon, followed by his uncomfortable expression and an attempt to remove the food. On the other hand, temporal grounding models can indeed localize text queries to specific intervals in a video [ 33 , 13 , 22 ] , but they require the text in advance — i.e., they can localize the evidence needed to answer the question if already provided with the evidence text, but they cannot identify and ground this evidence based on the question alone. For example, it might identify the moment when the dog approaches the girl (Fig. 1 ), but it may not establish how this action relates to the question (“why did she wave?”). To address these limitations, we propose to consolidate evidence generation, temporal grounding and question answering into one model that we call Vi deo T emporal E vidence D istillation for Video Understanding ( ViTED ). Our model is a temporally-aware VLM that is trained to generate both answers to given questions, as well as temporally grounded evidence chains to support the answer — time intervals of the video together with textual clues within the temporal span (see Fig. 1 , bottom). Since this kind of evidence data is not readily available, we present a framework to automatically synthesize high-quality evidence chains on top of existing VideoQA datasets. Specifically, we generate a pool of evidence containing textual evidence relevant to the question, extracted from video segments of various lengths and at multiple levels of detail (e.g., short clips with single actions, to high level activities across the full video). We then present a search-and-refinement algorithm over this evidence pool to find optimal sequences of evidence that are the most predictive of the correct answer. Finally, we augment the original VideoQA training data with our generated evidence chains, and train our model to predict both the answer and the evidence chain that supports it, thereby distilling the ability to localize and generate temporal evidence into the VLM. We demonstrate the effectiveness of ViTED on 6 representative VideoQA benchmarks, CinePile [ 32 ] , PerceptionTest [ 30 ] , NExT-QA [ 48 ] , STAR [ 47 ] , MVBench [ 17 ] . We show that ViTED is on par with or outperforms state-of-the-art models trained with 10 \times more video instruction data. Additionally, we show that ViTED provides the most faithful and interpretable temporal evidence chain for the answer compared with existing VideoLLMs through human studies. Finally, ViTED achieves SOTA zero-shot performance on NExT-GQA [ 49 ] — a benchmark squarely focused on temporally grounded VideoQA — surpassing GPT-4 driven agent approaches, highlighting our generalizable evidence grounding capability. In summary, we propose an approach that integrates evidence generation, grounding and reasoning towards complex video understanding. Our main contributions are: • We propose a novel framework to generate and search for evidence chain-of-thought data from existing VideoQA datasets. • We propose an evidence distillation approach to train a temporally-aware video model on our high-quality evidence data. • Our ViTED sets new SOTA results among models of the same size on four VideoQA benchmarks, and surpasses GPT-4 driven agent on NExT-GQA, while providing high-quality explanations for its predictions. Figure 2 : Overview of ViTED evidence generation framework. There are three main stages: (1) We first generate the evidence pool — detailed captions for segments at multiple granularities — and rank them based on relevance to the question (left, Sec. 3.1 ) (2) Next, we search over the evidence pool to derive evidence chains that are most predictive of the target answer, and summarize it into a coherent and logical chain-of-thought (top-right, Sec. 3.2 ) (3) Finally, if the evidence chain successfully leads to the correct answer, we add it to our dataset for training our model (bottom-right, Section 3.3 ) 2 Related Work 2.1 Video Understanding with LLMs Recent

Read the complete text