Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval

Kevin Flanagan, Dima Damen, Michael Wray · 2025

Read Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval by Kevin Flanagan, Dima Damen, Michael Wray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-08544.txt.

Introduction

Kevin Flanagan, Dima Damen, and Michael Wray wrote this 2025 paper on a blind spot in video search: questions about moments that never happen. Standard moment retrieval assumes the described scene is in the video, so models happily return time stamps for absent events. The team define a tougher task that also grades rejection.

Their Negative-Aware task separates two kinds of traps: queries from the same domain that simply do not occur, and queries from far outside it. New benchmark splits for two popular data sets put both to work. Existing top systems, tested as-is, stumbled on saying no. The adapted model from the team, built on UniVTG, rejected about 98 percent of negatives while keeping its retrieval scores within a few points.

Catalog record

CREATORKevin Flanagan, Dima Damen, Michael Wray
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-08544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval Kevin Flanagan, Dima Damen, Michael Wray Abstract Video Moment Retrieval is a common task to evaluate the performance of visual-language models—it involves localising start and end times of moments in videos from query sentences. The current task formulation assumes that the queried moment is present in the video, resulting in false positive moment predictions when irrelevant query sentences are provided. In this paper we propose the task of Negative-Aware Video Moment Retrieval (NA-VMR), which considers both moment retrieval accuracy and negative query rejection accuracy. We make the distinction between In-Domain and Out-of-Domain negative queries and provide new evaluation benchmarks for two popular video moment retrieval datasets: QVHighlights and Charades-STA. We analyse the ability of current SOTA video moment retrieval approaches to adapt to Negative-Aware Video Moment Retrieval and propose UniVTG-NA, an adaptation of UniVTG designed to tackle NA-VMR. UniVTG-NA achieves high negative rejection accuracy (avg. 98.4 % 98.4\% ) scores while retaining moment retrieval scores to within 3.87 % 3.87\% Recall@1. Dataset splits and code are available at https://github.com/keflanagan/MomentofUntruth 1 Introduction With the ever-increasing amount of video data that is accessible to the public through streaming websites, the ability to quickly search through this data is attaining an increased importance. Not only is it necessary to search for relevant videos, it is also desirable to search through the videos themselves. The video moment retrieval task addresses this by searching for relevant moments within videos using text queries as input. Currently, video moment retrieval models focus only on producing accurate start and end times for text queries, under the assumption that the moment always exists within the video. Video moment retrieval datasets are composed of video-sentence pairs which all have a direct correspondence through labelled start and end times. However, this raises the question: How do models perform with irrelevant textual queries for a given video? Figure 1 : Video moment retrieval models are designed to predict start and end times in a video given a query sentence. Regardless of whether the text query is positive (exists in the video), in-domain negative (from the same domain but not present) or out-of-domain negative (from an entirely different scenario), current models such as UniVTG still produce a start-end time prediction. For example, suppose you have a video of a person eating dinner and then you ask the model to localise the sentence “the gazelle leaped over the shrubs” . Should a video moment retrieval model return a start and end time for this sentence even though it clearly doesn’t correspond with anything in the video? Current models will always provide a start and end time prediction regardless of the relevancy of the sentence, which can lead to hallucination (see Figure 1 ). Whilst there has been a great amount of progress in improving these models, they do not handle sentences which are irrelevant to the video, known as negative queries. We believe that if these models are to be robust and useful, they should be able to indicate when an input text query is irrelevant to the given video. We believe this task of negative rejection is a crucial aspect to ensure trustworthy and explainable AI models. In this paper, we demonstrate, quantitatively and qualitatively, that current methods are not robust to negative queries and are not designed to distinguish between positive and negative queries. We do this by testing a combination of both in-domain, i.e. queries from the same dataset not relevant to the query video, and out-of-domain negatives, i.e. queries from a different scenario. We highlight examples of In-Domain (ID) and Out-Of-Domain (OOD) negatives within Figure 1 with example failure cases. To combat this, we propose a method to reject negative queries while maintaining video moment retrieval performance. Our method uses an additional head which is explicitly trained to predict whether the sentence is relevant. We train the model with both ID and OOD negatives, showing both are necessary. Overall, our contributions are as follows: (i) To the best of our knowledge, we conduct the first analysis of the robustness of SOTA video moment retrieval models to negative queries, while making the important distinction between in-domain and out-of-domain negative queries. (ii) We outline a method for sampling in-domain and out-of-domain negative queries for use during training and evaluation and provide new evaluation benchmarks for two popular Video Moment Retrieval datasets: QVHighlights [ 21 ] and Charades-STA [ 12 ] . (iii) We propose a new framework which enables existing video moment retrieval methods to train for Negative-Aware Video Moment Retrieval. (iv) We showcase strong results on negative rejection while retaining high moment retrieval performance from a model which has been adapted under our framework. 2 Related Work Moment Retrieval. The task of video moment retrieval was first introduced in [ 12 , 1 ] , with the aim of expanding the action localisation task [ 34 , 38 ] to the open vocabulary setting. Moment retrieval methods require cross-domain interactions between video and text in order to determine the correspondences between them. Approaches have historically been divided largely between proposal-based [ 1 , 2 , 12 , 13 , 19 , 27 , 25 , 26 , 43 , 4 , 36 , 40 , 50 ] and proposal-free methods [ 7 , 8 , 6 , 9 , 15 , 24 , 30 , 33 , 37 , 45 , 48 , 51 , 49 ] . Recent works have been designed to jointly perform moment retrieval and highlight detection [ 21 , 14 ] , being trained with both moment start/end times and saliency scores which are the ground truths for the highlight detection task. Certain datasets such as QVHighlights [ 21 ] contain both human-annotated moments and saliency scores. Datasets which do not have human-annotated saliency scores instead use pseudo-saliency scores from the moment start/end times, with a non-zero score within ground truth moments. Moment Retrieval Methods. Moment-DETR [ 21 ] is one such method that jointly trains on highlight detection and has served as a base for many recent moment retrieval methods [ 29 , 32 , 31 , 23 , 18 , 20 , 44 , 39 , 28 , 42 ] . This approach takes inspiration from DEtection TRansformer (DETR) [ 3 ] methods in object detection, viewing moment retrieval as a direct set prediction problem, generating moment candidates and associated scores with which to rank them in an end-to-end manner. It concatenates text and video features as input and passes them through a transformer encoder-decoder. Trainable positional embeddings known as moment queries are inputted to the decoder to generate candidate moment predictions. It has separate prediction heads for the moment span; the score of each proposed moment; and the saliency scores. While Moment-DETR uses foreground/background labels during training to supervise the moment score predictions, these are not designed to be used during evaluation for determining positive/negative queries. Some approaches based on this framework have included the addition of extra prior information to initialise the queries [ 18 ] or the addition of an extra modality [ 29 ] . Others have focused on improving the video-text interactions in the model [ 32 ,

Read the complete text