Aligning Moments in Time using Video Queries
Yogesh Kumar, Uday Agarwal, Manish Gupta, Anand Mishra · 2025
Read Aligning Moments in Time using Video Queries by Yogesh Kumar, Uday Agarwal, Manish Gupta, Anand Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15439.txt.
Introduction
Yogesh Kumar, Uday Agarwal, Manish Gupta, and Anand Mishra present MATR, a model that finds a moment in one video by using another video as the query. Their twenty twenty-five paper tackles video-to-video moment retrieval. A user gives a clip of an event, and the system must find the matching stretch in a target video, with no labels to guide it.
MATR is a transformer that aligns the two videos in two stages, at the level of frames and of meaning. It then marks which frames are foreground and where the moment starts and ends. A self-taught pre-training step, finding random clips in videos, warms it up. Tests on two data sets beat past methods by clear margins.
Catalog record
| CREATOR | Yogesh Kumar, Uday Agarwal, Manish Gupta, Anand Mishra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15439
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning Moments in Time using Video Queries Yogesh Kumar, Uday Agarwal, Manish Gupta, Anand Mishra Abstract Video-to-video moment retrieval ( Vid2VidMR ) is the task of localizing unseen events or moments in a target video using a query video. This task poses several challenges, such as the need for semantic frame-level alignment and modeling complex dependencies between query and target videos. To tackle this challenging problem, we introduce MATR ( M oment A lignment TR ansformer), a transformer-based model designed to capture semantic context as well as the temporal details necessary for precise moment localization. MATR conditions target video representations on query video features using dual-stage sequence alignment that encodes the required correlations and dependencies. These representations are then used to guide foreground/background classification and boundary prediction heads, enabling the model to accurately identify moments in the target video that semantically match with the query video. Additionally, to provide a strong task-specific initialization for MATR, we propose a self-supervised pre-training technique that involves training the model to localize random clips within videos. Extensive experiments demonstrate that MATR achieves notable performance improvements of 13.1% in R@1 and 8.1% in mIoU on an absolute scale compared to state-of-the-art methods on the popular ActivityNet-VRL dataset. Additionally, on our newly proposed dataset, SportsMoments, MATR shows a 14.7% gain in R@1 and a 14.4% gain in mIoU on an absolute scale over strong baselines. We make the dataset and code public at: https://github.com/vl2g/MATR . * * footnotetext: Equal Contribution 1 Introduction Figure 1 : ( Top ): Activity where text falls short to explain complex action or events, such as bicycle kick , necessitating the need for a more intuitive query modality for localizing semantically matching moments. ( Bottom ): Shows the Vid2Vid moment retrieval setting, which is the goal of this work. Video moment retrieval is the task of temporally localizing the start and end times of a moment 1 1 1 A moment is a continuous segment of frames within a target video that best represents the actions, events, or interactions described by the query. in a target video described by a given query. Although text-based video moment retrieval has been extensively explored [ 25 , 33 , 32 , 28 , 57 ] , it often poses challenges for users attempting to describe specific moments verbally. For example, consider a spectacular bicycle kick in soccer (Fig. 1 top). Although an expert can search by naming or describing it in detail, a beginner in soccer may struggle to describe this impressive move accurately. They might say something like “The player kicked the ball while in the air”, which lacks the nuance needed to convey the athleticism and artistry involved. Further, such a description may lead to poor results for someone searching for similar moments. In contrast, if a soccer trainee shows a short video clip of a bicycle kick , the input is rich and crisp, making it easier for a retriever to locate similar moments in a target soccer match video. This approach aligns with how users naturally would prefer to search content within the same (video) modality, making it a more effective method for retrieving specific moments. Fig. 1 (bottom) shows a video bicycle kick query in soccer that is localized in a target video. Therefore, in this work, we study video-to-video moment retrieval ( Vid2VidMR ) where the aim is to temporally localize a moment in the target video with a high semantic match between the moment and the query video. Vid2VidMR , a task formally introduced by Feng et al. [ 14 ] , has many potential application areas, such as sports video analytics, educational content creation and e-learning, and surveillance systems. This task is challenging and requires semantic frame-level alignment and modeling complex dependencies between query and target videos. The need for a semantic understanding of video content and variety in video length, context, and action speed calls for adaptive models capable of generalizing across diverse scenarios. Addressing these challenges demands advanced temporal video modeling techniques. To address the aforementioned challenges, we propose MATR ( M oment A lignment TR ansformer) – a method that uses explicit ‘dual-stage sequence alignment’ to capture the required correlation and temporal details essential for accurate moment localization. By conditioning target video representations on query features, MATR produces query-aligned representations that encode the necessary correlations and temporal dependencies between the two videos. We use differentiable dynamic time warping loss [ 9 ] for aligning the query and target videos, and represent the target video by conditioning it on the query video to focus on correlated temporal features. These representations guide a classification head to discriminate relevant moments from the irrelevant background and a boundary prediction head to mark the start and end of the identified moment in the target video. Further, to enhance our model’s generalization using unlabeled videos, we introduce a self-supervised pre-training strategy which involves training MATR to localize randomly sampled clips within the same video, enabling it to learn the moment localization skill in a self-supervised manner. We evaluate MATR on public ActivityNet-VRL [ 14 ] benchmark and on our newly introduced dataset on sports domain, viz. SportsMoments, covering two of the most popular sports, namely soccer and cricket. Our approach achieves significant performance gains, with an improvement of 13.1% and 8.1% in R@1 and mIoU, respectively, on ActivityNet-VRL outperforming the state-of-the-art methods. Furthermore, MATR outperforms the implemented strong baselines on our proposed dataset with 14.7% gains in R@1 and 14.4% gains in mIoU, all on an absolute scale. Our contributions are as follows: (i) We introduce MATR, which uses explicit dual-stage sequence alignment within a transformer framework between target and query video to capture temporal correlations and dependencies for accurate moment localization. (ii) We propose a self-supervised pre-training objective that enhances model initialization by understanding rich video structure without requiring any labeled data. (iii) We conduct extensive experiments and ablations to study the efficacy of our framework against competitive baselines and state-of-the-art methods. Our findings offer valuable insights into our design choices, and our approach advances the state-of-the-art on Vid2VidMR . 2 Related Work Video Moment Retrieval (VMR): VMR has recently gained significant interest in the research community [ 18 , 24 , 15 , 25 , 33 , 28 , 57 , 32 , 53 , 55 , 29 , 34 , 1 ] . Unlike video action understanding tasks such as action classification [ 44 , 27 , 56 , 47 , 6 , 54 , 13 , 60 ] or temporal action localization [ 59 , 64 , 43 ] , VMR focuses on identifying segments that semantically align with a broader range of queries, which may describe complex and context-specific moments. Based on the query modality, existing VMR methods can be broadly grouped into (i) textual query-based approaches like Moment-DETR [ 25 ] , QD-DETR [ 33 ] , Uni-VTG [ 28 ] , and (ii) video query-based approaches like GDP [ 7 ] , FFI+SRM [ 19 ] , SRL [ 51 ] . Our work falls under video query-based VMR ( Vid2VidMR ). However, in addition to developing a new approach tailored for video query, we also adapt several text-query-based methods to make