Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection

Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim · 2025

Read Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection by Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02504.txt.

Introduction

Sung Jin Um and three co-authors improve how software finds moments in videos in a 2025 paper. Given a text query, the system must locate the matching segment and flag the highlights. Many systems read the query word by word without weighing which words matter most in the video's own context.

Their module first clusters the whole video into a compact summary of its context. It then learns keyword weights with a contrastive method that ties words to the visual features they fit. The result sharpens the match between language and footage. On three public benchmarks, for search and highlight tasks, the method clearly beat earlier approaches. The authors release their code.

Catalog record

CREATORSung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim Abstract The goal of video moment retrieval and highlight detection is to identify specific segments and highlights based on a given text query. With the rapid growth of video content and the overlap between these tasks, recent works have addressed both simultaneously. However, they still struggle to fully capture the overall video context, making it challenging to determine which words are most relevant. In this paper, we present a novel Video Context-aware Keyword Attention module that overcomes this limitation by capturing keyword variation within the context of the entire video. To achieve this, we introduce a video context clustering module that provides concise representations of the overall video context, thereby enhancing the understanding of keyword dynamics. Furthermore, we propose a keyword weight detection module with keyword-aware contrastive learning that incorporates keyword information to enhance fine-grained alignment between visual and textual features. Extensive experiments on the QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that our proposed method significantly improves performance in moment retrieval and highlight detection tasks compared to existing approaches. Our code is available at: https://github.com/VisualAIKHU/Keyword-DETR . 1 Kyung Hee University, Yong-in, South Korea 2 Sungkyunkwan University, Seoul, South Korea {sungzin1, rlaehdwls310, ju.kim}@khu.ac.kr, [email protected] Introduction With the exponential growth of video content, precise video moment retrieval and highlight detection have become crucial ( Snoek, Worring et al. 2009 ; Apostolidis et al. 2021 ) . Video moment retrieval enables users to find specific segments within videos based on natural language queries ( Anne Hendricks et al. 2017 ) , while highlight detection helps extract the most engaging parts from long-form videos ( Gygli et al. 2014 ) . These technologies enhance user experience and productivity across various applications, including video searching, video editing, social media, and e-learning, by enabling quick and accurate access to relevant content. Extensive research has been conducted on moment retrieval ( Gao et al. 2017 ; Hendricks et al. 2018 ; Xiao et al. 2021 ; Sun et al. 2022 ) and highlight detection ( Sun, Farhadi, and Seitz 2014 ; Xu et al. 2021 ; Wei et al. 2022 ; Badamdorj et al. 2022 ) as separate tasks. However, with the introduction of Moment-DETR and the QVHighlights dataset ( Lei, Berg, and Bansal 2021 ) , which allows for the simultaneous execution of these tasks, new studies ( Liu et al. 2022 ; Moon et al. 2023 ; Xiao et al. 2024 ; Sun et al. 2024 ) have emerged that aim to address moment retrieval and highlight detection based on text queries concurrently. Following the Moment-DETR ( Lei, Berg, and Bansal 2021 ) , UMT ( Liu et al. 2022 ) utilizes audio-visual multi-modal learning. TR-DETR ( Sun et al. 2024 ) leverages the reciprocal relationship between two tasks, refining visual features through textual guidance to enhance both tasks simultaneously. UVCOM ( Xiao et al. 2024 ) introduces integration module for progressive intra- and intermodality interaction across multi-granularity. Figure 1: Text keywords can vary by video context. The less frequently a word appears in the video clip, the more important it becomes within the text query. In Video #1, ‘ dog ’ is important, while in Video #2, ‘ garden ’ is important. Despite these advancements, existing methods often fail to capture the dynamic importance of keywords within the context of the video, which is crucial for accurate moment retrieval and highlight detection. As illustrated in Figure 1 , the importance of each word in a text query can vary depending on the video content. For instance, when we consider a text query ‘A dog is playing in the garden’ , the importance of words such as ‘dog’ or ‘garden’ can shift based on the predominant scenes in the video. In Video #1, where most scenes contain ‘garden’ , the word ‘dog’ is more critical than ‘garden’ for specifying the desired video segment. Conversely, in Video #2, ‘dog’ is predominantly ‘playing’ indoors, making ‘garden’ more essential than ‘dog’ for identifying the relevant video segment. This indicates that the importance of words in a text query can vary significantly depending on the video context. Therefore, it is necessary to consider such keyword variations for effective moment retrieval and highlight detection. However, existing methods fall short in addressing this keyword variation, as they rely on text features extracted independently of the video context, failing to capture the dynamic importance of words relative to the visual content. In this paper, we propose a Video Context-aware Keyword Attention module that effectively captures keyword variations by considering the overall video context. Our approach addresses two main challenges: ( i ) how to effectively encode the overall context of a video to capture keyword variation, and ( ii ) how to capture and utilize desired text keywords within their relevant video contexts. First, effective keyword extraction requires a comprehensive understanding of the overall context of the video. To address this, we tackle the challenge ( i ) by introducing a video context clustering module that leverages temporally-weighted clustering to group similar video scenes. This approach allows our model to grasp the high-level flow and structure of the video. The resulting cluster assignments provide a concise representation of the overall video context and are leveraged to understand keyword dynamics. Furthermore, since these clustered features contain information about scene changes, they are further used as additional hints for moment retrieval and highlight detection. To address the challenge ( ii ), we propose a keyword weight detection module. This module recognizes less frequently occurring but important words in the text query and calculates the similarity between clustered video features and text features to generate a keyword weight vector. This vector captures information about the important words in the text query within the video context, allowing our framework to adjust the keywords based on the overall video context dynamically. Based on this, we introduce keyword-aware contrastive learning to incorporate keyword weights and facilitate a fine-grained alignment between visual and text features. As a result, our method allows for accurate moment retrieval and highlight detection. The major contributions of our paper are as follows: • We propose a video context-aware keyword attention module to capture keyword variations by considering overall context of the video for effective moment retrieval and highlight detection. To the best of our knowledge, this is the first work to address this aspect in video moment retrieval and highlight detection tasks. • We introduce keyword-aware contrastive learning to integrate keyword weight information, enhancing the fine-grained alignment between visual and text features. This approach improves the ability of model to understand the relationship between textual queries and video content. • Experimental results on QVHighlights, TVSum, and Charades-STA demonstrate the effectiveness of our method for moment retrieval and highlight detection. Figure 2: Overall configuration of our moment retrieval and highlight detection. \otimes indicates element-wise multiplication. Related Works Moment Retrieval The connection between visual and language cues has become important in machine learning ( Lee et al. 2022b ; Park et al. 2024b ; Lee et al. 2024 ) . Moment retrieval aims to locate relevant moments in a video based on a

Read the complete text