Moment Quantization for Video Temporal Grounding

Xiaolong Sun, Le Wang, Sanping Zhou, Liushuai Shi, Kun Xia, Mengnan Liu, Yabing Wang, Gang Hua · 2025

Read Moment Quantization for Video Temporal Grounding by Xiaolong Sun, Le Wang, Sanping Zhou, Liushuai Shi, Kun Xia, Mengnan Liu, Yabing Wang, Gang Hua (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02286.txt.

Introduction

Xiaolong Sun and co-authors find video moments matching a text query by quantizing time spans, in work from 2025. Locating the right seconds in a long video is a core search task. Earlier methods blurred the line between matching and non-matching moments.

Their method keeps a learned codebook of typical moments. Each video span is matched to a codeword, which sharpens the contrast between hits and misses. Matching runs as soft clustering, so varied looks for the same moment lose no detail. Careful starting values and a joint projection step strengthen the codebook. The pipeline stays simple while gaining discrimination. Tests target standard grounding benchmarks for video search.

Catalog record

CREATORXiaolong Sun, Le Wang, Sanping Zhou, Liushuai Shi, Kun Xia, Mengnan Liu, Yabing Wang, Gang Hua
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Moment Quantization for Video Temporal Grounding Xiaolong Sun, Le Wang, Sanping Zhou, Liushuai Shi, Kun Xia, Mengnan Liu, Yabing Wang, Gang Hua Abstract Video temporal grounding is a critical video understanding task, which aims to localize moments relevant to a language description. The challenge of this task lies in distinguishing relevant and irrelevant moments. Previous methods focused on learning continuous features exhibit weak differentiation between foreground and background features. In this paper, we propose a novel M oment- Q uantization based V ideo T emporal G rounding method (MQVTG), which quantizes the input video into various discrete vectors to enhance the discrimination between relevant and irrelevant moments. Specifically, MQVTG maintains a learnable moment codebook, where each video moment matches a codeword. Considering the visual diversity, i.e., various visual expressions for the same moment, MQVTG treats moment-codeword matching as a clustering process without using discrete vectors, avoiding the loss of useful information from direct hard quantization. Additionally, we employ effective prior-initialization and joint-projection strategies to enhance the maintained moment codebook. With its simple implementation, the proposed method can be integrated into existing temporal grounding models as a plug-and-play component. Extensive experiments on six popular benchmarks demonstrate the effectiveness and generalizability of MQVTG, significantly outperforming state-of-the-art methods. Further qualitative analysis shows that our method effectively groups relevant features and separates irrelevant ones, aligning with our goal of enhancing discrimination. 1 Introduction The rapid expansion of short-form video content on social media platforms has made video a highly engaging multi-medium format [ 1 , 19 ] . The significant surge prompts users to selectively engage with brief moments of interest rather than passively watch entire videos. This derives the topic of video temporal grounding (VTG), which aims to ground video clips based on natural language descriptions. Aside from the moment retrieval [ 6 , 13 , 16 ] , recent works in VTG have also been interested in highlight detection [ 15 , 35 ] . As a multi-modal video understanding task, variable and open-vocabulary language descriptions complicate video representation learning in VTG. This makes it difficult to distinguish between relevant and irrelevant moments, posing significant challenges to the task. Figure 1 : (a) Comparison of codebook vectors, foreground and background features between TR-DETR [ 27 ] and our method for a given example. Compared to previous methods focused on learning continuous features, our method, aided by codebook vectors, achieves better foreground aggregation and foreground-background separation. (b) Visualization of moment quantization. The moment quantization discriminates foreground and background moments. The foregrounds are represented by red-related vectors, while the backgrounds are represented by other discrete vectors. The quantized features bring discriminative information to generate more accurate localization. To address this challenge, previous works [ 34 , 36 , 11 ] learn continuous features with complex modal interactions. However, videos contain much more redundant information, while VTG aims to separate video clips into foreground and background, focusing on essential discriminative information. As shown in Fig. 1 (a), we present an example with multiple non-contiguous foreground moments. Due to video information redundancy, previous methods ( e.g . , TR-DETR [ 27 ] ) struggle with distinguishing foreground and similar background features ( i.e . , red and blue points are close in feature space) and poorly grouping foreground features ( i.e . , different red points are far apart in feature space). Additionally, the interested foreground can be described concisely by a discrete language query [ 33 , 32 ] . For example in Fig. 1 (b), we can accurately describe the video moment of a spoon stirring curry using the discrete language query. That motivates us to ask: Can we describe the continuous video moments via discrete vectors to enhance discrimination between relevant and irrelevant moments? Existing works [ 5 , 40 , 39 ] have explored various patch-level quantization on images, however, moment-level quantization on videos remains an unexplored area. To achieve our goal, we propose a M oment- Q uantization based V ideo T emporal G rounding method (MQVTG), which quantizes the video moments into discrete vectors. The foreground and background are distinguished by different vectors, thus improving the discrimination between relevant and irrelevant moments. As shown in Fig. 1 (b), the foreground moments are represented by red-related vectors, while the background moments are represented by other discrete vectors. To construct this moment quantization, a simple implementation, named clip quantization, is proposed to quantize individual video clips into a discrete codebook similar to traditional patch quantization on images. However, clip quantization overlooks the characteristics of the video moments: 1) A moment crosses multiple video clips (cross-clip); 2) A moment described by the same language may express various visual forms (visual diversity). Drive by the two characteristics, the moment quantization is built on the clip quantization with three innovations. First, we maintain the moment codebook based on the video clips after temporal modeling rather than isolated video clips to meet the cross-clip nature of the moment. Second, due to the visual diversity of moments, directly using discrete codewords loses distinctive information. For example in Fig. 1 (a), directly replacing all video features with limited codebook vectors is clearly inappropriate. Thus, we deliver the quantized continuous features into the downstream localization module, to retain the visual diversity. Furthermore, two effective prior initialization and joint projection strategies are proposed to enhance the moment codebook. We conduct extensive experiments on six popular video temporal grounding benchmarks to validate the effectiveness of our method, which achieves state-of-the-art performances for all benchmarks. The moment quantization also serves as a plug-and-play component, performing well in both encoder-only and encoder-decoder architectures. As shown in Fig. 1 (a), our method can effectively group foreground and separate fore/background features, aligning with our goal of enhancing discrimination. Our main contributions are summarized as follows: • To our knowledge, this is the first introduction of vector quantization to the VTG task. We propose a Moment-Quantization based Video Temporal Grounding method that describes the continuous video moments via discrete vectors to enhance discrimination between moments. • To adapt vector quantization from images to videos, we introduce two progressive implementations, clip quantization and moment quantization, both capable of quantizing video features to capture discriminative information. • Clip quantization, as a naive implementation, simply aligns with image quantization, while moment quantization considers the characteristics of video moments, i.e . , cross-clip and visual diversity. • Extensive experiments demonstrate the effectiveness of our method, and it can be integrated into existing grounding models, showing strong generalizability. 2 Related Work Figure 2 : Comparison of three quantization methods including (a) the classic image quantization, (b) clip quantization that is a simple implementation of vector quantization for videos, and (c) the improved moment quantization for video temporal grounding. The design of the moment codebook used in moment quantization is shown in (d). Video Temporal Grounding. Video temporal grounding can be divided into moment retrieval (MR) [ 6 , 10 , 13 , 21 ] and highlight detection (HD) [ 2 , 9 , 28 ] , which localizes the relevant moments and scores the clip-wise correspondence to the query. Recent research [ 13 ] constructs the QVHighlights dataset to facilitate joint learning of MR&HD and proposes a baseline model based on

Read the complete text