Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval

WonJun Moon Cheol-Ho Cho Woojin Jun, Minho Shim Taeoh Kim Inwoong Lee Dongyoon Wee, Jae-Pil Heo · 2025

Read Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval by WonJun Moon Cheol-Ho Cho Woojin Jun, Minho Shim Taeoh Kim Inwoong Lee Dongyoon Wee, Jae-Pil Heo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13035.txt.

Introduction

A research team tackles partially relevant video retrieval in this 2025 paper. In this task a text query matches only part of a long video, and systems face a trade: richer context at many time scales improves accuracy but costs more compute and memory.

Their framework squeezes a video's varied contexts into a fixed set of prototypes, balanced units that stand for its content. Extra objectives keep the prototypes diverse and tied to the right text, and two reconstruction tasks keep them both searchable by text and faithful to the video. A video mixing trick adds weak alignment guidance. Tests on three standard benchmarks check the balance of speed and accuracy.

Catalog record

CREATORWonJun Moon Cheol-Ho Cho Woojin Jun, Minho Shim Taeoh Kim Inwoong Lee Dongyoon Wee, Jae-Pil Heo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13035

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prototypes are Balanced Units for Efficient and Effective Partially Relevant Video Retrieval WonJun Moon Cheol-Ho Cho Woojin Jun, Minho Shim Taeoh Kim Inwoong Lee Dongyoon Wee, Jae-Pil Heo Abstract In a retrieval system, simultaneously achieving search accuracy and efficiency is inherently challenging. This challenge is particularly pronounced in partially relevant video retrieval (PRVR), where incorporating more diverse context representations at varying temporal scales for each video enhances accuracy but increases computational and memory costs. To address this dichotomy, we propose a prototypical PRVR framework that encodes diverse contexts within a video into a fixed number of prototypes. We then introduce several strategies to enhance text association and video understanding within the prototypes, along with an orthogonal objective to ensure that the prototypes capture a diverse range of content. To keep the prototypes searchable via text queries while accurately encoding video contexts, we implement cross- and uni-modal reconstruction tasks. The cross-modal reconstruction task aligns the prototypes with textual features within a shared space, while the uni-modal reconstruction task preserves all video contexts during encoding. Additionally, we employ a video mixing technique to provide weak guidance to further align prototypes and associated textual representations. Extensive evaluations on TVR, ActivityNet-Captions, and QVHighlights validate the effectiveness of our approach without sacrificing efficiency. 1 Introduction Figure 1 : Comparisons in video encoding process. (a) MS-SL utilizes exhaustive clip modeling based on varying lengths of clip windows to encode contexts at diverse temporal scales. (b) GMMFormer performs a similarity-aware feature aggregation via self-attention constrained with predefined Gaussian kernels to reflect locality. Such an adaptive scheme provides high efficiency. (c) To exploit the semantic richness in exhaustive clip modeling without sacrificing efficiency, we learn a fixed number of prototypes that aggregate diverse (varying lengths or potentially disjointed) contexts within a video. (Down) Our proposed method has superiority over previous works in terms of the accuracy-efficiency trade-off. The sheer volume of video content makes it increasingly challenging for users to find the video they are looking for. This underscores the need for accurate and efficient video retrieval systems [ 39 ] that can search through videos consisting of short moments the users are seeking. In response to this demand, Partially Relevant Video Retrieval (PRVR) [ 9 ] has been introduced to extend text-video retrieval with long complex videos where only specific segments correspond to the text query. Typical approaches such as MS-SL [ 9 ] prioritize the accuracy by storing lengthy frame-wise [ 10 ] features and exhaustively modeling clips of varying lengths using different window sizes [ 22 ] , as shown in Fig. 1 (a). With these representations, a video that yields the highest similarity to the input text description is retrieved. While such approaches have made significant progress, they often overlook the importance of efficiency in retrieval systems that operate within extensive databases. To address efficiency, GMMFormer [ 54 ] introduced a local clip modeling, which applies Gaussian constraints with predefined variances around selected frames, as shown in Fig. 1 (b). Specifically, these constraints are integrated within self-attention [ 51 , 11 ] . While this approach improves efficiency by primarily encoding local contexts, it struggles to accurately capture moments when matching queries that describe complex and extended events within video sequences. This limitation arises because local contexts are confined within fixed-length temporal segments. Likewise, we observe the trade-off between achieving efficiency and attaining a thorough understanding of contents. To address the challenge of simultaneously achieving efficiency and embracing diverse contexts, we propose a prototypical network for PRVR. Specifically, we leverage the advantages of the exhaustive clip modeling strategy to encode both the local and global contexts (Fig. 1 (a)), while our prototypes effectively aggregate these features across diverse temporal scales into a smaller footprint, as shown in Fig. 1 (c). Then, only a small set of video prototypes, significantly fewer than the video frame or clip representations, are stored for efficient retrieval. Despite the potential of prototypes, prototypical learning does not inherently guarantee text associativity or the preservation of all contextual details necessary for retrieval due to its implicit learning nature without explicit guidance on how each prototype should encode specific information. To address this, we propose several enhancements to prototypical learning to strengthen the association with text descriptions and improve video understanding. First, we introduce dual reconstruction tasks, consisting of cross-modal (video-to-text) and uni-modal (video-to-video) reconstruction tasks. The cross-modal mask reconstruction task enhances the prototypes’ comprehension of text queries, while the uni-modal reconstruction task preserves the visual content during prototype aggregation. In addition, to guarantee alignment between text queries and the visual prototypes that attend to text-relevant video segments without explicit moment supervision, we employ a video mixing strategy that concatenates videos at a higher frame rate. This strategy trains the model to identify and retrieve a prototype that focuses on the clip within the video that corresponds to the given query. Finally, with these components combined with the orthogonal loss for prototype diversification, our method achieves state-of-the-art results on TVR, ActivityNet Captions, and QVHighlights datasets with both CNN- and Transformer-based backbones. 2 Related Work Text-to-Video Retrieval (T2VR) aims to retrieve the query-relevant videos [ 7 , 8 , 19 , 40 , 46 , 56 , 52 , 50 , 23 ] . One notable stream is utilizing multi-scale spatio-temporal similarity [ 55 , 16 , 41 ] . These works consider the similarity between videos and sentences and also the local similarity between the patches and words for precise retrieval. Another stream is to account for the uncertainty in retrieval. By tackling the many-to-many problems in text-video matching due to diversity in expression in both domains, UATVR [ 12 ] employed Gaussian augmentation to address the uncertainty, and PAU [ 29 ] exploited uncertainty for training and reranking. Yet, T2VR assumes that the full-video-level text descriptions are available, limiting their applicability in practice. Moment Retrieval localizes partial moments in videos that corresponds to the text descriptions [ 2 , 14 , 48 , 35 , 34 , 4 , 60 , 36 , 65 , 61 , 43 , 21 , 32 , 59 , 27 , 37 ] . Despite the similarity with PRVR in the need for localizing the relevant clips, moment retrieval only assumes the pair-wise searching problem. Thus, video-text interactive designs are popularly employed [ 58 , 42 , 49 ] . Video Corpus Moment Retrieval requires multiple videos to be considered at once, thus modality-interactive designs are impractical to be adopted straightforwardly [ 17 , 64 ] . To maintain efficiency and take advantage of interactive designs, a two-stage pipeline is widely adopted where moment retrieval is implemented after the video retrieval [ 18 , 63 ] . The difference to PRVR is the presence of localization labels. Figure 2 : An overview of the clip branch in our prototypical framework which is consistent with the frame branch except for the existence of exhaustive clip modeling. For the video stream, prototype aggregation is implemented upon clip features formed by an exhaustive clip modeling strategy. Text queries are encoded and aggregated through an attention-pooling layer. Finally, the similarity matching between the visual prototypes \hat{\mathbf{P}}^{c} and query token \hat{\mathbf{T}}^{c} is implemented to calculate the text-to-video score S^{c} for

Read the complete text