Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval

Jun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang, Yaowei Wang, Shu-Tao Xia, Bin Chen · 2026

Read Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval by Jun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang, Yaowei Wang, Shu-Tao Xia, Bin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03653.txt.

Introduction

Jun Li and colleagues tackled video search where a text query matches only part of a long, uncut video. Their DreamPRVR system first sketches broad highlights across the whole clip, then narrows in on fine matches between words and frames. Built from a probabilistic sampler plus a text-guided diffusion step, it beat leading rivals on partially relevant retrieval tests.

Catalog record

CREATORJun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang, Yaowei Wang, Shu-Tao Xia, Bin Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53192
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval Jun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang, Yaowei Wang, Shu-Tao Xia, Bin Chen Abstract Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos based on text queries that describe only partial events. Existing methods suffer from incomplete global contextual perception, struggling with query ambiguity and local noise induced by spurious responses. To address these issues, we propose Dream PRVR , which adopts a coarse-to-fine representation learning paradigm. The model first generates global contextual semantic registers as coarse-grained highlights spanning the entire video and then concentrates on fine-grained similarity optimization for precise cross-modal matching. Concretely, these registers are generated by initializing from the video-centric distribution produced by a probabilistic variational sampler and then iteratively refined via a text-supervised truncated diffusion model. During this process, textual semantic structure learning constructs a well-formed textual latent space, enhancing the reliability of global perception. The registers are then adaptively fused with video tokens through register-augmented Gaussian attention blocks, enabling context-aware feature learning. Extensive experiments show that Dream PRVR outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/CVPR26-DreamPRVR . 1 Introduction Figure 1 : (a) Limited contextual awareness causes spurious local spikes on a globally irrelevant “people boating” video, incorrectly outscoring the ground-truth “demonstrating the accordion through the camera” video which exhibits more overall relevance. (b) MIL considers only the closest pair, leading to sparse supervision where others remain undertrained. (c) DreamPRVR first imagines global registers through text-supervised diffusion, then concentrates on fine-grained learning, thereby realizing coarse-to-fine cross-modal alignment and jointly optimizing all video tokens to form a coherent embedding space. Text-to-Video Retrieval (T2VR) 53 ; 31 ; 76 ; 33 ; 13 facilitates access to large-scale video collections. While most T2VR methods assume a fully relevant setting, where short trimmed videos perfectly match the query, this premise may contrast with real-world applications, where untrimmed videos are common and queries often describe only a partial clip. To bridge this gap, Partially Relevant Video Retrieval (PRVR) 11 was introduced, which aims to retrieve untrimmed videos based on given partial relevant queries. A core challenge in PRVR lies in query ambiguity 79 ; 7 , where a general query may match its ground-truth clip while inadvertently aligns with local clips from other videos. Such ambiguous associations inject noise into precise cross-modal matching, degrading retrieval accuracy. We attribute this to incomplete global contextual perception in the partially relevant learning setting, where queries may be easily misled by globally irrelevant videos with coincidentally similar events, producing local spiky activation responses and retrieval failures, as shown in Fig. 1 (a). Moreover, Fig. 1 (b) shows that the widely used multiple instance learning paradigm (MIL) in PRVR exacerbates this issue by rewarding only the best-matching clip, leaving others undertrained and lacking sufficient contextual grounding to resolve ambiguity. Despite growing attention to global context, most PRVR methods 11 ; 73 ; 25 lack explicit modeling. HLFormer 35 models semantic entailment and RAL 79 captures global uncertainty, yet both treat global context as training-only regularization, leaving video embeddings unrefined at inference. DL-DKD 14 introduces CLIP-based global knowledge, but its teacher model remains temporally constrained. Motivated by register tokens 10 ; 68 ; 2 , we also introduce global registers to store holistic video semantics, which interact with all tokens to enhance local representations, mitigate MIL under-training and suppress noisy local spike activations. However, extracting reliable global registers remains non-trivial due to redundancy and noise in untrimmed videos. Simple pooling or one-step mapping methods may fail to disentangle trustworthy semantics. To address this, we propose Dream PRVR , performing coarse-grained contextual imagination before concentrating on fine-grained representation learning . As depicted in Fig. 1 (c), it first generates global registers through a text-supervised diffusion process that iteratively denoises and refines video semantics, yielding reliable holistic context to enhance representation learning. Executing the generation entails two core challenges: ( i ) obtaining reliable textual supervision to guide semantic generation and ( ii ) decoupling trustworthy global registers from noisy untrimmed videos. For (i), existing query diversity loss 73 ; 72 ; 35 blindly separates all queries, ignoring intra-video correlations. We address with a Query Similarity Preservation (QSP) Loss , treating queries from the same video as complementary positive views of its global semantics. Together with the diversity loss, QSP jointly enhances intra-video compactness and inter-video separability, yielding a well-structured latent space. To further explicitly model this space under textual uncertainty, we introduce a Textual Perturbation Sampler (TPS), which approximates the latent space by sampling within controllable perturbations, providing semantically aligned supervision for register generation. For (ii), a truncated diffusion model is designed to facilitate generation. Rather than initializing from random noise, a Probabilistic Variational Sampler (PVS) constructs a learnable probabilistic latent space to generate video-centric distribution as a semantically grounded generation starting point. Guided by TPS, the Diffusion Register Estimator (DRE) then performs iterative refinement, progressively denoising PVS-initialized embeddings into pure and holistic registers encoding comprehensive contextual semantics. Finally, the Register-augmented Gaussian Attention Blocks (RAB) adaptively fuse the refined registers with video tokens via asymmetric contextual attention masks, enhancing fine-grained representation learning in the end. Empirical results on ActivityNet Captions 32 , Charades-STA 19 and TVR 34 validate the state-of-the-art performance of Dream PRVR . Unlike large-scale diffusion models 55 ; 48 , our model is lightweight, requiring only a few registers and timesteps to achieve notable gains, highlighting high efficiency. Extensive ablations and visualizations further confirm that the registers progressively acquire cleaner semantics through iterative refinement, while textual semantic structure learning enforces a well-organized latent space. To summarize, we make the following contributions. \bullet We propose Dream PRVR , a contextual imagination framework for PRVR, which first generates registers to capture coarse-grained holistic video semantics and then concentrates on fine-grained representation learning, achieving hierarchical and progressive cross-modal alignment. \bullet We supervise register generation with a structured textual space constructed via textual semantic structure learning, estimate registers through a truncated diffusion model initialized from the video-centric distribution and employ register-augmented Gaussian attention for adaptive fusion. \bullet Extensive experiments validate our model’s superiority, with analyses showing effectiveness of global registers and acceptable efficiency of the imagination process. 2 Related Works Partially Relevant Video Retrieval Text-based video retrieval 39 is a core area of information retrieval 57 ; 64 ; 70 ; 38 . Text-to-Video Retrieval (T2VR) 53 ; 33 ; 31 ; 76 ; 65 retrieves fully relevant pre-trimmed videos. Video Corpus Moment Retrieval (VCMR) 60 ; 34 ; 6 localizes specific temporal moments across large video corpora. Partially Relevant Video Retrieval (PRVR) 4 ; 82 ; 59 ; 44 ; 75 ; 7 ; 45 ; 27 , introduced by 11 extends retrieval to untrimmed videos where only partial segments match the query. Existing PRVR methods primarily emphasize clip-level modeling. MS-SL 11 constructs dense clip embeddings via a sliding-window scheme, while ProtoPRVR 44 accelerates retrieval by employing representative prototypes. GMMFormer 73 introduces implicit clip modeling based on Gaussian attention and HLFormer 35 leverages hyperbolic space to capture hierarchical video structures. For learning objectives, DL-DKD 14 distills dynamic CLIP 50 knowledge, ARL 7 formulates query ambiguity with dedicated optimization goals and RAL 79 advances probabilistic query–video embedding learning. Despite these efforts, global contextual semantics remain underexplored,

Read the complete text