Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval
Jun Li, Peifeng Lai, Xuhang Lou, Jinpeng Wang, Yuting Wang, Ke Chen, Yaowei Wang, Shu-tao Xia · 2026
Read Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval by Jun Li, Peifeng Lai, Xuhang Lou, Jinpeng Wang, Yuting Wang, Ke Chen, Yaowei Wang, Shu-tao Xia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06083.txt.
Introduction
Jun Li and seven co-authors build a video retrieval system that models its own uncertainty explicitly. In partially relevant retrieval, a short text query describes only part of a long untrimmed video. Vague queries blur meaning across videos, and sparse timing labels give little matching evidence inside each video.
Their framework, Holmes, gathers evidence at two levels in a hierarchy. Between videos, similarity scores become evidential support under a Dirichlet model, guiding calibrated learning with fine grained query checks. Within a video, a soft alignment between query and clips, built on flexible optimal transport, gathers denser evidence while damping false local matches. Experiments show the system beating leading methods for this retrieval task.
Catalog record
| CREATOR | Jun Li, Peifeng Lai, Xuhang Lou, Jinpeng Wang, Yuting Wang, Ke Chen, Yaowei Wang, Shu-tao Xia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06083
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval Jun Li, Peifeng Lai, Xuhang Lou, Jinpeng Wang, Yuting Wang, Ke Chen, Yaowei Wang, Shu-tao Xia Abstract Partially relevant video retrieval aims to retrieve untrimmed videos using text queries that describe only partial content. However, the inherent asymmetry between brief queries and rich video content inevitably introduces uncertainty into the retrieval process. In this setting, vague queries often induce semantic ambiguity across videos, a challenge that is further exacerbated by the sparse temporal supervision within videos, which fails to provide sufficient matching evidence. To address this, we propose Holmes , a hierarchical evidential learning framework that aggregates multi-granular cross-modal evidence to quantify and model uncertainty explicitly. At the inter-video level, similarity scores are interpreted as evidential support and modeled via a Dirichlet distribution. Based on the proposed three-fold principle, we perform fine-grained query identification, which then guides query-adaptive calibrated learning. At the intra-video level, to accumulate denser evidence, we formulate a soft query-clip alignment via flexible optimal transport with an adaptive dustbin, which alleviates sparse temporal supervision while suppressing spurious local responses. Extensive experiments demonstrate that Holmes outperforms state-of-the-art methods. Code is released at ICML26-Holmes . 1 Introduction Figure 1: (a) Precise queries produce a clear response margin, with the ground-truth video distinctly outperforming all candidates. (b) Under-determined queries are poorly interpreted, resulting in weak and non-discriminative similarity responses. (c-d) Polysemous queries are semantically ambiguous and correspond to multiple plausible candidates, necessitating more consideration to account for the associated uncertainty. (e) Optimizing only the closest pair, MIL yields sparse supervision and under-trained clips, making the model vulnerable to local noise from a globally irrelevant “running” video, causing spurious spiky activations and unreliable retrieval. With the rapid growth of online videos, Text-to-Video Retrieval (T2VR) ( 30 ) has received increasing attention. While most T2VR methods are designed for pre-trimmed videos with fully descriptive queries, they may deviate from real-world settings, where videos are generally untrimmed and queries correspond only to partial segments. Motivated by this discrepancy, Partially Relevant Video Retrieval (PRVR) ( 11 ) was proposed for retrieving untrimmed videos given partially relevant queries. However, this task faces a fundamental challenge rooted in data asymmetry: compared with rich video content, text queries are often brief and vague, inevitably introducing inherent uncertainty into the retrieval process. Modeling this uncertainty involves two compounded challenges that exist at both inter- and intra-video levels: ( i ) Inter-video Semantic Ambiguity. Owing to the inherent uncertainty, queries often exhibit fine-grained heterogeneity. As illustrated in Figure 1 (a-d), beyond ideal precise queries , real-world data contains under-determined queries (insufficient semantics yielding low confidence), and polysemous queries (ambiguous semantics matching multiple candidates). Most prior methods ignore this heterogeneity ( 54 ) or handle it implicitly via regularization ( 64 ; 41 ) . While ARL ( 8 ) attempts to detect ambiguous video-text pairs, it lacks a mechanism to explicitly quantify the specific type of uncertainty. ( ii ) Intra-video Sparse Supervision. The inter-video ambiguity is also exacerbated by the lack of reliable evidence within the video. The prevalent multi-instance learning (MIL) paradigm supervises only the single best-matching clip, leading to sparse supervision . As shown in Figure 1 (e), this supervisory imbalance provides limited training signals and prevents the model from learning a holistic representation, making it vulnerable to spurious local noise. While distilling cues from CLIP ( 45 ) can alleviate sparsity ( 15 ) , such frame-level fragments fail to capture the temporal dynamics required to confirm a video’s relevance, thereby fueling the uncertainty at the inter-video level and resulting suboptimal results. To address these challenges, we argue that the model needs to go beyond deterministic similarity scores and should evaluate the uncertainty of its predictions. In this paper, we propose Holmes , a unified evidential learning framework that operates at dual levels to disentangle and model uncertainty. First, to handle inter-video uncertainty , we treat cross-modal similarities as evidence parameterized by a Dirichlet distribution. This allows us to move beyond simple matching scores and perform a diagnosis of the query type. We devise an Uncertainty Guided Identification (UGI) strategy based on a three-fold principle, namely epistemic uncertainty , label consistency , and aleatoric uncertainty , to categorize queries and apply query-adaptive label calibration. Second, to resolve the intra-video sparsity that fuels uncertainty, we seek to accumulate denser evidence. We formulate a Flexible Optimal Transport (FOT) scheme to establish soft alignments between the query and multiple relevant clips. To ensure robustness against noise, we integrate a dustbin bucket to absorb irrelevant clips, thereby providing more comprehensive and reliable temporal supervision. Extensive experiments on ActivityNet Captions ( 29 ) , Charades-STA ( 20 ) , and TVR ( 31 ) demonstrate that Holmes achieves state-of-the-art performance. Beyond standard retrieval metrics, our analysis reveals that Holmes effectively interprets query ambiguity and suppresses spurious MIL activations. Our main contributions are summarized as follows: \bullet Conceptual : We provide a fresh perspective on PRVR by revisiting the problem through the lens of uncertainty, explicitly linking inter-video semantic ambiguity with intra-video sparse supervision. \bullet Methodological : We propose Holmes , a dual-level evidential learning framework. It features a three-fold principle for fine-grained query identification and label calibration, coupled with Flexible Optimal Transport to enable dense and robust temporal alignment. \bullet Empirical : Holmes achieves significant improvements on three benchmarks. Detailed ablation studies further validate its capability in quantifying query uncertainty and effectively suppressing spurious local noise. 2 Related Works Partially Relevant Video Retrieval Unlike traditional Text-to-Video Retrieval (T2VR) methods ( 46 ; 30 ; 28 ) , which retrieve fully relevant pre-trimmed videos, Partially Relevant Video Retrieval (PRVR) ( 33 ; 26 ; 57 ; 50 ; 25 ) , first introduced by MS-SL ( 11 ) , targets untrimmed videos where only partial segments match the query. Video Corpus Moment Retrieval (VCMR) ( 65 ; 7 ) is another widely studied retrieval task that aims to localize specific moments within a large video corpus. PRVR is similar to the first stage of VCMR, as both conduct coarse video-level retrieval. But VCMR requires moment-level annotations, which hinders scalability. Existing PRVR methods largely ignore semantic ambiguity and primarily focus on clip modeling. MS-SL ( 11 ; 6 ) constructs redundant clip embeddings via a sliding window scheme, whereas ProtoPRVR ( 40 ) accelerates retrieval by generating a small set of prototypes. GMMFormer ( 54 ; 53 ) implicitly models clips through Gaussian attention. While RAL ( 64 ) leverages probabilistic embeddings and MSC-PRVR ( 41 ) alleviates semantic collapse through auxiliary training losses to implicitly model query ambiguity, ARL ( 8 ) explicitly identifies ambiguous query-video pairs. Nevertheless, these methods fail to distinguish fine-grained query categories and overlook the sparse supervision induced by multiple instance learning, which leads to suboptimal retrieval performance. We propose Holmes , which identifies query heterogeneity through inter-video evidential learning and establishes soft query-clip associations via intra-video evidential learning, providing denser supervisory signals. Uncertainty Learning Despite their success, deep neural networks remain largely deterministic and lack explicit mechanisms for uncertainty quantification. Early attempts to estimate uncertainty primarily relied on Bayesian Neural Networks (BNNs) ( 18 ) and Monte Carlo Dropout ( 19 ) . More recently, Evidential