Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining

Lu Dong, Haiyu Zhang, Hongjie Zhang, Yifei Huang, Zhen-Hua Ling, Yu Qiao, Limin Wang, Yali Wang · 2025

Read Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining by Lu Dong, Haiyu Zhang, Hongjie Zhang, Yifei Huang, Zhen-Hua Ling, Yu Qiao, Limin Wang, Yali Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06557.txt.

Introduction

Lu Dong and co-authors improve weakly supervised video search, where a model finds the moments matching a sentence without frame-level labels. Their 2025 paper tackles a training flaw: similar clips from other videos are usually treated as negatives, which fights the learning goal and wastes real similarity signals.

Their framework instead mines positive samples. For each anchor clip, the rest of the training set splits into look-alikes and non-matches by text similarity. A new contrastive loss pulls the anchor's proposal toward the look-alikes and away from the rest. A ranking loss adds that look-alikes should sit closer to the anchor than a same-video negative does. Mining positives gives cleaner supervision for a task used in video retrieval and editing.

Catalog record

CREATORLu Dong, Haiyu Zhang, Hongjie Zhang, Yifei Huang, Zhen-Hua Ling, Yu Qiao, Limin Wang, Yali Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining Lu Dong, Haiyu Zhang, Hongjie Zhang, Yifei Huang, Zhen-Hua Ling, Yu Qiao, Limin Wang, Yali Wang Abstract The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence. For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning. However, some training samples are highly similar to the anchor sample, directly regarding them as negative samples leads to difficulties for optimization and ignores the correlations between these similar samples and the anchor sample. To address this, we propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision. Specifically, for a given anchor sample, we partition the remaining training set into semantically similar and dissimilar subsets based on the similarity of their text queries. To effectively leverage these correlations, we introduce a PSM-guided contrastive loss to ensure that the anchor proposal is closer to similar samples and further from dissimilar ones. Additionally, we design a PSM-guided rank loss to ensure that similar samples are closer to the anchor proposal than to the negative intra-video proposal, aiming to distinguish the anchor proposal and the negative intra-video proposal. Experiments on the WSTSG and grounded VideoQA tasks demonstrate the effectiveness and superiority of our method. I Introduction Fig. 1: a) Existing methods regard all samples in the remaining training set as negative and push them further from the anchor proposal, which ignores the diversity of the training set. b) Our method mines similar training samples for each anchor sample to leverage more discriminative supervision from other video samples. Two schemes are proposed. 1) A PSM-guided contrastive loss is proposed to distinguish between similar and dissimilar sample pairs. 2) A PSM-guided rank loss is proposed to improve the distinction between the anchor proposal and the negative intra-video proposal. Temporal sentence grounding is a fundamental task for video understanding, where the goal is to locate the start and end timestamps of one segment in the video that semantically corresponds to the given sentence query. This task has the potential for a wide range of applications such as video summarization [ 1 , 2 , 3 ] , video retrieval [ 4 , 5 , 6 ] and human-computer interaction systems [ 7 , 8 , 9 ] . For temporal sentence grounding, the supervised approaches [ 10 , 11 , 12 ] have achieved impressive results but heavily rely on manual annotations of temporal locations for each query-video pair. The annotation process is both time-consuming and label-intensive, limiting the scalability of these methods. To address this limitation, weakly supervised temporal sentence grounding (WSTSG) approaches [ 10 , 13 , 14 , 15 , 16 , 17 ] have been proposed, requiring only query-video pairs without temporal annotations for training. This reduces the dependency on manual labeling, enabling the use of large-scale data that is readily available online, as query-video pairs can be easily obtained from the Internet. Due to the absence of temporal annotations, most previous studies mine negative samples from the training set for contrastive learning, aiming to provide supervision signals for generating proposals. These weakly supervised methods for temporal sentence grounding are broadly categorized into multiple instance learning (MIL)-based and reconstruction-based approaches. Specifically, MIL-based approaches [ 10 , 18 , 19 , 20 ] generate negative samples from mismatched query-video pairs ( e.g. , a sentence query paired with a mismatched video in the training set). The model is trained to distinguish between matched (positive) and mismatched (negative) samples. On the other hand, reconstruction-based researches [ 21 , 22 , 23 , 16 , 17 ] generate negative proposals from segments within the same video, enhancing the model’s ability to handle more confusing video segments. Given an anchor query-video sample, all previous methods mine negative samples from the training set for contrastive training. However, they treat all samples of remaining training set equally, ignoring the inherent diversity among them. As shown in Fig. 1 , for an anchor sample with the query “Person walking in the room open a refrigerator”, there exists a very similar training sample with the query “Person open the refrigerator door”. As they have the same object “refrigerator” and action “open”, their corresponding proposal segments are also semantically similar. If these similar samples are treated as negative samples, the model is forced to push away samples that are semantically alike. This complicates optimization process and leads to suboptimal results. Therefore, for each anchor sample, the remaining training set should be partitioned into two subsets: a similar subset and a dissimilar subset, based on the similarity of each sample to the anchor sample. Different operations can then be applied to each subset to better leverage their unique characteristics. Building on this insight, we propose Positive Sample Mining (PSM), a novel WSTSG framework that mines positive samples for each anchor sample, which aims to leverage more discriminative supervision from other video samples. Specifically, for an anchor sample, we first divide other samples in the remaining training set based on the semantic similarity of their text queries. This will result in a similar sample subset and a dissimilar sample subset. To discriminate these similar and dissimilar sample pairs, we propose a PSM-guided contrastive loss to ensure that the anchor proposal is closer to the similar subset and further from the dissimilar subset, thereby enhancing the model’s ability to capture sophisticated semantic correlations in the sample space. Furthermore, we propose a PSM-guided rank loss to leverage the intra-video information, which guarantees that similar samples are closer to the anchor proposal than to the negative intra-video proposal. It can effectively enhance the distinction between the anchor proposal and the negative intra-video proposal. In summary, our contributions are threefold: • We propose a Positive Sample Mining (PSM) for the WSTSG task. To the best of our knowledge, this is the first attempt to mine positive samples within the training set for this task. • We propose a PSM-guided contrastive loss to discriminate between similar and dissimilar samples. Furthermore, we design a PSM-guided rank loss to enhance the distinction between the anchor proposal and the negative intra-video proposal. • We validate the effectiveness of our approach through comprehensive evaluations on two tasks: WSTSG and grounded VideoQA. Our approach achieves superior performance compared to existing methods. II Related Works II-A Fully Supervised Temporal Sentence Grounding Many previous works [ 10 , 24 , 13 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] focus on fully supervised temporal sentence grounding, where each query-video pair is annotated with precise start and end timestamps. TALL [ 10 ] first produces proposals of various lengths through sliding windows, then uses query-proposal fusion for moment regression. 2D-TAN [ 14 ] uses a two-dimensional map to model the temporal relation between proposal candidates, enabling an exhaustive enumeration of proposals with diverse durations. CRNet [ 31 ] formulates temporal sentence grounding as a video reading comprehension task, enabling the model to capture comprehensive relational information from multiple

Read the complete text