Enhancing Weakly Supervised Video Grounding via Diverse Inference Strategies for Boundary and Prediction Selection
Sunoh Kim, Daeho Um · 2025
Read Enhancing Weakly Supervised Video Grounding via Diverse Inference Strategies for Boundary and Prediction Selection by Sunoh Kim, Daeho Um (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23181.txt.
Introduction
Sunoh Kim and Daeho Um improve how AI finds the moment a sentence describes in a video, when training data lacks the true time marks. This task, weakly supervised video grounding, usually leans on bell-shaped guesses centered on a likely moment. The moment's edges are set by a fixed rule, and the best guess is picked by how much it overlaps its rivals.
The authors attack both habits at the inference stage, with no retraining. They propose new ways to draw boundaries from several overlapping bell curves, capturing more varied candidate spans. They also propose selection rules that weigh each candidate's own quality, not just its agreement with others. On two standard datasets of captioned activities and household videos, the new strategies lifted accuracy across existing models.
Catalog record
| CREATOR | Sunoh Kim, Daeho Um |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23181
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Weakly Supervised Video Grounding via Diverse Inference Strategies for Boundary and Prediction Selection Sunoh Kim, Daeho Um Abstract Weakly supervised video grounding aims to localize temporal boundaries relevant to a given query without explicit ground-truth temporal boundaries. While existing methods primarily use Gaussian-based proposals, they overlook the importance of (1) boundary prediction and (2) top-1 prediction selection during inference. In their boundary prediction, boundaries are simply set at half a standard deviation away from a Gaussian mean on both sides, which may not accurately capture the optimal boundaries. In the top-1 prediction process, these existing methods rely heavily on intersections with other proposals, without considering the varying quality of each proposal. To address these issues, we explore various inference strategies by introducing (1) novel boundary prediction methods to capture diverse boundaries from multiple Gaussians and (2) new selection methods that take proposal quality into account. Extensive experiments on the ActivityNet Captions and Charades-STA datasets validate the effectiveness of our inference strategies, demonstrating performance improvements without requiring additional training. 1 Introduction Weakly supervised video grounding (WSVG) is a fundamental task in video understanding, aiming to localize relevant temporal boundaries based on a given natural language query, without requiring explicit ground-truth annotations of temporal boundaries [ 14 , 17 , 19 , 5 ] . This problem has attracted significant attention due to its potential applications in video retrieval [ 2 ] , video summarization [ 12 ] , and human-computer interaction [ 15 ] . However, the absence of ground-truth annotations presents unique challenges in generating effective temporal boundaries. Figure 1 : Overview of the proposed inference process, consisting of boundary prediction strategies and top-1 prediction selection strategies. To improve the performance of Gaussian proposal-based methods, we explore five strategies for boundary prediction and four strategies for top-1 prediction selection to derive an optimal combination of inference strategies. Previous methods for WSVG have primarily relied on Gaussian-based proposals, where a Gaussian function is used to model temporal moments within a video. In early methods, single Gaussian-based proposals [ 8 , 11 , 18 , 19 , 20 ] provide a structured way to represent temporal moments. However, the unimodal nature of the single Gaussian proposals limits their ability to capture diverse event occurrences in complex videos. To overcome this limitation, Gaussian mixture proposals [ 5 ] are introduced, generating multiple Gaussians with different weights. Gaussian mixture proposals enable a comprehensive representation by modeling multiple events, thereby improving the likelihood of capturing relevant temporal moments. While significant progress has been made in improving proposal generation, the effectiveness of these proposals is ultimately dependent on inference strategies used to determine temporal boundaries and select final predictions. However, the inference strategies have been largely overlooked in existing methods. To determine temporal boundaries, most existing methods define the boundaries as half a standard deviation away from the mean of a single Gaussian proposal. While this approach is somewhat effective, it does not sufficiently explore the diverse range of predictions derived from the multiple Gaussians, potentially leading to suboptimal boundary predictions. Similarly, the top-1 prediction selection process presents another key challenge in existing methods. Most existing methods rely heavily on intersection-over-union (IoU) scores between proposals to rank and select the best proposal, without considering the semantic relevance of proposals to the given query. As a result, high-IoU proposals may still fail to capture the most meaningful moments in the video. To address these issues, we explore various inference strategies by proposing two major enhancements: For boundary prediction, we introduce novel strategies to determine temporal boundaries by leveraging properties of Gaussian mixtures, enabling more precise localization. For top-1 prediction selection, we introduce novel strategies that assess both IoU and proposal quality, ensuring that the most meaningful proposal is reflected. Incorporating previous inference strategies from [ 5 , 20 ] , we experimentally explore and compare five strategies for boundary prediction and four strategies for top-1 prediction selection, analyzing their effectiveness to derive an optimal combination of inference strategies. Our key contributions can be summarized as follows: 1) We investigate the underexplored role of inference strategies and identify their limitations in existing Gaussian-based methods. 2) We propose novel boundary prediction and top-1 prediction selection strategies to enhance the temporal localization of Gaussian mixture proposals and capture the most meaningful proposal. 3) Through extensive experiments on the ActivityNet Captions [ 9 ] and Charades-STA [ 3 ] datasets, we demonstrate that our inference strategies enhance performance without requiring additional training. 2 Related Work Most weakly supervised video grounding methods can be broadly categorized into two main approaches: sliding window-based methods and reconstruction-based methods. Each approach has distinct strategies for generating temporal proposals. Sliding window-based methods [ 4 , 14 , 17 ] rely on a predefined sliding window strategy to generate candidate temporal segments, where the most relevant proposal is selected based on its probability of containing the target moment. To enhance the effectiveness of these proposals, [ 14 ] introduces a multi-level co-attention mechanism that facilitates the learning of visual-semantic representations by integrating contextual dependencies. Similarly, [ 4 ] exploits the relationships between multiple sentence queries to capture cross-moment dependencies within videos, thereby improving localization accuracy. Despite their potential, sliding window-based methods suffer from inefficiencies due to their exhaustive proposal generation. They produce a large number of overlapping proposals with fixed lengths, necessitating the application of Non-Maximum Suppression (NMS) [ 13 ] to eliminate redundancy. However, this process not only demands substantial computational resources but also relies on prior knowledge about the distribution of temporal segment lengths within each dataset. Reconstruction-based methods [ 10 , 19 , 20 ] approach the problem from a different perspective, assuming that high-quality temporal proposals should be capable of reconstructing the original query from a randomly masked query. Early work in this category [ 10 ] focuses on capturing and aggregating contextual information from video-sentence pairs to assign scores to proposals sampled across different temporal scales. However, these methods often face a significant computational burden, as they must select the optimal proposal from a large candidate set, which can be inefficient. To address this issue, Zheng et al . [ 19 ] introduce a learnable Gaussian-based proposal mechanism that reduces the number of candidate proposals while maintaining high localization performance. This Gaussian proposal framework has since been widely adopted and further extended in multiple studies, including [ 5 , 6 , 8 , 11 , 18 , 20 ] . Kim et al . [ 5 ] propose Gaussian mixture proposals, which provide more expressive modeling capabilities compared to the single Gaussian proposals. However, existing methods largely overlook the role of inference strategies in determining temporal boundaries and selecting the final prediction. In contrast, our work systematically explores and proposes novel inference strategies to enhance the effectiveness of Gaussian-based proposals. 3 Proposed Method For weakly supervised video grounding, we predict temporal boundaries relevant to a given query. The query-relevant boundaries are defined by the start time s\in\mathbb{R} and end time e\in\mathbb{R} . Adopting Gaussian mixture proposals introduced in [ 5 ] , we generate N Gaussian mixture proposals \{\mathbf{P}^{(n)}\}_{n=1}^{N} to predict the query-relevant boundaries. Let \mathbf{P}^{(n)} denote the n -th Gaussian mixture proposal that is represented by a set of Gaussian