SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning
Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Minju Jeon, Hyungee Kim, Dong-Jin Kim · 2026
Read SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning by Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Minju Jeon, Hyungee Kim, Dong-Jin Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05437.txt.
Introduction
Ye-Chan Kim and team offer SAIL for dense video captioning. Prior work used Gaussian masking. Current methods make non-overlapping masks without semantic consideration. SAIL constructs masks via alignment. A similarity-aware objective guides masks to emphasize regions similar to event captions. An LLM-based plan makes synthetic captions for sparse annotations. Synthetic captions enter through an inter-mask mechanism. ActivityNet Captions and YouCook2 tests show state-of-the-art scores.
Catalog record
| CREATOR | Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Minju Jeon, Hyungee Kim, Dong-Jin Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42641 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05437
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Minju Jeon, Hyungee Kim, Dong-Jin Kim Abstract Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL , which constructs semantically-aware masks through cross-modal alignment. Our similarity-aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics. 1 Introduction Figure 1 : (a) Previous work generates masks that simply cover different temporal regions without considering semantic alignment with corresponding events. (b) Our proposed method leverages cross-modal similarity to guide masks toward event-relevant regions and addresses caption sparsity through LLM augmentation. In diverse multimodal applications 20 ; 3 ; 17 , Dense Video Captioning (DVC) is one of the key research area in the video domain. Unlike conventional video captioning 10 ; 5 ; 34 , DVC requires jointly localizing events and generating captions in long, untrimmed videos. Although diverse DVC methods have been proposed, research has mainly focused on fully-supervised settings 42 ; 13 ; 15 ; 14 . These methods depend on exhaustively labeled data with precise event boundaries and descriptive captions for each event, making annotation prohibitively expensive and labor-intensive for real-world applications. To address these issues, several Weakly-Supervised Dense Video Captioning (WSDVC) studies have emerged 8 ; 7 ; 11 ; 38 , which do not rely on event boundaries and only use captions for training. Figure 2 : (a) The fixed mask baseline applies Gaussian masks with equal width uniformly distributed according to the number of events. It performs comparably to the existing method 11 . (b) shows the dataset’s annotation sparsity example, where potential events are missed. (c) The majority of video samples have events spanning the entire duration but contain only a small number of events (red box). Recently, ILCACM 11 , the current state-of-the-art WSDVC method, introduces a Gaussian masking strategy that applies differentiable masks to video features, enabling implicit event localization through complementary captioning. Although it shows promising results, we observe that this method overlooks the semantic alignment between masked features and their corresponding events, which we believe is a critical aspect for learning discriminative event representations. As shown in Figure 1 , when constructing masks for target events, the existing approach merely focuses on generating mutually exclusive masks without ensuring semantic consistency with corresponding events, often resulting in simplistic and uniformly distributed masks. These simplistic and uniformly distributed masks lead the model to fail to learn meaningful event emphasis, resulting in low-quality captions and inaccurate localization. As shown in Figure 2 (a) , we find that even a fixed, non-trainable mask baseline performs comparably to the existing method 11 on both captioning and localization tasks. This indicates that the existing approach merely learns to emphasize distinct temporal regions without capturing semantic event relevance. To address this issue, our primary goal is to construct genuinely informative masks for target events. We define an informative mask as one that, rather than simply covering a distinct temporal region, highlights video segments whose features exhibit high cross-modal similarity with the corresponding event caption. To achieve this, we leverage the strong cross-modal alignment of Vision-Language model CLIP 31 and guide the model to generate masks based on cross-modal similarity signals rather than enforcing only non-overlap constraints. We train the model to align masked video features with corresponding ground-truth captions while distinguishing non-corresponding ones. Despite the strong overall performance of the cross-modal similarity guide, we observe some failure cases when videos have very sparse annotations (e.g., only 1-2 events). We hypothesize that these failures stem from the sparse characteristic of ground-truth annotations 24 , which leads to insufficient signal for the model to effectively learn the alignment between masked features and their corresponding captions. For example, as illustrated in Figure 2 (b) , a sample video from ActivityNet 22 spans 235 seconds yet contains only three event annotations, leaving numerous potential events unannotated. Also, as shown in Figure 2 (c) , although the majority of videos have annotations covering their entire duration, the event density remains consistently low, demonstrating that most videos show sparse event distributions. In such sparse annotation regimes, we conjecture that over-reliance on only ground-truth caption guidance provides insufficient supervision for learning accurate mask-event alignment, especially in WSDVC. To overcome these limitations, we introduce a Large-Language Model (LLM) based caption augmentation strategy that leverages recent advances in LLMs’ contextual understanding and reasoning capabilities 1 ; 37 . Specifically, we construct prompts using consecutive pairs of ground-truth captions and task the LLM with generating plausible descriptions for the temporal segments connecting them. These synthetic captions are then interleaved with the original annotations, creating a dense supervision signal that captures a more diverse range of potential events. By training on both real and synthetic captions, our cross-modal objective learns more nuanced similarity guidance, enabling masks to capture finer-grained event details. To this end, we propose SAIL , S imilarity- A ware Guidance and I nter-Caption Augmentation-based L earning for WSDVC. Our contributions are as follows: • We propose similarity-aware mask guide, which leverages the cross-modal alignment information to generate event-focused masks. • We propose a novel method that generates synthetic captions via LLM and utilizes them as auxiliary supervision to address annotation sparsity, enabling the model to learn from dense alignment signals. • We validate our method on ActivityNet and YouCook2, achieving state-of-the-art results in both the localization and captioning tasks. 2 Related Work 2.1 Dense Video Captioning DVC is a challenging multi-task that requires the simultaneous execution of event localization and event captioning in untrimmed videos. In the early stages of DVC 22 ; 35 ; 12 , many studies adopted a “localize-then-caption” strategy, where events are first localized and then a caption is generated for each event. However, this 2-stage approach overlooked the interaction between the localization and captioning tasks; also, they rely on complex, hand-crafted components and extensive hyper-parameter tuning. To overcome these limitations, DETR-based transformer architecture 14 ; 15 ; 13 ; 42 emerged to decode both event timestamps and captions in parallel. 42 proposed a method of pre-training on a large-scale narrated video dataset using ASR and then fine-tuning. Based on the above pre-trained model, 14 ; 15 ; 13 ; 42 enhances captioning ability by utilizing an external memory of captions. Despite extensive research into various DVC methods, most studies still operate in fully supervised settings.