DEL : D ense E vent L ocalization for Multi-modal Audio-Visual Understanding
Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert · 2025
Read DEL : D ense E vent L ocalization for Multi-modal Audio-Visual Understanding by Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23196.txt.
Introduction
Mona Ahmadian, Amir Shirian, Frank Guerin, and Andrew Gilbert present DEL, a system to find actions in long videos. Their twenty twenty-five paper deals with real clips where acts overlap and link in complex ways. The task is to spot each act and say when it starts and ends, at fine time steps.
DEL has two main parts. One lines up the sound and image streams and sharpens each on its own with masked attention. The other mixes the two streams at many scales, so both big themes and small details count. On four known data sets, DEL beat past systems by clear margins.
Catalog record
| CREATOR | Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-23196
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DEL : D ense E vent L ocalization for Multi-modal Audio-Visual Understanding Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert Abstract Real-world videos often contain overlapping events and complex temporal dependencies, making multimodal interaction modeling particularly challenging. We introduce DEL , a framework for dense semantic action localization, aiming to accurately detect and classify multiple actions at fine-grained temporal resolutions in long untrimmed videos. DEL consists of two key modules: the alignment of audio and visual features that leverage masked self-attention to enhance intra-mode consistency and a multimodal interaction refinement module that models cross-modal dependencies across multiple scales, enabling high-level semantics and fine-grained details. Our method achieves state-of-the-art performance on multiple real-world Temporal Action Localization (TAL) datasets, UnAV-100, THUMOS14, ActivityNet 1.3, and EPIC-Kitchens-100, surpassing previous approaches with notable average mAP gains of +3.3%, +2.6%, +1.2%, +1.7% (verb), and +1.4% (noun), respectively. The source code will be made publicly available. 1 Introduction Temporal action localization (TAL) is a crucial and challenging task in video understanding, focusing on identifying the temporal boundaries of action instances and classifying them within untrimmed videos [ 41 ] . Understanding real-world scenes and events is inherently a multimodal perception process, integrating visual and auditory cues to achieve comprehensive video understanding [ 18 , 30 , 32 ] . For instance, distinguishing between a person speaking and merely mouthing words is difficult with visual information alone, as both actions involve similar lip movements. Incorporating audio cues, however, helps resolve this ambiguity by detecting the presence or absence of vocal sounds. Despite the complementary nature of visual and audio information, modeling multimodal interactions remains challenging due to modality misalignment, the length of the actions, and the complex interplay of cross-modal dependencies—particularly in videos with overlapping, concurrent events [ 40 ] . Figure 1 : Real-world videos contain overlapping events of varying durations, making precise localization challenging. The image presents ground-truth (GT) annotations alongside the predictions of DEL, an audio-only model (A), and a visual-only model (V). The unimodal models struggle with a certain category. In contrast, DEL effectively detects and classifies both short and long-duration events, even when they co-occur, demonstrating its superior multimodal fusion capability. Prior efforts have made progress in Audio-Visual Event Localization (AVE) for trimmed videos, where each video typically contains a single, isolated audio-visual event [ 12 , 42 , 33 ] . Unlike AVE, dense localization of audio-visual events requires identifying and recognizing all events within an untrimmed video, capturing short and long-duration interactions that may overlap [ 13 , 17 , 3 , 9 ] . Recent advances have demonstrated the effectiveness of transformer networks and Feature Pyramid Networks (FPN) for TAL [ 53 , 58 , 46 , 37 ] , significantly improving performance by leveraging multi-resolution visual features. However, those approaches overlook the role of audio. A persistent challenge in audio-visual event localization is the effective extraction and fusion of information across modalities, especially when multiple events co-occur. Traditional approaches either process audio and visual streams independently or rely on late fusion, limiting their ability to model fine-grained temporal dependencies. Moreover, most multi-modal methods employ pretrained models to extract features, which, while beneficial for representation learning, often lead to misaligned features due to differences in pretraining objectives and domain shifts between audio and visual modalities. Additionally, contrastive learning techniques often focus on instance-level alignment while neglecting intra-video relationships such as temporal coherence and cross-event correlations, crucial for distinguishing similar events occurring at different times. To address this challenge, we propose DEL , a novel framework designed to model cross-modal dependencies while explicitly preserving fine-grained temporal structure. Our approach enables precise temporal alignment and fine-grained spatial understanding across multiple audio-visual events. DEL uses a multi-scale fusion strategy, ensuring robust event localization even in densely overlapping scenarios. We incorporate two key modules into our transformer-based model: 1) a multimodal adaptive attention mechanism using a masked self-attention mechanism that ensures temporal coherence and intra-modal consistency within the audio-visual information. 2) a path aggregation network that introduces multiple temporal resolutions to capture fine-grained and high-level semantics in the sequences. Additionally, we introduce an intra-sample contrastive loss to refine feature discrimination within a single data mode and an inter-sample contrastive loss for better alignment of audio-visual representations across different samples. Instead of manually defining positive and negative samples, our method introduces a feature scoring mechanism that enables the model to automatically assign confidence scores and select contrastive pairs during training. This enhances feature discrimination and cross-modal coherence for precise event localization. We demonstrate state-of-the-art performance on multiple benchmarks, including UnAV-100 [ 13 ] , THUMOS14 [ 17 ] , ActivityNet 1.3 [ 3 ] , and EPIC-Kitchens-100 [ 9 ] , achieving significant improvements over prior methods. 2 Related Works 2.1 Single-Modality Temporal Localization Tasks Deep learning has significantly advanced temporal action localization (TAL), enabling the detection and classification of actions in untrimmed videos. TAL methods are generally categorized into two-stage and single-stage approaches. Two-stage methods generate action proposals with confidence scores before refining and classifying them [ 23 , 22 ] . In contrast, single-stage approaches directly localize actions in a single pass, eliminating the need for proposal generation. Our work follows a single-stage TAL. These methods can be further classified into anchor-based [ 6 , 28 , 59 , 44 ] and anchor-free approaches [ 21 , 54 , 29 , 48 , 49 ] . Anchor-based methods employ predefined multi-scale temporal anchors to detect potential action segments, refining their boundaries through regression techniques. Conversely, anchor-free methods predict action boundaries directly by modeling temporal dependencies or regressing distances to action start and end points. Recent advances integrate graph neural networks (GNNs) [ 57 , 54 , 51 ] and transformers [ 43 , 39 , 5 ] to enhance temporal modeling. Transformers, in particular, have demonstrated superior performance by leveraging self-attention mechanisms to capture long-range dependencies. Inspired by progress in object detection [ 35 , 8 ] and saliency detection [ 21 ] , recent state-of-the-art approaches incorporate transformer-based feature pyramid networks (FPNs) [ 7 , 46 , 58 , 37 ] . These networks effectively combine multi-scale feature representation with classification and regression heads, improving accuracy and efficiency over conventional methods. Figure 2 : Overview of our proposed DEL framework. Our model integrates (1) an adaptive attention mechanism for aligning audio and visual features, (2) inter- and intra-sample contrastive learning to enhance event discrimination, and (3) a multi-scale path aggregation network for feature fusion. \mathbin{\|} represents the concatenation operation. 2.2 Multimodal Audio-Visual Event Localization While joint audio-visual representation learning has been extensively studied in video retrieval and classification [ 19 , 50 , 1 ] , its application in TAL is less explored, primarily due to the challenges of modality misalignment and the need for fine-grained temporal modeling. Most existing methods assume that the audio and visual events are perfectly aligned [ 40 , 2 , 55 ] , limiting their effectiveness in real-world settings where events may be asynchronous or overlapping. As highlighted in our introduction, real-world scenarios often involve a complex interplay between modalities and concurrent events. Current audio-visual TAL models employ two-stage late fusion strategies, where audio and visual modalities interact only at the final classification stage. This approach