Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos
2025
Read Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06598.txt.
Introduction
Locating sounds in ordinary stereo videos, and judging how far away they are, is tackled by Davide Berghi and one co-author in a 2025 audio-visual paper. The task mixes naming what made a sound, when it happened, and where it sits in three-dimensional space. Meaning is the hardest part to model, and multichannel recordings limit training data.
The authors enrich a standard system with pretrained models that link language to sound and to images. These embeddings feed a modified Conformer block built for cross-modal fusion. Tests on the DCASE 2025 development set separate each model's contribution, and the team also describes large synthetic audio and audio-visual datasets built for pretraining, expanded by swapping left and right channels. The full system, with ensembling and visual cleanup, placed second in its challenge track.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06598
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos Abstract In this study, we address the multimodal task of stereo sound event localization and detection with source distance estimation (3D SELD) in regular video content. 3D SELD is a complex task that combines temporal event classification with spatial localization, requiring reasoning across spatial, temporal, and semantic dimensions. The last is arguably the most challenging to model. Traditional SELD approaches typically rely on multichannel input, limiting their capacity to benefit from large-scale pre-training due to data constraints. To overcome this, we enhance a standard SELD architecture with semantic information by integrating pre-trained, contrastive language-aligned models: CLAP for audio and OWL-ViT for visual inputs. These embeddings are incorporated into a modified Conformer module tailored for multimodal fusion, which we refer to as the Cross-Modal Conformer. We perform an ablation study on the development set of the DCASE2025 Task3 Stereo SELD Dataset to assess the individual contributions of the language-aligned models and benchmark against the DCASE Task 3 baseline systems. Additionally, we detail the curation process of large synthetic audio and audio-visual datasets used for model pre-training. These datasets were further expanded through left-right channel swapping augmentation. Our approach, combining extensive pre-training, model ensembling, and visual post-processing, achieved second rank in the DCASE 2025 Challenge Task 3 (Track B), underscoring the effectiveness of our method. Future work will explore the modality-specific contributions and architectural refinements. \NoHyper 1 Introduction Sound Event Localization and Detection [ 1 ] is a combined task that integrates sound event detection (SED) [ 2 ] and sound source localization (SSL) [ 3 ] . The goal is to identify active sound events from predefined target classes, track their temporal activity, and estimate their spatial positions. SELD systems are crucial for a wide range of real-world applications, including human-robot interaction [ 4 ] , security monitoring, and immersive media production [ 5 ] . Since 2019, SELD has been a dedicated task in the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge. Over successive editions, the task has evolved to include increasingly complex scenarios, such as moving sound sources [ 6 ] , ignoring external interfering sounds [ 7 ] , incorporating visual input to enable multimodal SELD in 360 ∘ videos [ 8 ] , and estimating source distance [ 9 ] . For the 2025 edition, the challenge has shifted from the traditional 4-channel first-order ambisonics (FOA) and microphone array (MIC) formats to stereo SELD using conventional frontal video content, i.e., leveraging only left and right audio channels and perspective video. This format is better aligned with the requirements of conventional media content. In stereo SELD the DOA prediction is limited to the azimuth angles in the range [-90 ∘ , 90 ∘ ], as elevation is ill-defined when using just two horizontally arranged channels, and distinguishing between front and back sources is inherently ambiguous. Source distance estimation, introduced in the previous edition, remains part of the task, which is therefore often referred to as 3D SELD. Additionally, in the audio-visual track, a new subtask has been introduced: predicting whether sound sources are onscreen or offscreen. SELD is a non-trivial task, as it requires reasoning across spatial, temporal, and semantic dimensions: spatial information gives cues for direction and distance estimation; temporal information marks movements and onsets/offsets of sound source activity; semantic information identifies objects, their relations and likely behaviors. Recent advances in large language models (LLMs) [ 10 ] , vision-language models (VLMs) [ 11 ] , and audio-language models (ALMs) [ 12 ] demonstrate that language is a powerful lens for enabling semantic understanding and complex reasoning in relation to media content. Building on this, we posit that leveraging language-aligned models can enhance a SELD model’s semantic reasoning and hence indirectly benefit spatial and temporal reasoning. Traditionally, spatial localization relies on multichannel audio, allowing models to infer source positions through inter-channel time and level differences. This dependence limits the use of large-scale pre-training datasets. While synthetic audio [ 13 ] and visual [ 14 ] data can partially address this limitation, effective language alignment typically demands extensive pre-training on large-scale, heterogeneous data. Therefore, we extend our SELD architecture by integrating two existing language-aligned models: CLAP [ 15 ] for the audio modality and OWL-ViT [ 16 ] for the visual modality. Specifically, we extract audio and visual embeddings using their respective pretrained encoders and combine them with SELD embeddings obtained from a CNN-Conformer backbone, a widely adopted architecture in SELD research [ 17 , 18 , 19 ] . We argue that the embeddings from CLAP and OWL-ViT are semantically rich and provide complementary information to the task. To fuse these embeddings, we employ an adapted version of the Conformer architecture [ 20 ] , which we refer to as the Cross-Modal Conformer (CMC). This module is used to integrate intra-modal embeddings from different sources (e.g., the SELD audio encoder and CLAP) and inter-modal embeddings (i.e., the combined audio representation and the visual embedding from OWL-ViT). This architecture is outlined in Sec. 2.1 , and an ablation study to demonstrate the individual contributions of the language-aligned models and benchmark against the DCASE Task 3 baseline systems is presented in Sec. 3 . Fig. 1 : Proposed audio-visual model (left). Blue blocks are frozen, red ones are pre-trained on the audio 5k and audio-visual 2k datasets, yellow on audio-visual 2k dataset only. Cross-Modal Conformer (CMC) architecture adapting the original Conformer [ 20 ] to process two generic modalities “Alpha” \alpha and “Beta” \beta (right). To tackle the distance estimation subtask, we partnered common SELD input features with short-term power of the autocorrelation stpACC features [ 21 ] in Sec. 2.2 . Furthermore, we synthesized and carefully curated large audio and audio-visual datasets used to pre-train our models, described in Sec. 2.3 . Final refinements includes a visual post-processing step based on human keypoint detection and model ensembling, which is described in Sec. 3 on experiments. Our best model ranked 2nd at the DCASE 2025 Challenge Task 3 (Track B), demonstrating the effectiveness of the proposed approach. This paper introduces three key contributions: (i) the integration of semantically rich, language-aligned models into an audio-visual SELD architecture; (ii) an adapted Conformer architecture designed to fuse multiple modalities; (iii) substantial performance gains via curated data synthesis and engineering refinements, resulting in a second-place ranking in the DCASE2025 Task 3 Challenge (Track B). 2 Method 2.1 Proposed Architecture The proposed model includes a SELD encoder that extracts embeddings from multichannel inputs, which are fused with CLAP audio embeddings via a cross-modal cross-attention mechanism. We adapted the Conformer architecture to accommodate inputs from different modalities. The resulting audio representation is then fused with visual features from OWL-ViT through a second Cross-Modal Conformer. A final feed-forward module predicts multi-ACCDDOA vectors for up to N{=}3 tracks [ 22 ] , including on/off-screen activity as in the challenge baseline. The model is trained using class-wise Auxiliary Duplicating Permutation Invariant Training (ADPIT) loss [ 23 , 24 , 25 ] . 2.1.1 SELD Encoder We adopt a CNN-Conformer architecture for the SELD encoder, as it is widely adopted in SELD research [ 17 , 18 , 19 ] . The stereo input