A Robust framework for sound event localization and detection on real recordings
2025
Read A Robust framework for sound event localization and detection on real recordings (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22156.txt.
Introduction
Jin Sob Kim, Hyun Joon Park, Wooseok Shin, and Sung Won Han describe their entry to a sound event localization and detection challenge run under DCASE 2022. The task asks systems to spot sounds in recordings, name their class, and estimate where each sound came from. The team built a ResNet-based model inside a system designed to survive messy real-world audio.
Their recipe leans on heavy data augmentation and a training mix that blends real sound scenes with simulated ones while keeping enough real samples in every batch. External sound sources add further variety. At test time they apply augmentation again and ensemble models by clustering confident predictions. The system beat the challenge baselines and stayed competitive on real recordings.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Robust framework for sound event localization and detection on real recordings Abstract This technical report describes the systems submitted to the DCASE2022 challenge task 3: sound event localization and detection (SELD). The task aims to detect occurrences of sound events and specify their class, furthermore estimate their position. Our system utilizes a ResNet-based model under a proposed robust framework for SELD. To guarantee the generalized performance on the real-world sound scenes, we design the total framework with augmentation techniques, a pipeline of mixing datasets from real-world sound scenes and emulations, and test time augmentation. Augmentation techniques and exploitation of external sound sources enable training diverse samples and keeping the opportunity to train the real-world context enough by maintaining the number of the real recording samples in the batch. In addition, we design a test time augmentation and a clustering-based model ensemble method to aggregate confident predictions. Experimental results show that the model under a proposed framework outperforms the baseline methods and achieves competitive performance in real-world sound recordings. Index Terms — DCASE2022, Sound event localization and detection, Framework, Test time augmentation, Ensemble 1 Introduction Sound event localization and detection (SELD) aims identification of both the sound event occurance (SED) and the direction of arrival from the sound source (DOA). Since the localization and detection joining task challenge launched in DCASE2019 [ 1 , 18 ] , the conceptual formulation of the SELD task has been established in DCASE2021 based on activity-coupled Cartesian direction of arrival (ACCDOA) [ 22 ] . Unlike the task has been trained and evaluated on the emulated sound scenes in control up to previous iterations, DCASE2022 engages the real sound scape recordings along with strong temporal and spatial annotations bringing the task to the real-world problem [ 19 ] . The challenge also inspires the exploitation of external data to solve the lack of real-world labeled data. In this report, we introduce the total training framework achieving the generality of model performance on the real sound recordings through utilizing the emulations from the external data sources. Emulated sound data consist of the synthesis of the individual audio samples corresponding to the task-predefined classes, extracted from the external datasets [ 8 , 7 , 16 , 15 , 14 , 2 , 12 ] , along with spatial room impulse responses (SRIR) and spatial ambient noises (SNoise) [ 17 ] . Considering the difference between the size of real sound recordings dataset and of emulated sounds, the framework exploits the down-sized bootstrapping of emulations per every training epoch. The framework also employees the data augmentation techniques to train further generalized model. Additionally, aiming at the increment of robustness in model prediction, the clustering-based test time rotation-augmentation ensemble method of ACCDOA is designed in this work. The method allows the effective aggregation of the predictions from confident majorities whilst selective rejection of the abnormal candidates. 2 Proposed Method In this section, we describe a proposed framework for SELD, and it consists of feature extraction, data augmentation, external mix, network, and test time augmentation. The overall process of the framework is shown in Figure 1 . Figure 1: Proposed framework overview. 2.1 Features We use first-order Ambisonic (FOA) format signals, and multichannel log mel-spectrograms and FOA intensity vectors are used as frame-wise features. The parameters (sr,nfft,hop,window)=(24000,2048,600,1200) are set for Short-time Fourier transformation. 2.2 Data augmentation We adopt three types of augmentation including each wave and spectrogram augmentation for guaranteeing the generalized performance on the test set. Regarding wave augmentation on the audio inputs, we adjust the gain, shift the pitch, and apply the band-pass filter on the audio. Furthermore, we rotate the directions of events in a sound source as in [ 11 ] . The rotation function sets are pre-defined as 16 patterns to change azimuth and elevation angles of a sound source. The rotation function sets are the combinations of channel swapping and channel sign inversion on the X , Y , and Z axis of FOA inputs. Finally, we use spectrogram augmentation [ 13 ] , which maskes time and frequency information after log mel-spectrogram transformation on the audio inputs. 2.3 External mix To keep the model fit in real-world scenario contexts while taking advantage of various audio samples from external datasets, dataset mixing technique is adopted to consist model training dataset. The technique balances the size of each dataset on the model training phase, between the small real recording set and the large emulated scenarios. The balancing of two different datasets is conducted as follows: the half of the model training dataset consists of the real-world sound scenarios, where it is maintained to the end of the training phase. The other half comprises with the scenarios down sampled from the emulated sounds, and the sampling occurs dynamically at the start of the every training epoch. 2.4 Network architecture Figure 2 presents the process of our SELD architecture consisting of an encoder, a self-attention pooling, and a decoder used in this study. First, we adopt the squeeze-and-excitation residual networks [ 9 ] (SE-ResNet), which have recently been applied to audio classification [ 23 , 21 ] , as the SELD encoder. Our SE-ResNet-34 is the same original SE-ResNet-34, except we use average pooling instead of the stride convolution in the second and third blocks. In addition, to preserve the frame-wise localization information of the input audio, we aggregate frequency dimensions into channel dimensions using the variant of self-attention pooling [ 3 , 4 ] . For the SELD decoder, we employ two bidirectional GRU layers followed by layer normalization and tangent hyperbolic activation. Then, the SELD output is obtained by applying two fully-connected layers followed by tangent hyperbolic activation. Figure 2: Model architecture. 2.5 Test time augmentation To maximize the generalization ability of models in the real recording test phase, we adopt a test time augmentation (TTA). TTA is widely used in computer vision to increase the robustness and performance of models [ 5 , 10 , 20 ] . Although they proposed better augmentation policies and aggregation methods, they are not appropriate to apply on SELD due to the different predicted outputs. In the specific, the unknown number of events and the presence of coordinates information make challenging to apply TTA on SELD. To utilize TTA on SELD, we propose a clustering-based aggregation method to obtain confident predicted outputs and aggregate them. We take 16 pattern rotation augmentation for test time augmentation, making 16 predicted outputs, that is candidates. To obtain confident aggregated outputs, we use DBSCAN [ 6 ] for clustering candidates, only when the candidates have the same class and the number of them is over the threshold. The epsilon of DBSCAN is the distance calculated by the threshold of unification for inference in degrees. By exploiting DBSCAN, the outliers of the candidates can be excluded and it is possible to divide the events which have the same class but are differently located. After acquiring clusters of candidates, we take an average to aggregate them. When the number of the aggregated outputs is over three tracks, we select top-3 outputs depending on the weight which is calculated by multiplication between the number of candidates per cluster and their norm. Furthermore, we fit