Self-Guided Target Sound Extraction and Classification Through Universal Sound Separation Model and Multiple Clues
2025
Read Self-Guided Target Sound Extraction and Classification Through Universal Sound Separation Model and Multiple Clues (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13741.txt.
Introduction
Younghoo Kwon and 3 co-authors present a self-guided system for target sound extraction and sorting in a 2025 paper. Their setting is the spatial semantic segmentation of sound scenes task in a 2025 audio challenge. Separating a complex mixture into single sounds and labeling them usually needs external guidance specifying what to extract.
The system joins models for three tasks: universal sound separation, single-label sorting, and target sound extraction. Separation first breaks a mixture into source waveforms. Each waveform is processed by sorting, generating the waveform and its class label as inputs for extraction, which isolates the source that matches. Because inputs are produced within the system, the extraction target is found autonomously, without external guidance. The extracted waveform can loop back into sorting, creating iterative refinement.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Guided Target Sound Extraction and Classification Through Universal Sound Separation Model and Multiple Clues Abstract This paper introduces a multi-stage self-directed framework designed to address the spatial semantic segmentation of sound scene (S5) task in the DCASE 2025 Task 4 challenge. This framework integrates models focused on three distinct tasks: Universal Sound Separation (USS), Single-label Classification (SC), and Target Sound Extraction (TSE). Initially, USS breaks down a complex audio mixture into separate source waveforms. Each of these separated waveforms is then processed by a SC block, generating two critical pieces of information: the waveform itself and its corresponding class label. These serve as inputs for the TSE stage, which isolates the source that matches this information. Since these inputs are produced within the system, the extraction target is identified autonomously, removing the necessity for external guidance. The extracted waveform can be looped back into the classification task, creating a cycle of iterative refinement that progressively enhances both separability and labeling accuracy. We thus call our framework a multi-stage self-guided system due to these self-contained characteristics. On the official evaluation dataset, the proposed system achieves an 11.00 dB increase in class-aware signal-to-distortion ratio improvement (CA-SDRi) and a 55.8% accuracy in label prediction, outperforming the ResUNetK baseline by 4.4 dB and 4.3%, respectively, and achieving first place among all submissions. \NoHyper 1 Introduction The DCASE 2025 Task 4 [ 1 ] , Spatial Semantic Segmentation of Sound Scenes (S5), aims to detect and separate individual sound events from multi-channel spatial audio inputs. The core objective is to isolate target sound events (foreground sources) from a mixture by distinguishing them from non-target sound events (interference sources) and background noise, and to perform classification. Each audio mixture can contain up to three foreground sources, optionally mixed with interference sources and background noise. Interference sources are differentiated from background noise by their non-diffuse spatial characteristics and their association with specific sound classes. The task defines a set of 18 target classes, while the non-target events encompass a broader set of 94 classes. The presence of various sound events, including both non-target events and background noise, complicates the application of Universal Sound Separation (USS) techniques. This complexity highlights the importance of Target Sound Extraction (TSE), a method that leverages specific clues to isolate the sound of interest. TSE models leverage various forms of clues, such as a class label [ 2 ] , an enrollment sample [ 3 , 4 ] , or a timestamp [ 5 ] , to isolate a desired waveform from a mixture. For instance, a class-conditioned TSE model [ 2 ] extracts sound events belonging to a specific class. The SoundBeam [ 3 ] using the enrollment clue can be viewed as an extension of this paradigm, where an enrollment waveform is provided as the clue. The model then extracts sounds from the mixture that match the acoustic characteristics of the enrollment waveform. Traditional TSE tasks depend on external cues that are separate from the input mixture. Conversely, in the S5 task, the system must initially detect the target sound events present in the mixture before extracting them. This indicates a self-directed approach: the necessary clues must be extracted from the input mixture itself to guide the retrieval of the target sound events. Fig. 1: Self-guided multi-stage framework The official DCASE 2025 Task 4 baseline [ 6 ] addresses the S5 task by combining Audio Tagging (AT) and TSE. It first performs AT on the mixture using Masked Modeling Duo for Audio Tagging (M2D-AT), a variant of M2D [ 4 ] specifically fine-tuned for this task. The resulting multi-hot label, representing the predicted classes, is then fed as a clue to a TSE model (ResUNet or ResUNetK) [ 7 ] to extract the target events. This approach, however, presents three notable limitations. First, performing AT directly on a complex polyphonic mixture is inherently more difficult than classifying an already isolated sound. Given that mixtures can contain up to six sources, including background noise, accurately identifying all target events is a formidable task. Second, the baseline’s AT module cannot leverage the spatial information available in the multi-channel input, which can be critical for disambiguating overlapping sources. Third, the framework lacks a mechanism for iterative refinement; the information flow is unidirectional, and the extracted waveforms are not used to improve the initial predictions. To overcome these drawbacks, we propose a multi-stage self-guided framework that combines the multi-clue derivation through USS, Single-Label classification (SC), and TSE. The core of our framework leverages a modified version of DeFT-Mamba [ 8 ] , a state-of-the-art (SOTA) model developed for universal audio separation in multi-channel polyphonic scenarios. This model, which we term DeFT-Mamba-USS, first performs USS to decompose the mixture into estimates for three foreground sources, two interference sources, and background noise. Subsequently, we perform SC on each separated target waveform using Masked Modeling Duo for Single-label Classification (M2D-SC), a version of M2D fine-tuned for classifying the 18 target classes. In the next step, both the separated waveforms (as enrollment clues) and their predicted classes (as class clues) are jointly supplied to DeFT-Mamba-TSE, a variant of DeFT-Mamba adapted for TSE. This multi-clue approach guides the extraction of refined target waveforms. Finally, this process is iterated: the newly extracted waveform is re-classified, and these refined enrollment and class clues are used for another round of TSE, creating a cycle of progressive refinement. This framework significantly lowers the difficulty of class estimation compared to direct AT on a mixture by performing classification on preliminarily separated sources. The multi-stage refinement process enables the system to achieve progressively more accurate waveforms and corresponding class labels, leading to superior classification accuracy on the evaluation dataset compared to other teams. This approach, which integrates multi-clue injection with iterative refinement, demonstrates great effectiveness, enabling us to achieve the highest position in the competition. Our leading standing has been shown through the class-aware signal-to-distortion ratio improvement (CA-SDRi), a comprehensive metric assessing both separation and classification performance. 2 Preliminary DeFT-Mamba [ 8 ] is a SOTA model designed to perform USS and classification concurrently. Operating on the complex spectrogram, the model’s architecture is composed of N_{b} stacked blocks of F-Hybrid Mamba and T-Hybrid Mamba modules, which are designed to model relationships along the frequency and time dimensions, respectively. A key distinction from previous speech enhancement models with similar architectures [ 9 , 10 , 11 , 12 ] is its replacement of the traditional Feed-Forward Network (FFN) within each transformer block with a Mamba Feed-Forward Network (Mamba-FFN). The model performs separation at the feature level. These separated features are then fed into two parallel decoders: an audio decoder to estimate waveforms and a class decoder to predict their corresponding labels. This dual-head decoder structure effectively resolves the pair-wise ambiguity between the estimated waveforms and their predicted classes, ensuring each separated sound is correctly associated with its label. 3 Proposed self-guided framework The self-guided multi-stage framework performs progressive separation and classification through a combination of USS, SC, and TSE. As illustrated in Fig. 1 , our framework consists of three main stages: 3.1 clue generation via DeFT-Mamba-USS and M2D-SC, 3.2 TSE guided by multi-clue conditioning, and 3.3 iterative refinement for enhanced separation and classification. 3.1 Stage 1: