A Multi-Stage Separation-and-Classification Framework Guided by Complementary Acoustic-to-Semantic Clues
2026
Read A Multi-Stage Separation-and-Classification Framework Guided by Complementary Acoustic-to-Semantic Clues (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24512.txt.
Introduction
This paper builds a multi-stage separation-and-classification framework guided by complementary acoustic-to-semantic alignment. The building builds. Their framework is multi-stage separation-and-classification. The guiding guides complementary acoustic-to-semantic. It is audio AI. The building is a.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24512
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Multi-Stage Separation-and-Classification Framework Guided by Complementary Acoustic-to-Semantic Clues Abstract This report describes the system proposed for the DCASE 2026 Challenge Task 4: Spatial Semantic Segmentation of Sound Scenes (S5). Specifically, we develop a multi-stage framework in which each stage couples a separation model with a classification model. The first stage performs source separation and classification directly on the multi-channel mixture. Its outputs are then propagated to the following stage as two complementary clues that progressively refine each target estimate: (i) an enrollment clue, the separated waveform itself, serving as a low-level acoustic reference; and (ii) a class clue, the predicted label encoded as a one-hot vector. The third stage reuses the second-stage outputs under the same scheme, forming an iterative self-guided refinement process. In addition, we use a fine-grained frame-level audio embedding from an audio encoder pretrained on a large audio corpus as an additional clue to further improve the audio separation performance. On the test set, the proposed system achieves a CAPI-SDRi of 15.51 dB, a mixture accuracy of 71.09%, and a source accuracy of 78.62%; with an improvement of 7.02 dB, 10.38%p and 8.22%p compared with the challenge baseline, respectively. Index Terms — Audio Source Separation, Audio Classification, Duration-Based Augmentation, Temporal-FiLM 1 Introduction Spatial Semantic Segmentation of Sound Scenes (S5) aims to detect and separate individual sound events from a multi-channel mixture. Given a recording of several directional sources, interfering sounds, and diffuse background noise, an S5 system must both recognize target classes and recover an isolated waveform for each detected source. S5 systems can be applied in various applications, such as auditory scene analysis. Following DCASE 2025 Task 4 [ 14 ] , the S5 task continues in DCASE 2026 Task 4 [ 8 ] with two modifications that bring the benchmark closer to real acoustic conditions. First, a single mixture may now contain multiple sources of the same class simultaneously. For example, several people may speak at once. Second, a mixture may contain zero target events, so that the system must reliably decide when no target sound is present while still being exposed to background noise and interference. Both changes substantially increase the difficulty of the task and require systems to be robust to source-count uncertainty and label ambiguity. As a foundation for this task, we build upon a multi-stage self-guided framework initially developed for DeepASA [ 7 ] and DCASE 2025 Task 4 [ 6 ] . These baseline frameworks progressively refine separation and classification using internally generated clues. Especially, the framework proposed in [ 6 ] employs a Universal Sound Separation (USS) model, DeFT-Mamba-USS, to decompose multi-channel mixtures into distinct sources, followed by a Single-label Classification (SC) model, M2D-SC, to predict their labels. The separated waveforms and predicted classes are then fed into a Target Sound Extraction (TSE) model, DeFT-Mamba-TSE, as clues to form a self-guided, iterative refinement loop. The initial baseline [ 6 ] faced notable inefficiencies in its classification model (M2D-SC). Specifically, fine-tuning the classifier on a relatively small dataset degraded the diverse representational capacity of the pretrained model. Furthermore, converting the separated waveforms into magnitude-only mel-spectrograms caused the loss of phase and fine-grained frequency information. To overcome these limitations, a Dual-Path Classifier (DPC) was proposed [ 5 ] . The DPC directly utilizes the intermediate object features produced from the USS model, alongside semantic features extracted from a frozen pretrained M2D model. This approach preserves fine-grained frequency information that is lost during Mel-spectrogram projection. Concurrently, an SEC module [ 5 ] integrates semantic embeddings from pretrained models (e.g., M2D or MGA-CLAP) into DeFT-Mamba-TSE to enrich the class clues. These embeddings are projected to match the dimensions of the one-hot class embedding vector and are added element-wise before being injected into the DeFT-Mamba-TSE blocks via FiLM layers. Figure 1: Overall architecture of the proposed multi-stage self-guided framework. Stage 1 derives initial clues using DeFT-Mamba-USS and M2D-DPC. Stages 2 and 3 iteratively refine target sound extraction using DeFT-Mamba-TSE, guided by the separated waveforms, one-hot predicted classes, and fine-grained AF-Whisper semantic embeddings. Despite these advancements, applying the existing framework reveals two major limitations. First, the conventional SEC approach simply combines embeddings from one-hot class vectors and embeddings from a pretrained model through element-wise addition. This can dilute information contained in each embedding required for precise and rich target extraction. Furthermore, the embeddings from pretrained models have a coarse time resolution, failing to capture the fine-grained temporal dynamics. Second, we observe a classification performance degradation in percussive classes (e.g., percussion), which are frequently misclassified as silence. Our assumption is that this degradation is caused by the prevalence of extremely short, transient samples in the training dataset, which provide insufficient temporal context for reliable detection. In this report, we propose two methods to overcome these limitations: • Fine-grained Semantic Conditioning : We utilize pretrained AF-Whisper [ 4 ] to obtain embeddings with a 20 ms time resolution to capture dense, frame-level temporal dynamics; and inject this embedding into the backbone through an independent Temporal Feature-wise Linear Modulation (Temporal-FiLM) layer [ 1 ] to prevent information dilution. • Duration-based Augmentation We introduce an augmentation strategy for difficult percussive classes by mixing short transient samples with longer sustained samples during training, thereby reducing the false-silence misclassification rate. • Class-specific Silence Threshold Optimization To efficiently handle the silence prediction, we conduct a class-specific threshold tuning process to maximize the official metric, CAPI-SDRi. In the official test set, our system achieves a CAPI-SDRi of 15.51 dB, a mixture accuracy of 71.09%, and a source accuracy of 78.62%, significantly outperforming the baseline by 7.02 dB, 10.37%p, and 8.23%p, respectively. 2 Proposed Method 2.1 Framework Overview Figure 1 illustrates our proposed three-stage self-guided framework for joint separation and classification. In Stage 1 (Clue Derivation), DeFT-Mamba-USS decomposes the multi-channel mixture into distinct object features, which are directly processed by the DPC to predict class labels and detect silence. In Stage 2 (Guided Extraction) and Stage 3 (Iterative Refinement), DeFT-Mamba-TSE progressively refines the separated waveforms, followed by enhanced classification performance. Each refinement stage is guided by three complementary clues derived from the preceding stage: the separated waveform (enrollment clue), the predicted one-hot label (class clue), and a fine-grained semantic embedding extracted by AF-Whisper. The enrollment clues are concatenated with the multi-channel mixture along the channel dimension, and the class clues are injected through the FiLM [ 10 ] layers interleaved between the separation blocks. 2.2 AudioFlamingo-whisper Embedding Although the features extracted by the pre-trained M2D model encode rich semantic information, they are derived from patchified representations with a temporal downsampling factor of 16, leading to relatively coarse temporal resolution and potentially limiting fine-grained temporal modeling. Therefore, we employ a fine-grained frame-level embedding from the AudioFlamingo (AF)-Whisper encoder [ 4 ] , pretrained on a massive audio corpus. As shown in Figure 2 , this embedding is injected through additional Temporal-FiLM layers placed after each FiLM layer handling the one-hot vectors. This injection operates at a 20 ms time resolution to capture frame-level temporal dynamics. Let c_{oh} denote the embedding from the one-hot class vector predicted by the DPC, and E_{AF} represent the sequence of semantic audio embeddings extracted by AF-Whisper. To preserve definitive categorical information, c_{oh} is injected into the intermediate feature maps