Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss

Nian Shao, Xian Li, Xiaofei Li · 2026

Read Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss by Nian Shao, Xian Li, Xiaofei Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29901.txt.

Introduction

Shao, Li, and Li improve sound event detection, the task of spotting and timing sounds such as alarms or breaking glass in audio recordings. Labelled audio is scarce, so their system learns in a semi-supervised way from mostly unlabelled data.

The work builds on a framework that fine-tunes a pretrained audio model with pseudo-labels, guessed labels for unlabelled clips. The authors add an embedding-level contrastive loss, which teaches the model to place similar sounds close together in its internal space. Mixup, blending two clips, serves both methods differently, so they propose conditional mixup to reconcile the two uses. The result reaches 0.645 and 0.822 on the two PSDS scores on the DESED validation set, a new state of the art.

Catalog record

CREATORNian Shao, Xian Li, Xiaofei Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29901

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss Nian Shao, Xian Li, Xiaofei Li Abstract Sound event detection (SED) is a core module for acoustic environmental analysis, yet its performance is often limited by scarce labeled data. Recent systems leverage large pretrained audio foundation models, but effective fine-tuning remains challenging because labeled data are limited while unlabeled data are abundant. A previous work, ATST-SED, addressed this problem with a pseudo-label based semi-supervised fine-tuning framework. In this work, we further improve the framework by adopting an embedding-level self-supervised contrastive loss inspired by ATST-Frame pretraining. This contrastive objective better exploits unlabeled data during fine-tuning. One challenge is that mixup serves different roles in the two objectives: pseudo-label learning uses composition mixup, while contrastive learning treats mixup as a perturbation. To resolve this mismatch, we propose conditional mixup, which combines composition mixup and perturbation mixup in one semi-supervised framework and defines the corresponding embedding-level contrastive losses. The resulting model achieves 0.645 PSDS 1 and 0.822 PSDS 2 on the DESED validation set, establishing a new state of the art. Keywords— sound event detection, semi-supervised learning, contrastive learning, mixup, ATST. I Introduction Sound conveys rich information about the surrounding environment, but recognizing sound events remains challenging for machines [ 26 ] . Sound event detection (SED) aims to detect sound events and their temporal boundaries in polyphonic audio scenes. Since frame-level annotation is expensive, SED suffers from severe label scarcity. In practice, SED systems are trained with multiple annotation types. These include weakly labeled data, which provide clip-level event tags without temporal boundaries [ 24 ] ; strongly labeled synthetic and real data, which provide time-aligned event annotations [ 26 , 7 ] ; and unlabeled data [ 20 ] . This setup makes semi-supervised learning a natural choice for modern SED benchmarks. Prior work has improved semi-supervised SED with data augmentation, pseudo-label based learning, and stronger architectures for frame-level modeling [ 12 , 10 , 29 , 18 ] . Recent SED systems also benefit from pretrained self-supervised audio models, including BEATs, AST, and ATST-Frame [ 3 , 15 , 16 , 8 ] . Trained on large-scale audio corpora, these models provide stronger representations than task-specific training alone. In particular, ATST-Frame is trained with a frame-level self-supervised contrastive objective and learns frame-level audio representations that transfer well to temporally localized downstream tasks such as SED [ 16 ] . In prior ATST-SED work [ 23 ] , ATST-Frame was introduced into a CRNN-based detector, and a two-stage pseudo-label based semi-supervised framework was proposed for fine-tuning ATST-Frame for SED, providing a strong baseline. However, the semi-supervised objectives commonly used in SED are dominated by consistency regularization [ 12 , 21 , 9 ] . These objectives are effective for lightweight SED models, but can be much less effective for SED systems built on large pretrained self-supervised encoders [ 22 , 17 , 13 ] . A likely reason is that consistency regularization is too weak compared with the much stronger pretraining tasks. By contrast, self-supervised contrastive learning is a more challenging representation-learning objective and has been shown to improve the linear separability and transferability of learned representations during pretraining [ 16 , 4 ] . Among existing self-supervised contrastive frameworks, ATST-Frame is particularly suitable for SED because it is trained with a frame-level objective that matches the frame-level nature of SED. We therefore introduce its embedding-level self-supervised contrastive objective into semi-supervised fine-tuning, so that unlabeled data can be better leveraged than pseudo-label consistency alone. Pseudo-label learning and contrastive learning, however, benefit from different uses of mixup. In pseudo-label learning, mixup follows a composition view: the mixed sample should preserve the content of both source samples. In contrastive learning, mixup follows a perturbation view: the newly added sample should only perturb the input sample [ 19 , 14 , 16 ] . Both strategies are effective in their own settings. To exploit both of them within one framework, we propose conditional mixup, which combines composition mixup and perturbation mixup according to the interpolation coefficient. Based on the two mixup cases, we further define the corresponding pseudo-label losses and embedding-level contrastive losses, so that decision-level pseudo-label supervision and embedding-level contrastive supervision can complement each other during fine-tuning. On the DESED validation set, ATST-SEDv2 achieves 0.645 PSDS 1 and 0.822 PSDS 2 with cSEBB post-processing, establishing a new state of the art. The main contributions are as follows. First, we propose conditional mixup to combine composition mixup and perturbation mixup in one framework. Second, we design the corresponding pseudo-label and embedding-level contrastive losses, which provide an effective semi-supervised objective for both lightweight and pretraining-based SED models. Third, we achieve new state-of-the-art performance on the DESED validation set, reaching 0.645 PSDS 1 and 0.822 PSDS 2 . II Method Polyphonic SED is a multi-class detection task in which a model detects the onset and offset of multiple sound events in an audio clip. This work focuses on the DESED dataset [ 26 ] , which provides strongly labeled, weakly labeled, and unlabeled training data. To make use of both supervised and unsupervised data, the SED model is trained in a semi-supervised manner. In this work, we aim to strengthen semi-supervised fine-tuning for ATST-SED by adopting an embedding-level self-supervised contrastive objective inspired by ATST-Frame pretraining [ 16 ] . In this section, we first review the pseudo-label based semi-supervised fine-tuning baseline. We then describe the mixup strategies used in the baseline system and in contrastive learning, and introduce conditional mixup to unify them. Next, we describe the corresponding embedding-level self-supervised losses and summarize the overall training objective. II-A Pseudo-Label Based Two-Stage Fine-Tuning Baseline Fig. 1: Model architecture of the ATST-SED model, which consists of an ATST-SED encoder and a SED classifier. We use ATST-SED as the baseline model [ 23 ] . As shown in Fig. 1 , the model consists of an ATST-SED encoder and a nonlinear SED classifier [ 10 ] . The encoder consists of a convolutional recurrent neural network (CRNN) [ 10 ] and a pretrained ATST-Frame model [ 16 ] . The pretrained ATST-Frame model serves as an additional feature extractor alongside the 7-layer convolutional neural network (CNN) feature extractor. The features from the two branches are temporally aligned and fused, after which a recurrent neural network (RNN) performs contextual modeling before classification. The baseline fine-tuning framework is divided into two stages. Both stages are trained in a semi-supervised manner. In stage 1, the ATST-Frame model is kept frozen, and only the CRNN and SED classifier are optimized. The supervised binary cross-entropy (BCE) loss ( {\cal L}_{\text{BCE}} ) is applied to the labeled data. For unlabeled data, the Mean Teacher (MT) loss ( {\cal L}_{\text{MT}} ) [ 25 ] is used. Specifically, an exponential moving average (EMA) teacher generates pseudo labels for the unlabeled data, and the MT loss minimizes the mean-square error between the student predictions and these pseudo labels. The total loss of stage 1 is \displaystyle{\cal L}_{\text{stage1}}={\cal L}_{\text{BCE}}+r_{\text{MT}}{\cal L}_{\text{MT}}, (1) where r_{\text{MT}} is the ramp-up weight of {\cal L}_{\text{MT}} . In stage 2, all parameters are fine-tuned together. Since the amount of labeled data in DESED is still limited for tuning a large pretrained encoder, the baseline

Read the complete text