Leveraging LLM and Text-Queried Separation for Noise-Robust Sound Event Detection

Han Yin 1 , Yang Xiao 2 , Jisheng Bai 1 , Rohan Kumar Das 2 · 2024

Read Leveraging LLM and Text-Queried Separation for Noise-Robust Sound Event Detection by Han Yin 1 , Yang Xiao 2 , Jisheng Bai 1 , Rohan Kumar Das 2 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01174.txt.

Description

Abstract Sound Event Detection (SED) is challenging in noisy environments where overlapping sounds obscure target events. Language-queried audio source separation (LASS) aims to isolate the target sound events from a noisy clip. However, this approach can fail when the exact target sound is unknown, particularly in noisy test sets, leading to reduced performance.

Catalog record

CREATORHan Yin 1 , Yang Xiao 2 , Jisheng Bai 1 , Rohan Kumar Das 2
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging LLM and Text-Queried Separation for Noise-Robust Sound Event Detection Han Yin 1 , Yang Xiao 2 , Jisheng Bai 1 , Rohan Kumar Das 2 Abstract Sound Event Detection (SED) is challenging in noisy environments where overlapping sounds obscure target events. Language-queried audio source separation (LASS) aims to isolate the target sound events from a noisy clip. However, this approach can fail when the exact target sound is unknown, particularly in noisy test sets, leading to reduced performance. To address this issue, we leverage the capabilities of large language models (LLMs) to analyze and summarize acoustic data. By using LLMs to identify and select specific noise types, we implement a noise augmentation method for noise-robust fine-tuning. The fine-tuned model is applied to predict clip-wise event predictions as text queries for the LASS model. Our studies demonstrate that the proposed method improves SED performance in noisy environments. This work represents an early application of LLMs in noise-robust SED and suggests a promising direction for handling overlapping events in SED. Codes and pretrained models are available at https://github.com/apple-yinhan/Noise-robust-SED . I Introduction Sound event detection (SED) [ 1 ] involves identifying sound event classes and corresponding timestamps, which has been widely applied in various applications such as smart city [ 2 , 3 ] and medical monitoring [ 4 ] . Recent deep learning advancements have significantly enhanced SED, supported by diverse datasets designed for specific scenarios. One such widely used dataset is domestic environment sound event detection (DESED) [ 5 ] , which is also considered for Task 4 of the detection and classification of acoustic scenes and events (DCASE) 1 1 1 https://dcase.community/challenge2024/ challenge series. Convolutional recurrent neural network (CRNN)-based models [ 6 , 7 , 8 , 9 ] have performed well on the DESED dataset in recent years. However, these studies typically consider testing data that closely matches the training data conditions, limiting the models’ ability to generalize to real-world conditions [ 10 ] . In noisy environments, audio may include sound events that were not be present during training, which can act as background noise, overlap with target sounds, and make detection more difficult, leading to reduced performance. As a result, developing noise-robust SED system remains a challenging problem. The pursuit of developing noise-robust SED has led to different methods for improving the performance in noisy conditions. One common approach is to fine-tune models using noisy data, which involves selecting noise types that represent real-world environments while being distinct from target sounds [ 11 , 12 , 13 , 14 , 15 ] . However, artificial selection can be difficult due to bias and unnatural correlations. Recently, large language models (LLMs) like GPT-4 [ 16 ] , GPT-3 [ 17 ] , and Llama [ 18 ] have shown great capabilities in analyzing the semantic relationships between different sound types. An LLM-powered dataset referred to as WildDESED was proposed in [ 19 ] , where SED models are trained with a noisy training set that included the same noise events as in the test sets. However, training of SED models in this way, may not be very ideal for real-world applications, as predicting the types of noise events during testing is often not feasible . To avoid the issue of predicting noise types during testing, we propose to use audio source separation to extract target sounds from noisy audio during testing. This method does not rely on knowing the specific noise events beforehand. Instead, it isolates the target sound directly from the audio mixture, making it adaptable to various noisy conditions. Language-queried audio source separation (LASS) models allow them to isolate audio signals that match the text query by pre-training on large audio-text paired datasets [ 20 ] . However, the text query is crucial, as it directly impacts the separation result. These models can only isolate sound events present in the audio. For example, if “dog barking” is requested in a clip containing only speech, the output may be unreliable. In noisy environments, the actual sound events are unknown, and therefore it is difficult to generate accurate text queries for separation. To address this, we propose using LLMs to develop a noise-robust model that can generate reliable event predictions for text queries. In this study, to address the challenge of noisy sound events in SED, we propose an LLM-based framework that integrates noise augmentation with the text-queried audio separation method for noise-robust SED. Our proposed approach first leverages LLMs to select appropriate noise types and fine-tune the SED model. The fine-tuned model then generates clip-wise event predictions, which serve as text queries for the LASS model during testing. Experimental studies in this work are conducted on DESED and WildDESED datasets to demonstrate the effectiveness of the proposed method to develop noise-robust SED system in challenging acoustic conditions. We introduce four different baseline methods for generating text queries to validate the effectiveness of our LLM-based approach. To the best of our knowledge it is the first study to apply LASS models for noise-robust SED. We have released the code to support future research. Fig. 1: The overview of LLM-based noise augmentation and text-queried separation for noise-robust sound event detection. II Proposed Method for Noise-robust SED As shown in Fig. 1 , the proposed noise-robust SED method can be decomposed into three stages: training, fine-tuning, and testing. Details are described as follows. II-A Training During the training process, similar to conventional SED training approaches [ 1 , 6 ] , we directly utilize an SED model to extract information from the input audio spectrogram \bm{X}\in\mathbb{R}^{T\times F} , where T and F represent the number of frames and frequency bins, respectively. (\bm{\tilde{Y}}_{\textrm{s}},\bm{\tilde{Y}}_{\textrm{w}})=g_{\theta}(\bm{X}) (1) where \bm{\tilde{Y}}_{\textrm{s}}\in\mathbb{R}^{T\times N} and \bm{\tilde{Y}}_{\textrm{w}}\in\mathbb{R}^{N} are frame-wise and clip-wise predictions. N is the number of target event classes. g(\cdot) is the SED model, and \theta is the training parameter. Then, the binary cross entropy (BCE) loss between outputs and labels is calculated, which is formulated as: Loss_{\textrm{train}}=l_{\textrm{BCE}}(\bm{\tilde{Y}}_{\textrm{s}},\bm{Y}_{\textrm{s}})+l_{\textrm{BCE}}(\bm{\tilde{Y}}_{\textrm{w}},\bm{Y}_{\textrm{w}}) (2) where \bm{Y}_{\textrm{s}}\in\mathbb{R}^{T\times N} and \bm{Y}_{\textrm{w}}\in\mathbb{R}^{N} are strong and weak labels. II-B Fine-tuning with LLM-based noise augmentation Denote the training data of the training stage as \mathcal{U}_{\theta} , which mainly contains N event classes. However, in real-life scenarios, sound events may present a more complex and variable distribution, denoted as \mathcal{U}_{\alpha} , with M event classes ( M\textgreater N ). Because of the gap between \mathcal{U}_{\alpha} and \mathcal{U}_{\theta} , the model trained on \mathcal{U}_{\theta} (i.e. model g_{\theta} ) may perform poorly on \mathcal{U}_{\alpha} . To improve the performance of g_{\theta} on \mathcal{U}_{\alpha} , we propose to use an LLM-based noise augmentation method to fine-tune g_{\theta} . As shown in Fig. 1 , during fine-tuning, the LLM is used to select noise events from the noise candidate dataset. We denote the distribution of the noise dataset as \mathcal{U}_{\gamma} , along with K event classes ( K\textgreater N ). Compared to \mathcal{U}_{\theta} , \mathcal{U}_{\gamma} should present a more complex event distribution. The selected noise signal is added to the input audio according to a specific signal-to-noise ratio (SNR). An important question is: how to set the input prompts of the LLM for selecting the noise? Inspired by WildDESED, we set prompts based on event classes of \mathcal{U}_{\gamma} and \mathcal{U}_{\theta} . Denote

Read the complete text