self-supervised learning method using multiple sampling strategies for general-purpose audio representation

2025

Read self-supervised learning method using multiple sampling strategies for general-purpose audio representation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18984.txt.

Introduction

Ibuki Kuroyanagi and Tatsuya Komatsu teach a model to represent all kinds of sound without labels, in a 2025 paper. Self-supervised learning lets models learn from raw audio alone. Classic approaches sample whole clips and contrast them, which suits tagging a clip but misses fine timing. The team adds two more sampling views to sharpen the learning.

One view works at the frame level, catching brief events inside a clip. The other is shaped for specific tasks such as pitch finding. Training used part of a large public audio set, and the learned features were then frozen for tests. Clip sorting improved by a quarter, sound event spotting by a fifth, and pitch finding by a smaller margin. The method aims at one sound model for many jobs.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18984

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

self-supervised learning method using multiple sampling strategies for general-purpose audio representation Abstract We propose a self-supervised learning method using multiple sampling strategies to obtain general-purpose audio representation. Multiple sampling strategies are used in the proposed method to construct contrastive losses from different perspectives and learn representations based on them. In this study, in addition to the widely used clip-level sampling strategy, we introduce two new strategies, a frame-level strategy and a task-specific strategy. The proposed multiple strategies improve the performance of frame-level classification and other tasks like pitch detection, which are not the focus of the conventional single clip-level sampling strategy. We pre-trained the method on a subset of Audioset and applied it to a downstream task with frozen weights. The proposed method improved clip classification, sound event detection, and pitch detection performance by 25 %, 20 %, and 3.6 %. amsmath, amssymb 1 Introduction Various sound-based applications have been studied, such as speech recognition, speaker identification, and command recognition as speech-related tasks, sound event detection (SED), pitch detection (PD), and instrument estimation as non-speech-related tasks. These tasks have achieved high performance with the development of neural networks [ 1 , 2 , 3 , 4 , 5 ] . However, these methods require much effort because they require collecting training data and annotating teacher labels for each task. For audio tagging tasks, fine-tuning with pre-trained models like PANNs [ 6 ] , which are obtained by supervised training with a large dataset such as Audioset [ 7 ] , have shown their effectiveness for other audio tagging datasets with small dataset size. However, their applicability is limited to tagging tasks because the models are also trained via supervised training for audio tagging. Task-specific large datasets are required for obtaining pre-training models for other tasks. There is a need to establish training methods for general pre-trained models, which can be trained with unlabeled datasets and applied to various tasks. Self-supervised learning is gaining attention to acquire useful representations for various tasks using unlabeled data. There are GPT [ 8 ] and BERT [ 9 ] in natural language processing, CPC [ 10 ] , SIMCLR [ 11 ] , MOCO [ 12 ] in the field of computer vision, and CURL [ 13 ] in the field of reinforcement learning. These methods have achieved high performance in their respective fields. In the field of audio, CPC-based methods design unsupervised loss functions based on regression tasks of representations [ 14 , 15 ] , and wav2vec2.0 solves a contrast task defined based on the quantization of the learned representation [ 16 ] . However, many methods have achieved high performance in speech-related tasks, and few studies have focused on non-speech-related tasks. TRILL [ 17 ] and [ 18 ] are examples of research to obtain general-purpose audio representations for both speech and non-speech-related tasks. These studies sampled anchors, positives, and negatives from unlabeled data. They used a metric learning method with triplet loss to minimize / maximize the distance between the anchor and positive / negative pairs. More recently, COLA [ 19 ] achieved higher performance, which is based on self-supervised learning with contrastive loss. COLA samples positive pair as segments from the same audio clip and negative pair as segments from different clips, and maximize / minimize the similarity between positive / negative pairs based on multiclass cross-entropy. COLA has been evaluated on multiple tasks, including speech, music, acoustic scenes, and animal sounds, and outperformed the performance of TRILL. Figure 1: Overview of proposed method. The key of training with contrastive loss is the sampling strategy, i.e., the definition of anchor, positive, and negative samples. COLA assumes that data from the same audio clip is similar in time, so a pair of segments sampled from the same audio clip is a positive sample. Different audio clips are assumed to have different characteristics from an anchor and are negative samples. The sampling strategy is reasonable to the wide range of audio tasks like audio-tagging and speaker identification because they are clip-level classification. The segments in clips should be the same classes in these tasks. However, in some tasks, the samples do not necessarily belong to the same class even when they are highly similar in time. For example, SED requires frame-level classification of events; different segments in the same audio clip can not be the same class. Also, it is necessary to identify the time changes from the same source in PD. COLA is also unsuitable for the task because it maximizes the similarity between the pair of segments in the same clip, even when the pitch of the sound is different. So that we need to design the strategy carefully for each task, and it is difficult to learn general-purpose representations with a single sampling strategy. We propose a self-supervised learning method using multiple sampling strategies for general-purpose audio representation. The proposed method performs self-supervised learning with multiple contrastive losses based on multiple strategies. By using multiple complementary strategies, a single model can be used for various tasks that the single conventional strategy cannot cover. In this study, we introduce two new strategies, frame-level strategy and task-specific strategy, in addition to the clip-level sampling strategy of COLA, to improve the performance of frame-level classification and identify the time variation from the same source. 2 Method In this study, we perform self-supervised learning using three sampling strategies shown in Fig. 1 . The proposed method is a multi-task learning method with three contrastive losses computed based on three different strategies. The first strategy is a clip-level strategy that focuses on differences between audio clips. The second strategy is a frame-level strategy that focuses on time changes in audio segments. The third strategy is a task-specific strategy that focuses on spectrum changes in a sound source. In the following, we discuss each strategy. 2.1 Clip-level strategy The first strategy is based on the clip-level sampling strategy used in COLA. The strategy defines a positive sample as audio segments from the same clip as an anchor segment, and negative segments are sampled from different audio clips. COLA calculates the similarity between speech segments in two steps. First, encoder f maps the acoustic feature data x_{1:T}\in\mathbb{R}^{N\times T} into embedding vectors h=f(x_{1:T})\in\mathbb{R}^{d\times T^{\prime}} , where N , T and T^{\prime} are the number of frequency bins, time frames and the frame length of h , respectively. Then, after global average pooling along with temporal frames, the shallow neural network g_{clip} maps h to the clip representations z_{clip}=g_{clip}(h)\in\mathbb{R}^{d} , where bilinear comparisons are performed. The similarity of the representations of two segments (x,x^{\prime}) can be expressed: \begin{split}s_{clip}(x,x^{\prime})=g_{clip}(f(x_{1:T}))^{T}Wg_{clip}(f(x^{\prime}_{1:T})),\end{split} (1) where W is the bilinear parameter, x^{\prime} is the feature value of the segment that is different from x . As an objective function, we rely on multiclass cross-entropy applied to the similarities: \begin{split}\mathcal{L}_{clip}=-{\rm log}{\frac{{\rm exp}(s_{clip}(x,x^{+}))}{\sum{{\rm exp}(s_{clip}(x,x^{-}))}}},\\ \scriptstyle x^{-}\in\mathcal{X}^{-}(x)\cup\{x^{+}\}\end{split} (2) where x^{+} is the positive associated to anchor x , \mathcal{X}^{-}(x) refers to the set of negative distractors. The embedding vectors h obtained from the loss are effective for clip-level classification. However, the embedding vectors are unsuitable for frame-level classification of events and identifying the time variation of the sound emitted from the same source. 2.2 Frame-level strategy

Read the complete text