Improving Audio Classification by Transitioning from Zero- to Few-Shot

James, Wolfgang · 2025

Read Improving Audio Classification by Transitioning from Zero- to Few-Shot by James, Wolfgang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20036.txt.

Introduction

James Taylor and Wolfgang Mack replace guessy text prompts with a handful of real examples to sort sounds better. Zero-shot audio classifiers match a sound's embedding against text descriptions of each class, and the best wording for a varied class is anyone's guess. Their few-shot approach groups actual audio embeddings by class and builds the class anchor from sound itself. The noisy text anchor retires.

Across their tests the example-based version usually beat the zero-shot baseline. The gain matters most where a class spans many unlike sounds. A few labeled clips, the work argues, are worth a lot of prompt craft.

Catalog record

CREATORJames, Wolfgang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20036

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Audio Classification by Transitioning from Zero- to Few-Shot James, Wolfgang Abstract State-of-the-art audio classification often employs a zero-shot approach, which involves comparing audio embeddings with embeddings from text describing the respective audio class. These embeddings are usually generated by neural networks trained through contrastive learning to align audio and text representations. Identifying the optimal text description for an audio class is challenging, particularly when the class comprises a wide variety of sounds. This paper examines few-shot methods designed to improve classification accuracy beyond the zero-shot approach. Specifically, audio embeddings are grouped by class and processed to replace the inherently noisy text embeddings. Our results demonstrate that few-shot classification typically outperforms the zero-shot baseline. Taylor Mack Cisco Systems, Inc.Scotland Cisco Systems, Inc.Germany keywords dimensionality reduction, few-shot, audio classification, noise classification, audio embeddings † † email: [email protected], [email protected] 1 Introduction Identification and classification of sound events play a vital role in understanding acoustic environments. Competitions like DCASE [ 1 ] and accompanying datasets like TUT [ 2 ] and CochlScene [ 3 ] showcase the importance and interest in acoustic scene classification and event detection. Practitioners working on audio machine learning rely on such tools for crucial downstream tasks, such as understanding urban environments and acoustic surveillance in smart cities [ 4 , 5 ] , fall alerts in care homes [ 6 ] , monitoring for conservation and ecological research [ 7 ] , and speech enhancement [ 8 ] . Classically, sound event classification is performed using neural networks that map to class activity, which require extensive labeled datasets for each class present in the system [ 9 , 10 ] . These models, which need retraining whenever a new class is introduced, are inherently limited in adaptability. In dynamic environments where new events frequently emerge, such constraints become increasingly apparent. To address these challenges, few-shot learning approaches have explored how to classify with minimal labeled examples, often leveraging classical machine learning techniques to reduce data dependence and increase generalization. While [ 11 ] explores the power of pre-training for a specific domain, [ 12 , 13 ] demonstrate how classes can be added incrementally and continually. In [ 14 , 15 ] , the authors explore how few-shot approaches perform in multi-label datasets. Recent exploration into lightweight models achieve robust performance with limited labeled data, providing a foundation for more adaptable systems. Alternatively, inspired by the multi-modality of Contrastive Language-Image Pretraining (CLIP) [ 16 ] , early attempts to constrastively model text and audio [ 17 , 18 ] modeled audio and its labels in the same space. Further models such as CLAP showed that by leveraging mass-amounts of audio and natural language descriptions, contrastive models can be successfully used to classify, retrieve, and even caption audio [ 19 , 20 , 21 , 22 ] . Bringing the robustness of language modeling to the problem allowed sound event classification to be approached as a zero-shot problem. These models, while successfully modeling audio and their descriptors together, still show room to improve in terms of understanding sound sequences, sound concurrency, and disambiguating foreground and background sounds [ 23 ] . In the latest rendition of audio-language models, Pengi framed all problems as text generation problems and has allowed for enhanced captioning, question answering, source-distance discrimination, sentiment/emotion recognition, and music analysis [ 24 ] . Meanwhile, Audio-Flamingo shows augmenting large language models to understand audio has added-on the ability to count sound occurrences, describe the ordering of sounds, and assess sound quality [ 25 ] . While these models have become increasingly flexible and impressive, they inherently rely on text embeddings. These text embeddings are general purpose, but also introduce variability, sometimes leaving practitioners to try and find the ‘best’ prompt. In an effort to de-noise these embeddings from other tasks and make them most applicable to such a scenario, we explore light-weight approaches to few-shot, closed-set audio classification. Our approach side-steps the use of these text embeddings by defining classes with a few audio samples instead. In this way, we harness the power of pre-trained audio encoders and remain adaptable to new classes. Our contribution is a practical and effective extension of modern zero-shot, text-based audio classification tools to a few-shot setting using audio embeddings, showing consistent improvements with minimal data. Another contribution is a rigorous evaluation of the proposed method and baselines. Our empirical results demonstrate that this approach significantly improves on existing methods, offering a more adaptable solution for audio classification in dynamic environments. The paper is structured as follows. In Section 2, we introduce a signal model and embedding-based zero-shot audio classification. The proposed method is presented in Section 3. Section 4 contains information about the hyperparameters and the data we used. Finally, we evaluate the performance in Section 5 followed by a brief conclusion in Section 6. 2 Fundamentals We consider a dataset \mathcal{X}=\{x_{n}\mid n=1,\ldots,N\} , consisting of N discrete audio signals x_{n}\in\mathbb{R}^{T_{n}} , where T_{n}\in\mathbb{N} specifies the number of samples in each signal. Correspondingly, we define a set of labels \mathcal{Y}=\{y_{n}\mid n=1,\dots,N\} , with y_{n}\in\mathcal{C} representing the class label of the n -th audio recording, and \mathcal{C} is the set of possible classes. To handle the variability in audio lengths and to extract meaningful representations, we employ an embedding-extractor model \mathcal{F}_{A}\colon\mathbb{R}^{T_{n}}\to\mathbb{R}^{L} . This model maps each audio recording x_{n} to a fixed-dimensional embedding e_{n}\in\mathbb{R}^{L} of length L\in\mathbb{N} , defined as e_{n}=\mathcal{F}_{A}(x_{n}), (1) resulting in the set of embeddings \mathcal{E_{A}}=\{e_{n}\mid n=1,\ldots,N\} . Similarly, we define a set of text embeddings E_{T} using an embedding-extractor \mathcal{F}_{T} , which processes text akin to \mathcal{F}_{A} . The training of \mathcal{F}_{A} and \mathcal{F}_{T} involves reducing the distance between paired audio-text embeddings in a contrastive manner. During inference, each audio class is described textually and transformed via \mathcal{F}_{T} to yield a reference embedding e_{c}\in\mathbb{R}^{L} for class c . Audio samples are classified by transforming them using \mathcal{F}_{A} and mapping them to the closest text embedding in a zero-shot manner [ 22 ] . The challenge lies in selecting text that yields the ‘optimal’ embedding for classifying a specific audio class, due to the complexity and variability of language. Different texts may describe the same audio class in various ways, making it difficult to determine which description will best facilitate classification. 3 Few-Shot Audio Classification Methods Table 1: Top shows zero-shot baselines, bottom shows FS methods. For ESC-50 and FSD50K, results are obtained from the respective papers. For BBL, results are obtained using open-source implementations. For methods that use feature selection via MI , we use the best K for a given method/dataset pair (see Figure 3 ). Model (ZS) BBL (dev/eval, acc) ESC-50 (5-fold, acc) FSD50K (dev/eval, mAP) CLAP22 0.602 0.826 0.302 PENGI 0.547 0.920 0.468 CLAP23 0.623 0.948 (FT: 0.983 ) 0.485 Model (FS) |\mathcal{E}^{c}| =10 |\mathcal{E}^{c}| =20 |\mathcal{E}^{c}| =50 |\mathcal{E}^{c}| =20 |\mathcal{E}^{c}| =10 |\mathcal{E}^{c}| =20 |\mathcal{E}^{c}| =50 \text{FS}^{|\mathcal{E}^{c}|}_{\text{AVG}} 0.678 0.700 0.716 0.970 0.540 0.561 0.579 \text{FS}^{|\mathcal{E}^{c}|}_{\text{LDA}} 0.615 0.623 0.632 0.968 0.440 0.487 0.526 \text{FS}^{|\mathcal{E}^{c}|}_{\text{MI+AVG}} 0.681 0.698 0.716 0.971 0.559 0.576 0.586 \text{FS}^{|\mathcal{E}^{c}|}_{\text{MI+LDA}} 0.563 0.612 0.650 0.969 0.509 0.551 0.559 In many practical scenarios, a small number of audio samples for all classes already exists or can be gathered with minimal effort. Extracting audio embeddings from such a

Read the complete text