Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors
2025
Read Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06710.txt.
Introduction
Vincent Kather, Burooj Ghani, and Dan Stowell compare the ears of fifteen bioacoustic models by looking at how they organize sound, not just how they classify it. Each model turns animal sounds into embedding vectors, which a classifier then labels. Accuracy scores alone cannot compare models trained on entirely different animal groups, nor judge species never seen in training.
The authors evaluate the embedding spaces directly, through clustering structure and nearest-neighbor classification. That isolates the feature extractor from its classifier head. The question is which extractors build spaces that adapt and generalize beyond their training classes. Such evidence matters for conservation work, where the target species is often rare or unrecorded in training data.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors Abstract In computational bioacoustics, deep learning models are composed of feature extractors and classifiers. The feature extractors generate vector representations of the input sound segments, called embeddings, which can be input to a classifier. While benchmarking of classification scores provides insights into specific performance statistics, it is limited to species that are included in the models’ training data. Furthermore, it makes it impossible to compare models trained on very different taxonomic groups. This paper aims to address this gap by analyzing the embeddings generated by the feature extractors of 15 bioacoustic models spanning a wide range of setups (model architectures, training data, training paradigms). We evaluated and compared different ways in which models structure embedding spaces through clustering and kNN classification, which allows us to focus our comparison on feature extractors independent of their classifiers. We believe that this approach lets us evaluate the adaptability and generalization potential of models going beyond the classes they were trained on. Keywords: deep learning, bioacoustics, embeddings 1 Introduction Human-driven climate change and deforestation have caused a rapid decline in global biodiversity [ 1 ] . Using sensor arrays for biodiversity monitoring, ecologists can gather information on environments and investigate how human pressures affect biodiversity and how we can halt its decline [ 2 ] . Passive acoustic monitoring (PAM) is one method that provides a low-cost and non-invasive way to monitor biodiversity [ 3 ] . The vast amount of data generated by PAM sensors has led to the rapid development of bioacoustic deep learning models to help researchers reduce the annotation effort [ 4 ] . The use of these state-of-the-art models has proven valuable in ecological studies, for example, in general species assessments [ 5 ] or detection of endangered species [ 6 ] . While bioacoustic deep learning models are a useful asset in ecology, it is crucial to understand the models’ limitations based on their training setup. The two main training strategies to develop bioacoustic deep learning models are supervised learning and self-supervised learning. Supervised learning models require large amounts of annotated data to be trained [ 7 ] . The models classify sounds based on a fixed number of predefined classes representing annotated sounds in the dataset. While supervised learning can have the benefit of instructing models to differentiate between species vocalizations, it requires annotated datasets. This limits supervised learning models to known and annotated classes and makes them sensitive to class imbalance and label quality. Alternatively, self-supervised learning can be trained without annotated data, enabling models to learn meaningful representations directly from the data. For example, in a paradigm called masked prediction, the model is trained to predict a masked portion of the audio, thus modeling bioacoustic characteristics without supervision [ 8 ] . With growing annotated databases, supervised learning models like BirdNET [ 9 ] improve in performance, yet recent developments in self-supervised learning on models like Animal2vec [ 10 ] indicate a promising new direction for the field requiring less manual annotation. Not requiring annotations, self-supervised learning models can be trained on far larger datasets, however there is no control of what is being learned. The model might, for instance, learn to differentiate between sounds based on characteristics very different from the species that produces them. To evaluate bioacoustic deep learning models, a comparison of just the classifier performance obscures the fine-grained differences between models and the way they analyze input sounds. Bioacoustic deep learning models consist of artificial neural networks that can be subdivided into feature extractors and classifiers. The feature extractor creates an embedding (vector representation) of an input sound and the classifier (which corresponds to the final dense layer of the model) maps the embedding onto classes. Commonly, in bioacoustics, a suite of established benchmarks is used to compare the classifier performance of state-of-the-art models [ 11 ] . This requires the models to have been trained on the classes present in the benchmarking datasets. However, there is an alternative: using the embeddings created by the feature extractors, the embedding spaces can be analyzed with regard to their structural characteristics, irrespective of what the classifiers were trained on. Due to their size, datasets that are used to train bioacoustic models for bird detection are often based on citizen science data (e.g., xeno-canto [ 12 ] ). The majority of these recordings are focal recordings of individuals which are weakly labeled with little polyphony. When models are applied to large PAM datasets, especially outside of North America (where the majority of the training data originates), the difference in recording conditions causes performance drops [ 13 ] . To accurately evaluate the models in this study, we use a PAM bird vocalization dataset from Colombia and Costa Rica [ 14 ] , as well as a dataset of frog vocalizations from Brazil [ 15 ] . Both datasets are comprised of PAM recordings in noisy environments and can therefore be considered as challenging datasets. With this, we hope to on the one hand emphasize the performance differences between the models and on the other hand produce results that will reflect in real world applications. This study aims to showcase the potential for evaluating and especially comparing bioacoustic deep learning models with regard to their training paradigm and training data. The evaluation focuses on the structuring capabilities of each feature extractor, as measured by clustering and classification performance. Classification in this case refers to recognition of novel classes, as none of the models have been pretrained on the classes selected here. This way, classification heads are attached to each feature extractor, all of which are trained on the same evaluation sets (bird and frog vocalizations), i.e. same classes and same data, allowing us to compare their performance. Classification is done using a k-nearest neighbor (kNN) approach. This method of analysis opens up the possibilities for a fair comparison of deep learning feature extractors guiding the field to a better understanding of how training configurations affect downstream performance. 2 Methods Table 1 : List of feature extractors compared in this study. Columns ”abbrev.” shows the an abbreviated name used in Fig. 1 . ”training” shows the training setup chosen during training, i.e. ssl for self-supervised learning, supl for supervised learning and ft for fine-tuning. The ”architecture” column more specifically describes the model architecture used. ”dimension” shows the output dimension of the feature extractor. ”trained on” summarizes the training data of the model. ”ref.” provides the respective publication. name abbrev. training architecture dimension trained on ref. Animal2vec_XC a2v_xc ssl d2v2.0 768 birds [ 10 ] Animal2vec_MK a2v_mk ssl + ft d2v2.0 1024 meerkats [ 10 ] AudioMAE aud_mae ssl + ft ViT 768 general [ 8 ] AVES_ESpecies aves ssl + ft HuBERT 768 general + animals [ 7 ] AvesEcho_PaSST aecho supl PaSST 768 birds [ 16 ] BioLingual bioling supl CLAP 512 animals + birds [ 17 ] BirdAVES_ESpecies birdaves ssl + ft HuBERT 1024 general + birds [ 7 ] BirdNET brdnet supl EffNetB0 1024 birds [ 9 ] Google_Whale g_whale supl EffNetB0 1280 whales - Insect459NET i459 supl EffNetv2s 1280 insects - Insect66NET i66 supl EffNetv2s 1280 insects - NonBioAVES_ESpecies nonbioaves ssl