No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings

Chenggang Chen, Zhiyu Yang · 2025

Read No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings by Chenggang Chen, Zhiyu Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10230.txt.

Introduction

Chenggang Chen and Zhiyu Yang wrote this paper in two thousand twenty-five. It warns that off-the-shelf audio pretraining gives no free lunch in bioacoustics, the study of animal sound.

Pulling embeddings from audio-trained nets, with no fine-tuning, has grown popular for wildlife tasks. The team grades a bank of models by shrinking their embeddings and clustering them. Un-tuned audio models lose even to a fine-tuned old net. Both tuned and un-tuned nets fail to part background from marked calls, though one net family does. Gains come when few background clips are in the tune set. Fine-tuning and post checks are a must, they say.

Catalog record

CREATORChenggang Chen, Zhiyu Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10230

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings Chenggang Chen, Zhiyu Yang Abstract Bioacoustics, the study of animal sounds, offers a non-invasive method to monitor ecosystems. Extracting embeddings from audio-pretrained deep learning (DL) models without fine-tuning has become popular for obtaining bioacoustic features for tasks. However, a recent benchmark study reveals that while fine-tuned audio-pretrained VGG and transformer models achieve state-of-the-art performance in some tasks, they fail in others. This study benchmarks 11 DL models on the same tasks by reducing their learned embeddings’ dimensionality and evaluating them through clustering. We found that audio-pretrained DL models 1) without fine-tuning even underperform fine-tuned AlexNet, 2) both with and without fine-tuning fail to separate the background from labeled sounds, but ResNet does, and 3) outperform other models when fewer background sounds are included during fine-tuning. This study underscores the necessity of fine-tuning audio-pretrained models and checking the embeddings after fine-tuning. Our codes are available: https://github.com/NeuroscienceAI/Audio_Embeddings I Introduction Computational bioacoustics with machine learning has been widely used for classifying and detecting animal sounds [ 1 ] . The major challenge is to extract embeddings, the high-dimensional representation of audio features. There are at least three methods to obtain the embeddings: 1) Predefined acoustic features like mel frequency cepstrum coefficient (MFCC) [ 2 , 3 , 4 , 5 , 6 , 7 ] , 2) Spectrograms of single vocalizations or syllables [ 8 , 9 , 10 , 11 ] , and 3) Audio-pretrained models. Using audio-pretrained models without fine-tuning is popular [ 12 , 13 , 14 , 15 , 16 ] . For example, [ 17 ] uses YouTube audio-pretrained VGGish [ 18 ] to calculate fingerprints of soundscapes from a variety of ecosystems. [ 19 ] shows that eleven human speech-pretrained models could distinguish the identities of ten animals. Do we still need fine-tuning? [ 20 ] recently found that under noisy conditions, MFCC outperforms all three audio-pretrained models without fine-tuning. To the best of our knowledge, no studies compare the embeddings of animal sounds before and after fine-tuning the audio-pretrained models. Fine-tuned audio-pretrained models are not considered state-of-the-art either. Two recent studies [ 21 , 22 ] fine-tuned audio-pretrained models: one is VGGish, and the other is bioacoustics-pretrained transformer models (AVES-bio), and benchmarked them over ten datasets. In three classification datasets, both the audio- and bioacoustics-pretrained models outperform all other models, achieving more than 0.8 accuracy. In contrast, in three detection tasks, they exhibit less than 0.4 precision and even underperform compared to other DL models. The reason is unclear, partly attributed to the absence of benchmarked embeddings from audio-pretrained and other deep learning models. Our contributions in bioacoustics are as follows: 1. First study benchmarks DL models with embeddings. 2. First study evaluates 11 DL models on same datasets. 3. Finds fine-tuning is necessary for pretrained models. 4. Explains why pretrained models fail on some datasets. 5. Improves task performance of the pretrained models. II Methods Figure 1 shows the four major steps of this study: learning embeddings, dimensionality reduction, clustering, and evaluation. Fig. 1: Spectrograms or waveforms from ten datasets are fed into eleven deep neural networks to learn the audio embeddings. Embeddings (high-dimensional and latent) are projected (t-SNE and UMAP) to two dimensions for visualization and clustering (KMeans). Clustering performance is evaluated with three metrics (NMI, ARI, and Silhouette coefficient). II-A Eleven deep learning models Fig. 2: High dimensional embeddings of five classification (row 1st, 2nd) and five detection (row 3rd, 4th) datasets are projected to two dimensions using t-SNE (perplexity=50, iteration=5000). First row shows the embeddings extracted from bioacoustics-pretrained model without fine-tuning. Deep blue dots (class 0, same as Figure 3) in the 3rd and 4th rows are the unlabeled background sounds. We trained deep learning models using PyTorch on a single GPU (RTX A5000, NVIDIA). We trained three ResNet (rn18, rn50, rn152) models from scratch and fine-tuned five ImageNet-pretrained AlexNet (alexp), ResNet (rn18p, rn50p, rn152p), and Swin Transformer (swinTp) models, one YouTube audio-pretrained VGG (vggish [ 18 ] ) model, and two audio (all sounds and animal sounds) pretrained HuBERT model (aves [ 22 ] ). We loaded the weights of alexnet, resnet and swinTp from Torchvision, vggish from PyTorch Hub (’harritaylor/torchvggish’), and aves that ported to Torchaudio from Github (earthspecies/aves). AVES models are fine-tuned with raw waveform and other models are trained or fine-tuned with spectrogram. Notice that the original BEANS only has resnet and vggish, and we integrated the alexp, swinTp, and two aves models into the BEANS. We used the same optimizer setting (Adam), batch size (32), and learning rates (0.00001, 0.00005, 0.0001) as BEANS. We picked the trained model that has the best performance on the validation dataset. The hyperparameters search (i.e., three learning rates), training, and validation have never been used on the test datasets. II-B Embeddings dimensionality reduction We used t-SNE [ 23 ] (Figure 2 ) and UMAP [ 24 ] to reduce the high dimensional embeddings. In t-SNE, the primary parameter is perplexity, and we tested three values: 25, 50, and 100. Since higher iterations number result in better performance, we used 5000 (default is 1000). In UMAP, the primary parameters are number of neighbor and minimal distance, and we tested two values for each of them (50, 100; 0.1, 0.5). TABLE I: Clustering (KMeans, k=class number) of two dimensional learnt embeddings (similar to Figure 2; t-SNE, perplexity=25, iteration=5000) is evaluated with normalized mutual information (NMI) on 11 models and 10 datasets. Larger values mean better clustering. The best and second best data are highlighted and underlined, respectively. Last six lines show data from untrained (1), pretrained without fine-tuning (2-4), and incomplete fine-tuned (5-6) models . Model wtkn bat cbi hbdb dogs dcase enab hiceas rfcx gib AlexNet image-pretrained 0.551 0.298 0.517 0.314 0.367 0.150 0.346 0.003 0.081 0.114 ResNet18 0.921 0.345 0.722 0.544 0.937 0.216 0.584 0.219 0.179 0.414 ResNet50 0.836 0.350 0.847 0.548 0.570 0.223 0.563 0.271 0.203 0.422 ResNet152 0.817 0.566 0.778 0.477 0.482 0.227 0.577 0.021 0.207 0.397 ResNet18 image-pretrained 0.909 0.331 0.901 0.453 0.886 0.221 0.590 0.158 0.195 0.513 ResNet50 image-pretrained 0.895 0.446 0.839 0.490 0.814 0.303 0.596 0.257 0.275 0.519 ResNet152 image-pretrained 0.919 0.520 0.889 0.612 0.760 0.324 0.579 0.405 0.295 0.347 Swin Transformer image-pretrained 0.825 0.516 0.818 0.471 0.714 0.212 0.571 0.011 0.114 0.179 VGGish audio-pretrained 0.919 0.830 0.828 0.739 0.845 0.293 0.599 0.104 0.204 0.314 AVES audio-pretrained 0.963 0.775 0.950 0.733 1.000 0.223 0.586 0.145 0.252 0.370 AVES bioacoustics-pretrained 0.971 0.653 0.981 0.776 1.000 0.223 0.568 0.146 0.253 0.410 ResNet152 no pre-training no fine-tuning 0.206 0.049 0.304 0.088 0.109 0.038 0.029 0.006 0.047 0.001 ResNet152 image-pretrained no fine-tuning 0.276 0.092 0.340 0.159 0.150 0.092 0.059 0.010 0.059 0.002 VGGish audio-pretrained no fine-tuning 0.516 0.142 0.339 0.442 0.163 0.145 0.164 0.011 0.095 0.015 AVES bioacous.-pretrained no fine-tuning 0.581 0.215 0.373 0.487 0.195 0.147 0.287 0.000 0.121 0.013 AVES bioacous.-pretrained 1 epoch 0.530 0.249 0.354 0.431 0.167 0.108 0.173 0.008 0.024 0.063 AVES bioacous.-pretrained 3 epochs 0.629 0.467 0.583 0.553 0.287 0.227 0.382 0.170 0.054 0.286 II-C Ten animal sounds datasets We train or fine-tune models on datasets used by BEANS [ 21 ] (Figure 3 ). We used the same codes for pre-processing the

Read the complete text