Multi-layer attentive probing improves transfer of audio representations for bioacoustics
Marius Miron, David Robinson, Masato Hagiwara, Titouan Parcollet, Jules Cauzinille, Gagan Narula Milad Alizadeh, Ellen Gilsenan-McMahon, Sara Keen, Emmanuel Chemla, Benjamin Hoffman Maddie Cusimano, Diane Kim, Felix Effenberger, Jane K. Lawton, Aza Raskin Olivier Pietquin, Matthieu Geist · 2026
Read Multi-layer attentive probing improves transfer of audio representations for bioacoustics by Marius Miron, David Robinson, Masato Hagiwara, Titouan Parcollet, Jules Cauzinille, Gagan Narula Milad Alizadeh, Ellen Gilsenan-McMahon, Sara Keen, Emmanuel Chemla, Benjamin Hoffman Maddie Cusimano, Diane Kim, Felix Effenberger, Jane K. Lawton, Aza Raskin Olivier Pietquin, Matthieu Geist (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10494.txt.
Introduction
Marius Miron and seventeen colleagues test whether better probing changes how bioacoustic encoders rank. Probing heads map audio representations learned by a model to downstream task labels, and most bioacoustic benchmarks settle on one fixed, low-capacity probe, like a linear layer over the encoder's last layer. That standardization may bias results by overlooking how encoder features interact with probe design. Their 2026 paper studies probing strategies across two benchmarks, BEANS and BirdSet.
The authors evaluate last-layer and multi-layer probing, across linear and attention probes. Larger probe heads that leverage time information perform better. Multi-layer probing lifts downstream scores across all tested models, while attention probing tops linear probing for transformer models. The results suggest current benchmarks may misjudge encoder quality when they rely on a last-layer setup. Readers get a caution: a benchmark that fixes the probe design risks underestimating some models and overestimating others.
Catalog record
| CREATOR | Marius Miron, David Robinson, Masato Hagiwara, Titouan Parcollet, Jules Cauzinille, Gagan Narula Milad Alizadeh, Ellen Gilsenan-McMahon, Sara Keen, Emmanuel Chemla, Benjamin Hoffman Maddie Cusimano, Diane Kim, Felix Effenberger, Jane K. Lawton, Aza Raskin Olivier Pietquin, Matthieu Geist |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-layer attentive probing improves transfer of audio representations for bioacoustics Marius Miron, David Robinson, Masato Hagiwara, Titouan Parcollet, Jules Cauzinille, Gagan Narula Milad Alizadeh, Ellen Gilsenan-McMahon, Sara Keen, Emmanuel Chemla, Benjamin Hoffman Maddie Cusimano, Diane Kim, Felix Effenberger, Jane K. Lawton, Aza Raskin Olivier Pietquin, Matthieu Geist Abstract Probing heads map the representations learned from audio by a machine learning model to downstream task labels and are a key component in evaluating representation learning. Most bioacoustic benchmarks use a fixed, low-capacity probe, such as a linear layer on the final encoder layer. While this standardization enables model comparisons, it may bias results by overlooking the interaction between encoder features and probe design. In this work, we systematically study different probing strategies across two bioacoustic benchmarks, BEANs and BirdSet. We evaluate last- and multi-layer probing, across linear and attention probes. We show that larger probe heads that leverage time information have superior performance. Our results suggest that current benchmarks may misrepresent encoder quality when relying on a last-layer probing setup. Multi-layer probing improves downstream task performance across all tested models, while attention probing has superior performance to linear probing for transformer models. † † address: Earth Species Project 1 Introduction Figure 1: The architecture of the probes we use for a given base model. We test three main probing strategies: A. Last-layer probing and B. All-layer probing C. Fully fine-tuning. The fire symbol denotes that the model is trainable and the snowflake that the parameters are frozen. Bioacoustics [ 2 ] is an interdisciplinary field that examines the production and perception of sound in animals, linking acoustic signals to behavioral, ecological, and evolutionary processes. Recently, the field benefited from the use of deep learning [ 20 ] , either by developing domain-specific approaches or by tailoring an existing audio encoder for a given dataset. Bioacoustic encoders learn representations from animal sounds for tasks such as species classification [ 7 ] , detection [ 10 , 22 , 14 ] , and individual identification [ 19 ] , supporting applications in biodiversity monitoring [ 20 ] or animal communication [ 2 ] . Recent work [ 22 , 12 ] has explored what matters for training such encoders, evaluating the impact of training data diversity, model architectures, and learning paradigms across large-scale benchmarks such as BEANS [ 7 ] and BirdSet [ 14 ] . These studies have produced versatile encoders applicable to species classification, individual identification, and behavioral analysis, yet they share a common limitation: evaluation typically relies on fixed, low-capacity neural networks on top of the final encoder, commonly known as probing heads . In speech representation learning, this practice has been questioned. Studies have shown that encoder rankings and performance can change dramatically when altering the probing head architecture [ 23 , 21 , 17 , 18 ] , suggesting that current benchmarks may conflate encoder quality with probe design [ 24 ] . Low-capacity probes may fail to exploit information spread across layers or time. As a result, benchmarks that fix the probe design risk underestimating some models and overestimating others. We address this issue for bioacoustics by benchmarking multiple audio encoders usage, probing strategies, and probing heads. We focus here on single- and multi-layer probing, comprising adapter modules for aligning layer outputs, and learned layer weights. We evaluate two probe types—linear and attention-based probes. In addition, we evaluate fully fine-tuning the model with the respective probe heads attached to the last layer as a top-line. Our experiments on BEANS and BirdSet benchmarks show that probe choice significantly affects encoder performance with multi-layer probing giving better performance than last-layer probing, and attention probing being particularly useful when using embeddings extracted from self-supervised learning models. We propose concrete recommendations for practitioners when opting for the right probing strategy and head, given a task and dataset. More, we advocate for introducing larger probe heads and different probing strategies in bioacoustics benchmarks. We open source the code as a Python library [ 1 ] . 2 RELATION TO PRIOR WORK In this paper we build upon the existing bioacoustics benchmarks and extend them in a four folded manner. Layer-wise probing of speech models has been studied in the context of bioacoustics benchmarks [ 3 , 16 ] . In contrast to these we propose to (1) leverage embeddings extracted from all the blocks of the base models , a de-facto strategy for speech benchmarks such as SUPERB [ 21 , 24 ] . Adding to that, we average the layers using a set of (2) interpretable learned weights that we can explored for each dataset. However, in contrast to the SUPERB setup we aim at evaluating architectures besides transformers, potentially aggregating embeddings of different shapes. Importantly, a wide choice as off-the-shelf models, EfficientNets such as BirdNET [ 10 ] are widely used and linear probing on the last layer is the de-facto strategy. Thus, (3) we introduce adapters as part of the probing strategy to project embeddings of different shapes to the same dimension. In contrast to the previous work on evaluating foundation models for bioacoustics [ 6 , 9 , 22 , 12 ] (4) we provide an extensive probing study to disentangle the base model types (self-supervised learning, SSL vs supervised learning, SL), architectures (transformer vs CNNs), probing strategies (multi- vs last-layer), and heads (attention vs linear), towards having a comprehensive evaluation of the popular design choices when transferring representations for downstream bioacoustics tasks. 3 METHODS The evaluation framework including fully fine-tuning and the two main probing strategies, last- and multi-layer, are presented in Figure 1 and defined in Section 3.1 . We introduce the probe and classification heads in Section 3.2 , and the base models in Section 3.3 . We give details about the datasets, tasks, evaluation metrics, hyperparameters in Section 3.4 . 3.1 Task definition: probe strategies Let f_{\theta} be a pre-trained base model with parameters \theta . For a base model with L layers, let h^{l}\in\mathbb{R^{*}} denote the hidden representation i.e the embeddings at layer l , with * representing variable dimensions that depend on the type of layer, or more precisely \mathbb{R}^{d_{1}\times d_{2}\times d_{3}} for convolutional layers and \mathbb{R}^{d_{1}\times d_{2}} for attention or recurrent layers. Given a downstream task and dataset \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N} with x_{i}\in\mathcal{X} audio files and y_{i}\in\mathcal{Y} the labels, we define different probing strategies to extract and utilize embeddings from the base model. A. Last-layer probing. The base model parameters \theta remain frozen, and only the probe parameters \phi are optimized: \hat{y}=g_{\phi}(h^{L}) . B. All-layer probing. Because choice of the layer may impact the performance on the downstream task, we extract representations from all blocks of the base models. We aim at aggregating heterogeneous layer representations resulting in embeddings of different dimensions. Thus, we need to project the embeddings to a common dimension so that the probe may work on them. We design adapters A_{\psi_{l}}:\mathbb{R}^{*}\rightarrow\mathbb{R}^{T_{\max}\times F_{\max}} to project a layer l to a unified two-dimensional format: (T_{\max},F_{\max}) , where T_{\max} is the maximum sequence length across all embeddings, and F_{\max} is the maximum feature dimension. Each adapter outputs \hat{h}^{l}=A_{\psi_{l}}(h^{(l)}) with \tilde{h}^{l}\in\mathbb{R}^{T_{\max}\times F_{\max}} . Importantly, we design adapters to process time and feature dimension separately. Note that for a CNN layer activations with dimensions