On feature representations for marmoset vocal communication analysis

Eklavya Sarkar, Kaja Wierucka, Alexandra B. Bossard, Judith Burkart, Mathew Magimai.-Doss · 2025

Read On feature representations for marmoset vocal communication analysis by Eklavya Sarkar, Kaja Wierucka, Alexandra B. Bossard, Judith Burkart, Mathew Magimai.-Doss (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14981.txt.

Introduction

Eklavya Sarkar and co-authors compared ways of turning marmoset monkey calls into numbers a computer can classify. Marmoset vocal studies inform debates about the roots of human language, but call analysis is still largely done by hand. The team tested three feature styles across three datasets: a hand-crafted statistical set called Catch22, features from neural networks pre-trained on human speech, and models that learn directly from the sound.

The speech-trained features and the end-to-end models generally beat the hand-crafted set at naming call types and identifying which animal called. Sex identification proved harder. The authors also show that the recording bandwidth, how much of the sound spectrum is kept, noticeably shifts performance, a practical warning for future automated studies.

Catalog record

CREATOREklavya Sarkar, Kaja Wierucka, Alexandra B. Bossard, Judith Burkart, Mathew Magimai.-Doss
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14981

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On feature representations for marmoset vocal communication analysis Eklavya Sarkar, Kaja Wierucka, Alexandra B. Bossard, Judith Burkart, Mathew Magimai.-Doss Abstract The acoustic analysis of marmoset ( Callithrix jacchus ) vocalizations is often used to understand the evolutionary origins of human language. Currently, the analysis is largely carried out in a manual or semi-manual manner. Thus, there is a need to develop automatic call analysis methods. In that direction, research has been limited to the development of analysis methods with small amounts of data or for specific scenarios. Furthermore, there is lack of prior knowledge about what type of information is relevant for different call analysis tasks. To address these issues, as a first step, this paper explores different feature representation methods, namely, HCTSA-based hand-crafted features Catch22, pre-trained self supervised learning (SSL) based features extracted from neural networks trained on human speech and end-to-end acoustic modeling for call-type classification, caller identification and caller sex identification. Through an investigation on three different marmoset call datasets, we demonstrate that SSL-based feature representations and end-to-end acoustic modeling tend to lead to better systems than Catch22 features for call-type and caller classification. Furthermore, we also highlight the impact of signal bandwidth on the obtained task performances. Journal: Bioacoustics 1 Introduction The advancements in human speech processing have accelerated and impacted research in non-human communication, such as bioacoustics, i.e. the study of animal sounds. Common marmosets ( Callithrix jacchus ) have recently gained prominence as a valuable research model among non-human primates. This is primarily due to their exceptional vocal abilities, which are rooted in their highly complex social behavior and cooperative breeding system [ 1 , 2 ] . They possess extensive vocal repertoires used in various social situations [ 3 , 4 ] , and their vocalizations have the capacity to encode a wide range of information, such as population, group affiliation, sex, and even individual identity [ 5 , 6 , 7 , 8 , 9 , 10 ] . These vocalizations are not limited to simple tonal signals but also encompass complex calls with multiple frequency components, some of which are within the ultrasonic range [ 11 ] . Moreover, marmosets have been observed to exhibit remarkable vocal adaptability. They can alter the duration [ 12 ] , intensity [ 12 , 13 , 14 ] , complexity [ 15 ] , or timing [ 16 , 15 ] of their calls, even when faced with disruptions in their environment that occur after the initiation of a call [ 14 ] . While these properties make marmosets an intriguing subject for the study of communication processes, they also pose a significant challenge when attempting to automate the analysis of their vocalizations. The literature on automatic marmoset vocalization analysis is relatively sparse. Turreson et al. compared different classification methods for marmoset ‘call-type’ classification using linear prediction coefficients as feature representation, and found that on a small data setup of 30 samples per call-type, k-NN, SVM and optimal path forest algorithms yield better performance than multilayer perceptron, Adaboost, and logistic regression [ 17 ] . Wisler et al. investigated different feature representations, namely, audio features (statistics based on energy entropy, signal energy, zero crossing rate, spectral rolloff, spectral centroid, and spectral flux), mel-frequency cepstral coefficients (MFCCs), and Teager energy operator-based features for marmoset vocalization and call-type detection [ 18 ] . On a synthetic dataset, created by taking a small set of calls and augmenting it with background noise and acoustic events, it was found that feature level combination leads to better performance. Verma et al. investigated discovering of different patterns in marmoset calls through unsupervised learning. Specifically, they developed an HMM-based approach to segment and cluster marmoset vocalizations into discrete units through multi-resolution and multi-rate analysis of the signal [ 19 ] . In [ 20 ] , it was demonstrated that marmoset vocalizations and call-types can be better detected/classified by feeding statistics of log-mel-filter bank energies as input to recurrent neural networks, when compared to feeding it to SVM or multilayer perceptrons. In the scenario of analyzing recordings obtained from a pair of marmosets, [ 21 ] investigated a deep learning approach where a spectrogram was fed as input to a convolutional neural network to jointly perform vocalization, detection, call type classification and caller detection. It was found that joint modeling yielded better performance than training systems individually for each task in this scenario. Recently, Highly Comparable Time-Series Analysis (HCTSA) features have been used to model source (caller) identification through an Adaboost-based hierarchical approach for marmosets [ 10 ] , as well as for 14 mammalian species [ 22 ] . In the bioacoustics field, breakthroughs in self-supervised learning (SSL), which leverages unlabeled data by creating surrogate labels from the data’s inherent structure, has led to works which explore birdsong detection [ 23 ] and bioacoustic event detection [ 24 ] by pre-training with a contrastive learning approach. In that direction, a study using different SSLs pre-trained on human speech, demonstrated that neural embeddings extracted in such a framework can also distinguish marmoset callers [ 25 ] . However, in the existing works, there are three main limitations. First, most of the studies have been carried out on small data sets. Second, these studies have been conducted on datasets intended for specific scenarios. Due to a lack of validation, it is unclear whether the methods studied on one dataset would scale to another. Third, there is limited prior knowledge about what type of information is relevant for different call analysis tasks. There is a need to overcome these limitations to advance the development of automatic analyses of marmoset vocalizations. The present paper is a step in that direction with a specific focus on feature representations for automatic marmoset call analyses, where we investigate three prominent feature representation methods, namely, (a) hand-crafted features, (b) self-supervised learning-based representations, and (c) end-to-end acoustic modeling, on three different marmoset call datasets and three different tasks (call type, caller identity, and caller sex classification). The paper is organized as follows. Section 2 presents the different datasets, tasks, and investigated feature representations. Section 3 and 4 present the studies and analysis of the results respectively. Finally section 5 concludes the paper. 2 Methodology 2.1 Datasets and tasks We conduct investigations on three different marmoset datasets, denoted as D_{1} , D_{2} , and D_{3} , respectively. D_{2} and D_{3} contain vocalizations produced by adult individuals, while D_{1} , InfantMarmosetsVox, originates from infant marmosets [ 25 ] . Consequently, D_{1} is expected to encompass different call types, likely characterized by higher frequencies compared to those in D_{2} and D_{3} . Furthermore, D_{2} and D_{3} are gathered from the same colony, while D_{1} was obtained from a different one. All the datasets consist of audio recordings of marmosets vocalizations segments, collected and hand-labeled with the start and end time by experienced researchers. In addition to call-type and caller identity annotations of each vocalization provided for all three datasets, D_{1} and D_{2} also include information about the sex of the vocalizing individual. For more details regarding the datasets, the reader is referred to A . Figure 1: Log distribution of vocalization lengths per dataset. The medians are calculated over the entirety of

Read the complete text