Challenges in Automated Processing of Speech from Child Wearables: The Case of Voice Type Classifier
Tarek, Marianne, Hadrien, Lucas, Joseph, Emmanuel, Alejandrina, Marvin · 2025
Read Challenges in Automated Processing of Speech from Child Wearables: The Case of Voice Type Classifier by Tarek, Marianne, Hadrien, Lucas, Joseph, Emmanuel, Alejandrina, Marvin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11074.txt.
Introduction
Tarek Kunze and a team of co-authors report hard lessons from three years of work on child speech recordings. Their twenty twenty-five paper deals with audio from devices worn by children, which capture their daily speech world. To use that flood of sound, tools must first sort voices by type: child, adult, and so on.
The team tried to improve that one task, voice type classification, in many ways. Better features, new designs, and wide tuning gave only small gains. Bigger gains came from data that fit the task and came in larger amounts. Their moral: for this field, the right shared data matters more than clever models.
Catalog record
| CREATOR | Tarek, Marianne, Hadrien, Lucas, Joseph, Emmanuel, Alejandrina, Marvin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Challenges in Automated Processing of Speech from Child Wearables: The Case of Voice Type Classifier Tarek, Marianne, Hadrien, Lucas, Joseph, Emmanuel, Alejandrina, Marvin Abstract Recordings gathered with child-worn devices promised to revolutionize both fundamental and applied speech sciences by allowing the effortless capture of children’s naturalistic speech environment and language production. This promise hinges on speech technologies that can transform the sheer mounds of data thus collected into usable information. This paper demonstrates several obstacles blocking progress by summarizing three years’ worth of experiments aimed at improving one fundamental task: Voice Type Classification. Our experiments suggest that improvements in representation features, architecture, and parameter search contribute to only marginal gains in performance. More progress is made by focusing on data relevance and quantity, which highlights the importance of collecting data with appropriate permissions to allow sharing. Kunze Métais Titeux Elbert Coffey Dupoux Cristia Lavechin DEC, ENS, EHESS, CNRS, PSL UniversityFrance Massachusetts Institute of TechnologyUnited States keywords wearables, in-the-wild audio, multi-label classification, voice type classification, LENA † † email: [email protected] 1 Introduction Child-worn recording devices have transformed both fundamental and applied speech sciences, capturing egocentric audio (or audio-video) that represents speech from the child’s perspective. They enable effortless, long-duration recordings, offering unprecedented insights into children’s language environments [ 1 ] . Unlike laboratory studies, which may misrepresent everyday speech, or short-form recordings, which are more prone to observer bias, long-form recordings provide more naturalistic data [ 2 ] . These recordings are key to reverse-engineering language acquisition [ 1 , 3 ] and have also been used in applied research, such as evaluating interventions to enhance parental communication and even providing feedback to parents [ 4 , 5 ] . The value of this technique hinges on the capabilities of advanced speech technologies, which are essential to transform the vast mounds of data collected via child-worn devices into intelligible descriptors usable by researchers and clinicians. Many users of this technology hope for interpretable metrics, such as how much parents talked to and around the child [ 6 ] , how advanced the child’s vocalizations were [ 7 ] , and, in an ideal world, what children and caregivers were talking about [ 8 , 9 , 10 ] . From the speech technology viewpoint, this results in users’ requests for algorithms that accurately identify, categorize, and transcribe verbal interactions assigned to different speakers. Where are we in delivering on this promise? For the last two decades, the technology predominantly used to fulfill this need has been developed by the LENA Foundation, which has made advances in the identification and categorization of vocalizations by children and adults [ 11 , 12 ] . Despite these successes, LENA faces significant obstacles to its future viability. Although the LENA software is proprietary and not openly accessible, available descriptions indicate that it still relies on technology that, by 2025, appears to be outdated. Specifically, the software uses a Gaussian Mixture Model with 36 Mel-frequency cepstral coefficients as input, structured within a minimum duration framework and developed under a maximum likelihood estimation approach to categorize audio segments. These segments are classified into broad speaker and nonspeaker categories, which are often simplified into 4 key categories: Key Child, Other Child, Adult Male, and Adult Female, with everything else being treated as background noise or silence. Despite LENA’s technological seniority, it remains the benchmark in the field, with anecdotal evidence suggesting that over 95% of scholarly articles rely on LENA technology. It was only in 2020 that an open-source alternative emerged, matching or even exceeding LENA’s accuracy in the fundamental task of segmenting audio into different speaker types. Capitalizing on the rise of deep learning, Lavechin et al. [ 13 ] introduced the Voice Type Classifier (VTC), a model that aimed to democratize access to cutting-edge speech processing technology by releasing as open-source the best-performing model, selected after quite extensive experimentation. The released VTC’s architecture combined SincNet, to extract low-level features with a stack of LSTMs (Long Short-Term Memory) to aggregate them into context-dependent representations. The best-performing model was trained to return a score for each of Key Child, Other Child, Adult Male, Adult Female, and Speech (a class that was on when any of the others were on, as well as for live speech that human annotators had not been able to assign to a specific source). Unlike LENA, the VTC is freely accessible, updatable, and unconstrained by proprietary recording hardware, making it advantageous for many different use cases (see [ 14 ] for a recent performance comparison). Despite our excitement about this open-source technology, users and machine learning enthusiasts will be unimpressed by the consideration that VTC outperformed LENA only by a slight margin (F-score of 69% vs. 55% for key child; 33% vs. 29% for other children; 63% vs. 43% for female adult; and 43% vs. 37% for male adult, see [ 13 ] for more details). These results teach us a humbling lesson: detecting who speaks when remains vexingly difficult, even for modern deep learning approaches. In this paper, we summarize three years of work attempting to improve VTC performance. Since only one previous study [ 15 ] investigated foundation models with English long-form data, we contribute novel data using a more diverse dataset and benchmark against the current open-source state-of-the-art [ 13 ] . First, we demonstrate that pre-trained representations learned by Whisper [ 16 ] are most effective for smaller training sets. Second, we demonstrate that these pre-trained representations benefit the rare male speech class the most. Third, we provide a detailed analysis of the types of errors made by our model. Finally, we show that there is still room for improvement by benchmarking against human-human agreement. We conclude by reflecting on the numerous directions we explored to improve performance, too many to detail in a 4-page paper. 2 Methods 2.1 End-to-end voice type classification As in [ 13 ] , we framed the voice type classification problem as a multi-label classification problem, where the input is the audio stream divided into N frames S=\{s_{1},s_{2},\ldots,s_{N}\} and the expected output is the corresponding sequence of labels \boldsymbol{y}=\{\boldsymbol{y}_{1},\boldsymbol{y}_{2},\ldots,\boldsymbol{y}_{N}\} where each \boldsymbol{y}_{i} is of dimension K (the number of labels) with y_{i,j}=1 if the j^{th} class is activated, y_{i,j}=0 otherwise. In training, multiple sequences from the training set were sampled for the total number of frames across all sequences to sum to M in each mini-batch. Our baseline model is PyanNet-VTC, PyanNet [ 13 , 17 ] retrained from scratch. The PyanNet architecture uses a SincNet [ 18 ] feature encoder to learn meaningful filter banks, followed by stacked bidirectional long short-term memory (LSTM) layers and feed-forward (FF) layers. In contrast, our proposed Whisper-VTC model replaces the SincNet encoder with frozen Whisper representations while introducing a different classification architecture. For Whisper-VTC, an 80-channel log-magnitude Mel spectrogram is computed using 25-millisecond windows with a 10-millisecond stride. These spectral features are processed through two convolutional layers before entering the frozen Whisper encoder. The encoder outputs are combined using a learnable weighted sum [ 19 ] . The key architectural difference is that these representations are then processed by shared bi-LSTM layers (size 256), whose output features are fed to K=4 independent binary classification heads, one for each