Can We Trust Machine Learning? The Reliability of Features from Open-Source Speech Analysis Tools for Speech Modeling

Tahiya, Veronica · 2025

Read Can We Trust Machine Learning? The Reliability of Features from Open-Source Speech Analysis Tools for Speech Modeling by Tahiya, Veronica (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11072.txt.

Introduction

Tahiya Chowdhury and Veronica Romero test whether open source speech tools give reliable features in this 2025 paper. Behavior models often read speech features from tools such as OpenSMILE and Praat. Those tools were rarely validated outside their design settings. Unreliable features threaten both reproducibility and fairness of the models built on them.

Their test case is adolescents with autism, a group far from typical tool tuning data. Features from the two tools varied a great deal on the same recordings. That variation changed model performance across contexts and demographic groups. A feature that looks solid in one tool may shift in another. The authors urge domain specific verification before clinical use of such pipelines.

Catalog record

CREATORTahiya, Veronica
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11072

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can We Trust Machine Learning? The Reliability of Features from Open-Source Speech Analysis Tools for Speech Modeling Tahiya, Veronica Abstract Machine learning-based behavioral models rely on features extracted from audio-visual recordings. The recordings are processed using open-source tools to extract speech features for classification models. These tools often lack validation to ensure reliability in capturing behaviorally relevant information. This gap raises concerns about reproducibility and fairness across diverse populations and contexts. Speech processing tools, when used outside of their design context, can fail to capture behavioral variations equitably and can then contribute to bias. We evaluate speech features extracted from two widely used speech analysis tools, OpenSMILE and Praat, to assess their reliability when considering adolescents with autism. We observed considerable variation in features across tools, which influenced model performance across context and demographic groups. We encourage domain-relevant verification to enhance the reliability of machine learning models in clinical applications. Chowdhury Romero Colby CollegeUSA Colby CollegeUSA keywords autism, reliable machine learning, speech processing tools. † † email: [email protected], [email protected] 1 Introduction Clinical conditions that manifest neurological, cognitive, and behavioral symptoms are typically diagnosed with standardized tools. Consider Autism Spectrum Disorder, which refers to a set of developmental conditions that restrict an individual’s ability for social communication and interaction, among other things. Currently, the clinical measures that characterize symptoms of autism are assessed by diagnostic instruments, which depend on subjective ratings of behavioral symptoms by an expert. While these measures are clinically valid and well understood, the current assessment process is cognitively demanding and costly. Many recent studies have shown promising results through the use of automated machine learning-based approaches using speech and language features as behavioral measures to classify clinical conditions such as Autism [ 1 , 2 , 3 , 4 ] and Dementia [ 5 , 6 ] . This approach relies on open-source software tools to extract behavioral features from speech samples recorded from individuals. While automated diagnostic tools have the potential to support existing diagnostic processes by providing objective behavioral measures, several challenges remain to translate them into clinical practice. First, the widely available open-source tools provide low-level speech features extracted from audio recordings, which are then used to model behavior and predict a diagnostic label of an individual. However, these features do not typically go through any process to check their validity in a given clinical context to be contextualized for behavioral modeling [ 7 , 8 ] . Second, various open-source tools exist to conveniently extract a large number of speech features including OpenSMILE [ 9 , 10 , 11 ] , Praat [ 7 , 4 ] , and ProsodyPro [ 12 ] , to name a few. However, these tools are designed for more general-purpose speech processing applications whose default settings may not translate well in clinical applications. Many design choices are involved in the feature extraction as well as the modeling process that is guided by the intention to improve classification scores instead of domain-specific knowledge [ 13 ] . As a result of this lack of standardization, over-optimistic results are reported that are actually the consequence of issues such as data leakage (e.g., sampling bias) [ 14 ] , model overfitting on small sample size [ 8 ] , and lack of generalizability across different contexts [ 1 ] . Finally, in the absence of clinically and contextually relevant features to characterize clinical conditions that work for diverse demographic groups and languages [ 15 ] , models tend to generalize poorly for marginalized groups in real-world clinical settings. In this work, we evaluate behavioral features measured using commonly used open-source speech analysis tools using audio data collected for autism diagnosis of adolescents. Specifically, we look at OpenSMILE [ 9 ] and Praat [ 16 ] as they provide speech features directly using default settings without any pre-processing, making them more accessible for model development. We focused on pitch and speech rate [ 17 , 12 , 18 , 19 , 20 ] based on prior literature where speech has been used as a biomarker to classify diagnostic label of autism. Specifically, Pitch standard deviation (pitch variability), Pitch range (pitch modulation during a specific conversation context) [ 21 ] and Loudness [ 10 ] are used to represent a person’s ability to adapt in a conversational setting. High mean pitch is generally used as a marker for distinctive speech patterns for ASD groups. Speech rate is generally used to understand the speed at which an individual speaks spontaneously, and slow speech rate can distinguish children (at developmental age) diagnosed with autism from their neurotypical peers [ 22 ] . We compare these five features measured with the two tools to understand the reliability of the measures and their impact on machine learning-based classification performance. We aim to answer the following questions in this work: • Are some speech features provided by widely used speech processing tools (OpenSMILE and Praat) measured the same way? • Do these tools measure speech features differently based on the gender of participants’ data? • Do these tools measure speech features differently based on diagnostic group membership? • Do machine learning classification models perform robustly for each diagnostic group regardless of the speech processing tool used? 2 Data We utilized data collected during sessions of the Autism Diagnostic Observation Schedule - Second Edition (ADOS-2), a standardized assessment tool designed to evaluate ASD-related impairments [ 23 ] . In this evaluation, an adolescent (‘Participant’) interacts with a certified adult assessor through a series of semi-structured activities that assess the child’s behavioral patterns. The dataset for this study comprised 14 distinct ‘Tasks’ from Module 3 of the ADOS-2, which is specifically designed for verbally fluent children and adolescents. Depending on the task, the participant may be asked to narrate a story, play with toys, act out a cartoon, or respond to questions on topics such as emotions, loneliness, and friendship. Since our focus is on features learned across different contexts, we incorporated all 14 tasks from the module. Our dataset consisted of recordings from 29 adolescents (14 with ASD and 15 typically developing, TD) between the ages of 10 and 15 ( M : 12.27, SD : 1.75). There were 21 male and 8 female participants. Each ADOS-2 session lasted approximately 40–60 minutes and was administered by the same psychologist. For this study, we first generated individual recordings for each subtask using annotations made by a research assistant based on video recordings of the sessions. The average length of these video segments is about five minutes per task. Speaker diarization was performed using pyannote [ 24 ] , and PyDub [ 25 ] was employed to segment the audio further based on speaker identity and turn boundaries obtained from pyannote . This process resulted in utterance-level audio files for both the participant and the psychologist. For the current project, we used the participants’ audio files for speech feature extraction. After filtering out empty utterances, our final dataset contained an average of 116 utterances per participant. 3 Method We first extracted speech features from the audio recordings of the participants using the two speech processing tools considered here: OpenSMILE [ 9 ] and Praat [ 16 ] to compare them

Read the complete text