Evaluating pretrained speech embedding systems for dysarthria detection across heterogenous datasets

Lovisa Wihlborg, Jemima Goodall, David Wheatley, Jacob J. Webber, Johnny Tam,, Christine Weaver, Suvankar Pal, Siddharthan Chandran,, Sohan Seth, Oliver Watts, Cassia Valentini-Botinhao · 2025

Read Evaluating pretrained speech embedding systems for dysarthria detection across heterogenous datasets by Lovisa Wihlborg, Jemima Goodall, David Wheatley, Jacob J. Webber, Johnny Tam,, Christine Weaver, Suvankar Pal, Siddharthan Chandran,, Sohan Seth, Oliver Watts, Cassia Valentini-Botinhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19946.txt.

Introduction

Slurred, hard-to-parse speech from conditions such as cerebral palsy or Parkinson's is called dysarthria, and detecting it automatically could speed referrals. Lovisa Wihlborg and ten co-authors grade seventeen ready-made speech embedding systems on that task. Six datasets of varied origin supply the tests. Small, skewed recordings make naive scores untrustworthy, so repeated cross-validation runs are compared against a crafted null model.

Only scores that beat the null distribution count as real detection. Within one dataset, results swung widely by dataset, whatever the system. Training on one dataset and testing on another exposed still sharper drops. The study is a reality check for a field short on data. Clinics need detectors that travel across recording setups, not just leaderboards.

Catalog record

CREATORLovisa Wihlborg, Jemima Goodall, David Wheatley, Jacob J. Webber, Johnny Tam,, Christine Weaver, Suvankar Pal, Siddharthan Chandran,, Sohan Seth, Oliver Watts, Cassia Valentini-Botinhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19946

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating pretrained speech embedding systems for dysarthria detection across heterogenous datasets Lovisa Wihlborg, Jemima Goodall, David Wheatley, Jacob J. Webber, Johnny Tam,, Christine Weaver, Suvankar Pal, Siddharthan Chandran,, Sohan Seth, Oliver Watts, Cassia Valentini-Botinhao Abstract We present a comprehensive evaluation of pretrained speech embedding systems for the detection of dysarthric speech using existing accessible data. Dysarthric speech datasets are often small and can suffer from recording biases as well as data imbalance. To address these we selected a range of datasets covering related conditions and adopt the use of several cross-validations runs to estimate the chance level. To certify that results are above chance, we compare the distribution of scores across these runs against the distribution of scores of a carefully crafted null hypothesis. In this manner, we evaluate 17 publicly available speech embedding systems across 6 different datasets, reporting the cross-validation performance on each. We also report cross-dataset results derived when training with one particular dataset and testing with another. We observed that within-dataset results vary considerably depending on the dataset, regardless of the embedding used, raising questions about which datasets should be used for benchmarking. We found that cross-dataset accuracy is, as expected, lower than within-dataset, highlighting challenges in the generalization of the systems. These findings have important implications for the clinical validity of systems trained and tested on the same dataset. † † address: 1 SpeakUnique Ltd., UK, 2 Anne Rowling Regenerative Neurology Clinic, University of Edinburgh (UoE), UK, 3 Institute of Adaptive and Neural Computation, UoE, UK, 4 Euan MacDonald Centre for MND Research, UoE, UK, 5 UK Dementia Research Institute, UK 1 Introduction Dysarthria is a motor speech disorder characterised by disordered articulation (slurred speech) and caused by damage to the nervous system, which leads to paralysis, slowing, or incoordination of the muscles of articulation. Causes include neurological disorders such as stroke, amyotrophic lateral sclerosis (ALS), Parkinson’s disease (PD) and developmental disorders such as cerebral palsy (CP). In particular, neurodegenerative disorders are a significant and increasing health burden globally, with urgent and unmet needs for accessible and accurate diagnostic and monitoring methods [ 1 ] . Speech has great potential to provide a non-invasive, remotely collected, and objective digital biomarker in these conditions. Speech processing and machine learning methods have recently shown great promise in both disease detection and severity prediction from only the speech signal [ 2 , 3 ] . However, clinical translation and adoption of these methods will depend on robust evaluation of speech processing methods and clinical validation. The success of speech embedding systems trained in a generic way when applied to specific tasks has led to an interest in applying them also to speaker health evaluation [ 4 ] . These systems are relatively easy to engineer as they rely on publicly available tools that often do not require in-depth expertise of either the task or the data. However, it is challenging to reliably evaluate their performance on health monitoring tasks. Results in this domain are often reported on closed test sets that cannot be shared for ethical and legal reasons. Where results are published using accessible data, their reliability is compromised by the small size and known artefacts of the available open datasets. Moreover, results are often presented without sufficient information about how data is preprocessed and partitioned, meaning that even when data can be accessed, it is doubtful whether results can be replicated. Several lines of work seek to remedy this situation. The creation of new larger and better open datasets (such as [ 5 ] ) is welcome, as are shared tasks that define firmer train-set partitions for existing datasets (e.g. [ 6 ] ). A separate line of work validates specific existing datasets and sheds light on the reliability of results obtained from them. Datasets can be flawed in many different ways, and two particular issues are to do with data balance and coverage. Data must be balanced in such a way that a system is incapable of learning to discriminate between underlying classes using factors that are not relevant to the task in a generalisable way. Balance is achieved by ensuring similar data acquisition procedures across classes, both speaker-intrinsically in terms of the demographics of the participants (age, gender, accent) and also extrinsically – in the case of audio data, for example, ensuring that no consistent differences of recording setup (microphone, environment) correlate with the classes or health measures to be detected. In line with these considerations we present a large and comprehensive evaluation of systems for dysarthria detection. Rather than relying on one or two potentially problematic datasets, we use multiple datasets covering related conditions. Furthermore we used repeated k -fold cross-validation, varying fold splits across runs. To verify that results are significantly above chance level we contrast the distribution of scores across these runs to a null hypothesis distribution of each feature–task combination. We evaluate 17 speech embedding systems across six different datasets and present cross-validation within-dataset performance of each system on each dataset. We also present cross-dataset results using a selection of closely related datasets. These results are obtained by training systems on one complete dataset and testing on another. Table 1: Datasets used in evaluation. Selected conditions: healthy control (HC), cerebral palsy (CP), Parkinson’s disease (PD), Huntingon’s disease (HD), amyotrophic lateral sclerosis (ALS), ∗ refers to: peripheral neuropathy, myopathic or myasthenic lesions. Access: academic non-profit purposes (A), commercial and non-commercial (C+NC). #Speakers after balancing. Dataset Selected conditions Language #Speakers (HC) Elicitation tasks Access Preparation EWA [ 7 ] HC/PD Slovak 190 (95) Mixed free license (C+NC) Balanced for class EasyCall [ 8 ] HC/PD/HD/ALS/ ∗ Italian 45 (24) Commands CC BY-NC Balanced for class Neurovoz [ 9 ] HC/PD Spanish 108 (55) Mixed CC BY-NC-ND 4.0 - SSNCE [ 10 ] HC/CP Tamil 20 (10) Mixed free license (A) Balanced for class TORGO [ 11 ] HC/CP/ALS N.A. English 15 (7) Mixed free license (A) - UASpeech [ 12 ] HC/CP N.A. English 28 (13) Word repetition CC BY 4.0 - 2 Background 2.1 Datasets of dysarthric speech Many datasets containing recordings of dysarthric speech have been reported, and several can be obtained relatively straightforwardly for research [ 5 , 7 , 8 , 9 , 10 , 11 , 12 ] . They differ in terms of conditions (what caused the impairment), language, number of speakers, amount of data collected per speaker, type of elicitation tasks and access. Motivations for creating and releasing such datasets include, among other things, the improvement of automatic speech recognition (ASR) of dysarthric speech [ 5 , 11 ] and automatic diagnosis [ 7 , 12 ] . Speech datasets can be imbalanced in terms of speaker characteristics (i.e. age, gender, accent and health condition) and speaker-extrinsic factors such as recording conditions (microphone, environment, data format). To ensure that a system discriminates between underlying classes using only task-relevant factors that are generalisable, it is crucial to understand and address these imbalances where possible. While some of the biases can be addressed straightforwardly (e.g. by choosing a subset of the dataset with equal numbers of males and females) others are harder to revert without loss of performance (e.g. removing noise and reverberation). The authors in [

Read the complete text