The Interspeech 2025 Speech Accessibility Project Challenge

Xiuwen, Bornali, Jonghwan, Ed, Kyu, Mark, Pan-Pan, Aadhrik, Colin, Bob, Gautam, Venkatesh, Leda, Katrin, Chang D, Chris · 2025

Read The Interspeech 2025 Speech Accessibility Project Challenge by Xiuwen, Bornali, Jonghwan, Ed, Kyu, Mark, Pan-Pan, Aadhrik, Colin, Bob, Gautam, Venkatesh, Leda, Katrin, Chang D, Chris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22047.txt.

Introduction

Xiuwen Zheng and co-authors report on a 2025 challenge to improve speech recognition for people with speech disabilities. Automatic speech systems have advanced fast, yet they still serve disabled speakers poorly, partly because public training data is scarce. The challenge used a project data set of more than 400 hours of speech from over 500 people.

Teams were scored on word error rate and on a meaning-based score. Of 22 valid teams, 12 beat a strong standard system on errors, and 17 beat it on meaning. The top team led on both scores at once. The results set new benchmarks for recognizing impaired speech. The challenge was hosted on an open evaluation platform.

Catalog record

CREATORXiuwen, Bornali, Jonghwan, Ed, Kyu, Mark, Pan-Pan, Aadhrik, Colin, Bob, Gautam, Venkatesh, Leda, Katrin, Chang D, Chris
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Interspeech 2025 Speech Accessibility Project Challenge Xiuwen, Bornali, Jonghwan, Ed, Kyu, Mark, Pan-Pan, Aadhrik, Colin, Bob, Gautam, Venkatesh, Leda, Katrin, Chang D, Chris Abstract While the last decade has witnessed significant advancements in Automatic Speech Recognition (ASR) systems, performance of these systems for individuals with speech disabilities remains inadequate, partly due to limited public training data. To bridge this gap, the 2025 Interspeech Speech Accessibility Project (SAP) Challenge was launched, utilizing over 400 hours of SAP data collected and transcribed from more than 500 individuals with diverse speech disabilities. Hosted on EvalAI and leveraging the remote evaluation pipeline, the SAP Challenge evaluates submissions based on Word Error Rate and Semantic Score. Consequently, 12 out of 22 valid teams outperformed the whisper-large-v2 baseline in terms of WER, while 17 teams surpassed the baseline on SemScore. Notably, the top team achieved the lowest WER of 8.11%, and the highest SemScore of 88.44% at the same time, setting new benchmarks for future ASR systems in recognizing impaired speech. Zheng Phukon Na Cutrell Han Hasegawa-Johnson Jiang Kuila Lea MacDonald Mantena Ravichandran Sari Tomanek Yoo Zwilling UIUC Microsoft Amazon Google Apple MetaUSA CDLIUK Inha University KAISTKR keywords SAP challenge, ASR, accessibility, dysarthria † † email: {xiuwenz2, bornalip, jhasegaw, zwillin1}@illinois.edu, {cutrell, aadhrikkuila}@microsoft.com, {kyujhan, veravic}@amazon.com, {pjiang, bmacdonald}@google.com, {colin.lea, gmantena}@apple.com, [email protected], [email protected], [email protected], [email protected] 1 Introduction Automatic Speech Recognition (ASR) has witnessed remarkable advancements in recent years, primarily driven by the development of deep neural networks (DNN) and the explosive growth of training data. End-to-end ASR systems, in particular, leveraging self-supervised learning (SSL) or large-scale weakly-supervised learning techniques, have become the standard paradigm which achieved significant improvements in recognition accuracy and overall performance. These progressions have enabled ASR technologies to become increasingly integrated into various applications. However, a notable performance gap remains when recognizing speech from individuals with disabilities compared to individuals without disabilities. As reported in [ 1 ] , between 2008 and 2023, the lowest error rates for dysarthric ASR decreased by a factor of three, while for individuals without dysarthria, the error rates decreased by a factor of five. One primary cause of this disparity is the limited availability of large-scale, diverse data for the standardized and repeatable training and testing of ASR models against dysarthric speech: The largest publicly available English corpus meeting these criteria from 2008 to 2023 was the UA-Speech [ 2 ] corpus, which includes 22 hr of speech by 16 speakers. To address the scarcity of specialized data that prevents ASR systems from effectively generalizing and adapting to impaired speech, the Speech Accessibility Project (SAP [ 1 ] ) is systematically collecting, transcribing, and distributing U.S., Canadian, and Puerto Rican English speech from individuals with speech disorders. This effort aims to prompt research and development in impaired speech recognition, thereby enhancing ASR technology for people with diverse speech patterns and disabilities. In alignment with these objectives, we launched the Interspeech 2025 Speech Accessibility Project (SAP) Challenge, aimed at rapidly advancing the state of the art in dysarthric speech recognition. To the best of our knowledge, this is the first impaired speech recognition challenge utilizing a large-scale speaker-independent corpus. The challenge utilizes the partial SAP dataset released on April 30, 2024, containing over 400 hours of speech data collected from more than 500 individuals with disabilities. For the remainder of this paper, we will refer to this dataset as SAP-240430. We host the SAP challenge 1 1 1 https://eval.ai/web/challenges/challenge-page/2362 through EvalAI [ 3 ] . To protect data privacy, we utilize EvalAI’s remote evaluation pipeline, ensuring that evaluations are conducted on a private server provided by The National Center for Supercomputing Applications (NCSA), which securely stores the test data. Competition model submissions are assessed based on two primary metrics: Word Error Rate (WER) and Semantic Score (SemScore). WER serves as a traditional measure of ASR accuracy by quantifying the discrepancies between the recognized words and the ground truth transcripts. SemScore, on the other hand, evaluates the semantic fidelity of the transcriptions, assessing the degree to which the transcription preserves the intended meaning and context. We divided the test data into two subsets: Test1 and Test2. Results from Test1 are used to benchmark participants on a public leaderboard, while results from Test2 remain hidden on a private leaderboard until the competition concludes. Final team rankings are based on the outcomes from the private leaderboard. As a result, the SAP Challenge received valid results from 22 teams, 12 of which surpassed the baseline (whisper-large-v2) on WER, and 17 showed better SemScore. The leading team attained the lowest WER of 8.11%, as well as the highest Semscore of 88.44%, establishing new benchmarks for future ASR systems in the domain of impaired speech recognition. 2 Data, Metrics, and Baseline 2.1 Data In this section, we provide a comprehensive overview of the SAP-240430 dataset utilized in the SAP Challenge. Comprising approximately 415 hours of impaired speech, the dataset includes contributions from 524 participants diagnosed with one of the following five etiologies: Parkinson’s Disease (PD), Down Syndrome (DS), amyotrophic lateral sclerosis (ALS), cerebral palsy (CP), or stroke. Contributors to the corpus were recruited by the University of Illinois, in collaboration with patient advocacy organizations; all procedures were approved by the Institutional Review Board at the University of Illinois. Speech for this competition was primarily recorded during the period April through December 2023, when the Speech Accessibility Project was only recruiting participants with PD, therefore the dataset for this competition was dominated by speech from people with Parkinsons Disease: 73.4% of the training corpus, and 75.9% of the test corpus, by speech duration, were recorded by people with PD. The most common type of dysarthria for people with PD is hypokinetic dysarthria [ 4 ] , which is characterized by constant low energy levels, flat pitch, articulatory reductions, and voicing that is often breathy or even whispered [ 5 , 6 ] . Some speakers, however, exhibited speech patterns that differed markedly from the norm, including seven speakers who produced extremely long sequences of stutter events, and others who exhibited spasticity, extremely rapid speech, and extremely slow speech. The dataset is partitioned into Training, Development, and Test sets following a 70:10:20 ratio, ensuring that each split contains unique speakers with no overlaps on the speaker-level. Participants were selected for dev and test sets with equal probability regardless of etiology. Furthermore, the Test set is subdivided into two equal parts, Test1 and Test2. During inference, we employ the “unshared” test subsets, which excludes any utterances whose text also appears in the training data. As outlined in section 1 , the evaluation results from the Test1 “unshared” subset are publicly available for benchmarking purpose, while the final rankings of participants are determined based on the results from the Test2 “unshared” subset. Table 1: Statics about the (processed) SAP-240430 dataset for each split, including the number of speakers (# of speakers), number of utterances (# of utterances), and the duration of speech data in hours (Duration ( \text{\,}\mathrm{h} )). Split # of speakers # of utterances Duration ( \text{\,}\mathrm{h} ) Train 369 131,420 290.35 Dev 55 19,275 43.56 Test1 50 18,397 42.16 unshared 50 7,601 25.17 Test2 50 17,752 38.77 unshared 50 8,043 23.39 Participants are provided with the unprocessed

Read the complete text