Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech

Karl, Enno, Sevada, Mathew · 2025

Read Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech by Karl, Enno, Sevada, Mathew (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01618.txt.

Introduction

Karl El Hajal, Enno Hermann, Sevada Hovsepyan, and Mathew Magimai-Doss wrote this 2025 study on making speech software work better for people with dysarthria, a condition that slurs and slows speech. Speech tools fail on such speech because speakers vary a lot and speak slowly. The team tested turning dysarthric speech into healthy-sounding speech before the tool listens.

Their method extends a rhythm and voice conversion frame with a syllable-based model of rhythm shaped for this speech type. On a standard test set, one speech system cut its word error rate a great deal after conversion, above all for the most severe cases. A large popular tool, fine-tuned on the converted speech, barely moved. The authors read the split as proof that conversion helps most where the base tool is weakest.

Catalog record

CREATORKarl, Enno, Sevada, Mathew
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech Karl, Enno, Sevada, Mathew Abstract Automatic speech recognition (ASR) systems struggle with dysarthric speech due to high inter-speaker variability and slow speaking rates. To address this, we explore dysarthric-to-healthy speech conversion for improved ASR performance. Our approach extends the Rhythm and Voice (RnV) conversion framework by introducing a syllable-based rhythm modeling method suited for dysarthric speech. We assess its impact on ASR by training LF-MMI models and fine-tuning Whisper on converted speech. Experiments on the Torgo corpus reveal that LF-MMI achieves significant word error rate reductions, especially for more severe cases of dysarthria, while fine-tuning Whisper on converted data has minimal effect on its performance. These results highlight the potential of unsupervised rhythm and voice conversion for dysarthric ASR. Code available at: https://github.com/idiap/RnV . El Hajal Hermann Hovsepyan Magimai.-Doss CH-1920 MartignySwitzerland CH-1015 LausanneSwitzerland keywords Rhythm Modeling, Voice Conversion, Unsupervised, Dysarthric Speech Recognition † † email: {karl.elhajal,enno.hermann,sevada.hovsepyan,mathew}@idiap.ch 1 Introduction Motor speech impairments like dysarthria can significantly hinder communication by affecting multiple aspects of speech production, including rhythm and articulation [ 1 ] . As a result, Automatic Speech Recognition (ASR) systems trained on typical speech often struggle to process dysarthric speech accurately [ 2 ] . This creates a need for specialized ASR systems that accommodate the unique speech patterns of individuals with dysarthria. However, developing such systems is challenging for two main reasons. First, dysarthric speech deviates significantly from typical speech patterns, and exhibits substantial inter-speaker variability. Speech recognition models particularly struggle with the slower speaking rates common in dysarthria. Second, the scarcity of dysarthric speech data complicates development, as collecting speech samples can be physically demanding for individuals with dysarthria. Recent studies have explored promising avenues such as the use of synthesis and conversion methods for data augmentation and dysarthric-to-healthy speech conversion [ 3 , 4 , 5 ] . Among these, unsupervised techniques leveraging Self-Supervised Learning (SSL) speech representations have demonstrated promise due to their zero-shot capabilities and low data requirements. In previous work, we presented an unsupervised Rhythm and Voice (RnV) conversion framework [ 6 ] that modifies dysarthric speech to resemble healthy speech, showing promise in improving ASR performance. Indeed, evaluations on the Torgo corpus [ 7 ] showed that rhythm conversion was particularly beneficial for speakers with more severe dysarthria, improving ASR performance on models trained solely on healthy speech. While RnV enables unsupervised and data-efficient dysarthric speech conversion, the rhythm modeling approach used was not specifically adapted to dysarthric speech, resulting in imprecise segmentation and rhythm modification. Furthermore, while ASR performance improved, it remained unsatisfactory, and the impact of training or adapting ASR models on converted speech was not explored. Input Speech SSL Encoder Rhythm Converter kNN-VC Vocoder Converted Speech Source Rhythm Model Target Rhythm Model Target Unit Database Figure 1 : RnV Framework Overview To address these limitations, this paper explores a syllable-based rhythm modeling approach aimed at improving segmentation and adaptation for dysarthric speech. We investigate the impact of training and adapting ASR models on converted speech by training an LF-MMI [ 8 ] model and fine-tuning Whisper [ 9 ] , providing a comparative analysis of their performance. 2 Background The RnV framework (Fig. 1 ) converts dysarthric speech into healthy speech in unsupervised fashion by leveraging properties of self-supervised speech representations [ 6 ] . Rhythm conversion is achieved through a modified version of Urhythmic [ 10 ] , replacing soft units with discrete speech representations and extending the any-to-one conversion approach to any-to-any using a general-purpose vocoder. This method first segments speech into three types: silences, sonorants, and obstruents. This is done by clustering discrete units from a speech dataset into 100 centroids, which are then hierarchically grouped. The group with the greatest overlap with silences is labeled as Silence, the one most associated with voiced sections as Sonorants, and the remaining type as Obstruents. Segmentation is performed by comparing each discrete unit to the centroids and computing its log-probability of belonging to each class. A dynamic programming algorithm then merges consecutive units into longer segments, with a parameter \gamma controlling segment length. This segmentation enables the calculation of speaking rate and duration distributions for each speech type per speaker. These can be used to modify rhythm through time-stretching, either at the utterance level (global) or at the level of individual segments and speech types (fine-grained). Voice conversion, on the other hand, employs kNN-VC [ 11 ] to map the phonetic content of the source speaker to the closest matching units of a target speaker. 3 Methods In this work, we extend the rhythm conversion module of the RnV framework by combining the unsupervised clustering-based method with syllable segmentation and modeling. We further train and adapt ASR models on the converted speech to assess more thoroughly whether conversion helps improve recognition performance. 3.1 Syllable-based rhythm modeling Syllables are fundamental to rhythm modeling in speech. Indeed, analyzing syllable durations helps characterize a speaker’s unique rhythmic patterns [ 12 ] . While time-aligned transcriptions are ideal for precise syllable segmentation and speaking rate calculation, alternative methods are needed when such transcriptions are unavailable. Prior work using the Urhythmic method estimates speaking rate by segmenting and counting sonorants per second, as sonorants serve as syllable nuclei [ 6 ] . However, obtaining precise sonorant segments proved challenging due to the unique characteristics of dysarthric speech. To address this, we aim to improve rhythm modeling by directly obtaining syllable segments in an unsupervised manner. The importance of syllable-level analysis, particularly segmentation and feature extraction, has been demonstrated in recent research on dysarthric speech detection [ 13 ] . Indeed, such approaches mirror human auditory processing, where sound is initially decomposed into frequencies by the cochlea and subsequently segmented into syllables by the cortex. Inspired by this work, we adopt a segmentation approach which leverages theta oscillations and the sonority envelope [ 14 ] . This method segments syllables based on either syllable onsets (valleys in the envelope) or syllable nuclei (typically vowels, represented by peaks in the envelope). By extracting peak and valley timestamps, we can define syllable segments as either valley-to-valley or peak-to-peak intervals. While effective, this method struggles with noisy speech, as noise can introduce false peaks and valleys. Given the presence of noise in the data which we will be using for evaluation, we incorporate a filtering step using the initial discrete unit clustering method. This allows us to ignore segments that fall outside speech regions. Specifically, we define speech segments as the combined obstruent and sonorant regions, while silence segments are classified as non-speech. This segmentation acts as a form of Voice Activity Detection (VAD), helping to distinguish speech from noise. In preliminary experiments, we observed that this approach was more robust than traditional VAD methods in identifying non-speech regions in the context of dysarthric speech. Any peak or valley within a non-speech region is discarded, improving segmentation accuracy. The segmentation steps are visualized in Figure 2 . After filtering, we focus on syllable nuclei (peaks), as they are less prone to false positives. By measuring peak-to-peak syllable durations for each speaker, we establish two rhythm modeling and conversion approaches: Global : We calculate the syllables-per-second rate for

Read the complete text