SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies

Jiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet Vonk, Brittany Morin, David Baquirin, Zachary Miller, Maria Luisa Gorno Tempini, Gopala Anumanchipalli · 2024

Read SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies by Jiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet Vonk, Brittany Morin, David Baquirin, Zachary Miller, Maria Luisa Gorno Tempini, Gopala Anumanchipalli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00265.txt.

Introduction

Jiachen Lian and co-authors present a 2024 paper on transcribing speech with all its stumbles kept in. Real speech holds pauses, repeats, and mispronunciations, and rich transcripts should mark them with accurate timing. Their earlier system, SSDM, was complex to train and weak at aligning sounds to text.

The new version, SSDM 2.0, rebuilds four parts. A neural flow builds scalable speech features, and a new aligner captures every kind of disfluency. A prompting scheme teaches a language model to learn pronunciations in context. The team also releases two datasets of dysfluent speech, one the largest of its kind. Tests on speech from patients with a language disorder beat the old system and all rivals.

Catalog record

CREATORJiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet Vonk, Brittany Morin, David Baquirin, Zachary Miller, Maria Luisa Gorno Tempini, Gopala Anumanchipalli
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SSDM 2.0: Time-Accurate Speech Rich Transcription with Non-Fluencies Jiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet Vonk, Brittany Morin, David Baquirin, Zachary Miller, Maria Luisa Gorno Tempini, Gopala Anumanchipalli Abstract Speech is a hierarchical collection of text, prosody, emotions, dysfluencies, etc. Automatic transcription of speech that goes beyond text (words) is an underexplored problem. We focus on transcribing speech along with non-fluencies (dysfluencies). The current state-of-the-art pipeline ( Lian et al. 2024 ) suffers from complex architecture design, training complexity, and significant shortcomings in the local sequence aligner, and it does not explore in-context learning capacity. In this work, we propose SSDM 2.0, which tackles those shortcomings via four main contributions: (1) We propose a novel neural articulatory flow to derive highly scalable speech representations. (2) We developed a full-stack connectionist subsequence aligner that captures all types of dysfluencies. (3) We introduced a mispronunciation prompt pipeline and consistency learning module into LLM to leverage dysfluency in-context pronunciation learning abilities. (4) We curated Libri-Dys ( Lian et al. 2024 ) and open-sourced the current largest-scale co-dysfluency corpus, Libri-Co-Dys , for future research endeavors. In clinical experiments on pathological speech transcription, we tested SSDM 2.0 using nfvPPA corpus primarily characterized by articulatory dysfluencies . Overall, SSDM 2.0 outperforms SSDM and all other dysfluency transcription models by a large margin. See our project demo page at https://berkeley-speech-group.github.io/SSDM2.0/ . Figure 1: SSDM 2.0 Demo. Audios available at https://shorturl.at/ITUu0 . 1 Introduction 1 1 footnotetext: Terms Dysfluency, Non-fluency, dysfluency are interchangeable. Current automatic speech recognition (ASR) systems ( Radford et al. 2023 ) and speech language models (SLMs) ( Wu et al. 2024 ) typically transcribe speech into the words that were spoken (lexicalized speech) rather than how the words were spoken (uttered speech) . For example, when someone says P-Please c-call st-ah-lla , these systems usually perform denoising and output please call stella . This approach is suitable for most spoken dialogue scenarios and services, and helps reduce confusion in communications. However, in pathological speech domain, uttered speech is required to accurately identify articulation and pronunciation problems for diagnostic purposes. The gap between lexicalized speech and uttered speech is referred to as dysfluency , which includes repetition, deletion, insertion, replacement of sounds, filler words, and hesitations ( Lian et al. 2023c ) . Accurate transcription of speech dysfluencies could substantially reduce the workload of speech language pathologists while facilitating diagnosis and serving as a powerful clinical assessment tool. Pathological speech disorders are typically caused by neurological or physiological factors and are associated with various dysfluencies, such as motor and phonological (articulatory) dysfluencies (e.g., nfvPPA, Parkinson’s disease, Broca’s aphasia) and higher-order (or semantic) dysfluencies (e.g., svPPA, Wernicke’s aphasia, ASD). Diagnosing these disorders is challenging due to case-by-case variability and differences in severity. However, they share a common set of dysfluencies at the behavioral level, making it possible to develop a general dysfluency transcription system that can accommodate all disorders and support follow-up diagnosis. Due to data constraints, however, it is not feasible to test such a system on all pathological speech data. Therefore, we focus on nfvPPA, leveraging the currently available data to develop a dysfluency transcription tool specifically targeting articulation-based dysfluencies . Technically, SSDM ( Lian et al. 2024 ) is the first end-to-end pipeline that can transcribe both lexicalized speech and uttered speech (with dysfluencies) . However, it faces challenges in representation learning complexity, limited dysfluency alignment coverage, and minimal performance boost from language modeling. Technically, there are four questions to address: (1) What are the most scalable speech representations? (2) How to align dysfluent speech with text? (3) How to curate large-scale dysfluent speech data with time-aware annotations? (4) How to leverage pronunciation in-context learning from large language models? We aim to address the aforementioned challenges and present SSDM 2.0 . Our key contributions are as follows: • We propose Neural Articulatory Flow , which encodes a semi-implicit speech representation that has been shown to be the most scalable dysfluency-aware representation, inspired by articulatory dysfluencies in pathological speech disorders. • We develop Full-Stack Connectionist Subsequence Aligner (FCSA) , achieving comprehensive coverage of dysfluency alignments and precise distribution estimation. • We curate and opensource Libri-Dys-Co , the largest simulated speech co-dysfluency corpus, featuring over 6,000 hours of time-aware dysfluency annotations (word/phoneme). • We introduce Mispronunciation In-Context Learning and Consistency Learning in langauge model to achieve zero-shot dysfluency transfer and joint fluent-dysfluent ASR tasks transfer. SSDM 2.0 significantly and consistently outperforms all existing methods, including speech language models, and can serve as a foundational dysfluency modeling tool. 2 SSDM 2.0 Overview SSDM 2.0 (shown in Fig. 2 ) processes dysfluent speech and a textual prompt as inputs, generating pronunciation transcriptions as output. To achieve this, we propose the Neural Articulatory Flow (NAF) , which generates scalable speech representations referred to as gestural scores . Subsequently, the gestural scores are aligned with the reference text using a Full-stack Connectionist Subsequence Aligner , producing aligned speech embeddings for each token in the reference text. These aligned embeddings, combined with pre-defined prompts, are then input into a LLaMA ( Touvron et al. 2023 ) module for instruction tuning ( Gong et al. 2023b ) , a process we term Non-fluency In-context Learning . The following subsections provide a detailed explanation of each module. 3 Neural Articulatory Flow Current speech representation modeling typically uses explicit D\times T matrices, where T represents time and D represents channel dimension, and these are learned densely in a data-driven manner. However, human speech is produced by a sparse set of articulators with sparse activation in time. If we define basic moving patterns of articulators as a dictionary and project articulatory data into this dictionary space, we obtain a sparse activation matrix. The dictionary is called gestures and the sparse activation matrix is called gestural scores ( Browman & Goldstein 1992 ) . This motivates us to ask: instead of densely learning elementwise speech representations, can we sparsely and implicitly learn such structural speech representations via human-prior rules? At the same time, as we focus on articulatory dysfluencies , modeling speech production processes helps identify the specific articulation problems in the gestural space and brings the representation closer to real human dysfluent speech. In this work, we propose Neural Articulatory Flow , which involves a Semi-implicit Speech Encoder (Section 3.1 ) to predict only the indices of active regions in articulation, and an Articulatory Flow (Section 3.2 ) to distill speech intelligibility from pretrained acoustic speech embeddings (WavLM ( Chen et al. 2022 ) ). We also optionally introduce Interpretable Posterior Training to visualize how speech is physically produced in articulation and to derive articulatory feedback for pronunciation. Figure 2: SSDM 2.0 architecture 3.1 Semi-Implicit Speech Encoder Given a speech waveform, we adopt UAAI ( U niversal A coustic-to- A rticulatory I I nversion) ( Cho et al. 2024 ) to obtain its articulatory trajectory X\in\mathbb{R}^{D\times T} at 50Hz. UAAI consists of a WavLM ( Chen et al. 2022 ) encoder followed by a linear layer, which takes speech waveform as input and predicts articulatory trajectories. The model was trained on the MNGU0 corpus ( Richmond et al. 2011 ) , which contains 75 minutes of EMA (Electromagnetic midsagittal articulography) data collected during newspaper

Read the complete text