Steamroller : A Multi-Agent System for Inclusive Automatic Speech Recognition for People Who Stutter

Ziqi Xu, Yi Liu, Yuekang Li, Ling Shi, Kailong Wang, Yongxin Zhao · 2026

Read Steamroller : A Multi-Agent System for Inclusive Automatic Speech Recognition for People Who Stutter by Ziqi Xu, Yi Liu, Yuekang Li, Ling Shi, Kailong Wang, Yongxin Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10223.txt.

Introduction

Ziqi Xu and co authors build speech tools for people who stutter. Voice tools now gate much of daily life. They run aids, log ins, and work calls. But most were trained on fluent speech. They fail for the many people who stutter.

The team made Steamroller, a team of agents for this task. It aims to hear stuttered speech as it is. It should not force it into fluent norms. The work frames the gap as a form of lock out. It seeks speech tools that serve all users, not just the fluent.

Catalog record

CREATORZiqi Xu, Yi Liu, Yuekang Li, Ling Shi, Kailong Wang, Yongxin Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Steamroller : A Multi-Agent System for Inclusive Automatic Speech Recognition for People Who Stutter Ziqi Xu, Yi Liu, Yuekang Li, Ling Shi, Kailong Wang, Yongxin Zhao Abstract People who stutter (PWS) face systemic exclusion in today’s voice-driven society, where access to voice assistants, authentication systems, and remote work tools increasingly depends on fluent speech. Current automatic speech recognition (ASR) systems, trained predominantly on fluent speech, fail to serve millions of PWS worldwide. We present Steamroller , a real time system that transforms stuttered speech into fluent output through a novel multi-stage, multi-agent AI pipeline. Our approach addresses three critical technical challenges: (1) the difficulty of direct speech to speech conversion for disfluent input, (2) semantic distortions introduced during ASR transcription of stuttered speech, and (3) latency constraints for real time communication. Steamroller employs a three stage architecture comprising ASR transcription, multi-agent text repair, and speech synthesis, where our core innovation lies in a collaborative multi-agent framework that iteratively refines transcripts while preserving semantic intent. Experiments on the FluencyBank dataset and a user study demonstrates clear word error rate (WER) reduction and strong user satisfaction. Beyond immediate accessibility benefits, fine tuning ASR on Steamroller repaired speech further yields additional WER improvements, creating a pathway toward inclusive AI ecosystems. 1 Introduction Speech disorders affect 90-95% of the global population at various stages ( 8 ) , with stuttering being a prevalent neurodevelopmental disorder characterized by involuntary disruptions including blocks, prolongations, and repetitions. For people who stutter (PWS), these disruptions create cascading societal impacts: students may avoid classroom participation, professionals miss career opportunities requiring presentations or client interactions, and individuals withdraw from social situations to avoid stigma and misunderstanding. In our increasingly voice driven society, PWS face compounding discrimination as they cannot reliably use voice assistants for basic tasks, are excluded from voice authenticated services, and find themselves disadvantaged in remote work environments that rely heavily on video calls. Traditional interventions such as speech therapy and electronic fluency devices, while beneficial for some, often require years of commitment with variable outcomes and fail to address the immediate technological and social barriers that systematically marginalize PWS in education, employment, and civic participation. Despite advances in automatic speech recognition (ASR) through deep learning architectures ( 16 ; 37 ; 17 ; 5 ) , current systems deployed by major technology companies ( 15 ; 26 ; 18 ) systematically fail to accommodate stuttered speech. These systems, designed and tested primarily on fluent speech, result in recognition failures and exclusion from voice based interfaces ( 7 ; 11 ; 25 ) . While generative AI presents opportunities for real time speech processing that could potentially transcribe, repair disfluencies, and synthesize corrected speech, these capabilities remain largely unexplored for accessibility applications. This technological gap highlights the urgent need for inclusive ASR design that recognizes stuttering not as an edge case but as a critical accessibility requirement affecting millions worldwide. To address this societal and accessibility gap, we explore the development of a system that transforms stuttered audio into fluent, clear speech. Crucially, achieving these goals requires addressing three challenges: C#1 : Modifying speech at the content level remains highly challenging. In particular, converting streaming stuttered audio directly into fluent speech is difficult given the current limitations of speech modeling technologies. C#2 : Semantic loss may occur during the transcription process ( 23 ) . It is crucial to preserve the semantically correct portions of the original speech while selectively repairing segments that may contain semantic errors, ensuring that the final output accurately and faithfully conveys PWS’s intended message. C#3 : Since the system is designed for real-time communication, the entire repair process must be efficiently managed to maintain low latency. In this work, we introduce Steamroller , which adopts a three-stage pipeline to address the challenges: transcribing stuttered audio, repairing the text, and synthesizing fluent speech. Instead of direct conversion, we use an intermediate textual representation to improve controllability and reliability. For repair, we design a multi-agent system that refines transcription with semantic consistency. To support real-time use, we minimize latency in each stage without sacrificing quality. We first conducted experiments on the FluencyBank ( 29 ) dataset to evaluate repair effectiveness. Across ASR systems, Steamroller achieves 10.46% WER, 8.57% MER, and 11.40% WIL reduction. It also improves semantic similarity by 10% over original ASR transcriptions. Steamroller enhances FluencyBank with stutter-free references (validated by the dataset authors) and improves annotation quality for SEP-28K ( 23 ) . Fine-tuning ASR with repaired speech yields an additional 3% WER drop. We also conducted a 23-participant user study. In the close-ended task study, Steamroller reduces WER by 7–11% around for moderate/severe stuttering and 3% for mild cases. Participants highly recognized its repair effectiveness, with a satisfaction score of 4.30\pm 0.27 ( p\ll 0.01 ). The open-ended task study investigates real-time use with PWS, showing positive user feedback (NPS = 23.53, SUS = 66.9). Overall, our approach effectively repairs stuttering in speech, highlighting its potential as a real-time stuttering audio translator that enhances communication. These applications illustrate the versatility and impact of Steamroller in real-world scenarios. This paper makes the following contributions to the field of AI for Social Impact: • We formalize the challenge of repairing semantically distorted ASR transcripts for PWS and justify our multi-stage, multi-agent pipeline design through systematic trade-off analysis. • We present Steamroller , a system which combines ASR, LLMs, and speech synthesis via a novel multi-agent framework. It models collaborative reflection to resolve semantic inconsistencies in disfluent speech. • We conduct mixed-methods evaluation with 23 PWS participants, combining quantitative FluencyBank benchmark analysis ( 29 ) with qualitative feedback from conversational tasks. • We demonstrate dual utility as both an assistive tool for PWS and a framework for dataset enhancement, showing that fine-tuning ASR on our system’s output improves accuracy and enables more inclusive AI development. 2 Background and Related Work Stuttering and Speech Disorders Stuttering is a multifaceted fluency disorder that disrupts speech flow and hampers communication ( 2 ; 21 ) . Its origins remain unclear, involving both biological and psychological factors ( 35 ) , often leading to social isolation and emotional distress ( 12 ) . Speech therapy aims to improve fluency, reduce anxiety, and boost self-esteem ( 27 ; 10 ) . Stuttering is categorized into five types-blocks, prolongations, sound repetitions, word/phrase repetitions, and interjections ( 23 ) , each posing distinct challenges for ASR recognition. This indicates the persistent difficulties PWS face when using ASR in modern smart devices ( 7 ; 22 ) . Existing study ( 22 ) suggests that improved speech technology accuracy could encourage more frequent use by PWS. This gap motivates our investigation into ASR system assessment using generated stuttering audio. By doing so, we aim to improve ASR precision and promote ASR accessibility for PWS. ASR System Accessibility Enhancement The application of machine learning and deep learning to enhance ASR for PWS is gaining increasing attention. Recent studies have reviewed stuttering classification methods ( 32 ) , evaluated machine learning techniques for stuttering event detection ( 1 ; 31 ) , and explored multi-task and adversarial learning for improved feature extraction ( 33 ) . Deep neural networks have also advanced stutter detection (

Read the complete text