A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data

Minh Tran, Yutong Pang, Debjyoti Paul, Laxmi Pandey, Kevin Jiang, Jinxi Guo · 2025

Read A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data by Minh Tran, Yutong Pang, Debjyoti Paul, Laxmi Pandey, Kevin Jiang, Jinxi Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12501.txt.

Introduction

Speech recognizers fail on new topics for lack of training voices, and a 2025 paper by Minh Tran and co-authors adapts them with fake data only. Their framework, DAS, never touches real recordings from the target domain.

First, a language model writes texts in the style of the new domain, such as music, weather, or sports talk. A speech synthesizer then reads those texts aloud, and the fake audio fine-tunes the Whisper recognizer through small adapter modules. A one-pass decoding trick merges answers from several adapters at once. Word error rates fell by ten to seventeen percent in the target domains, with barely any loss elsewhere. The method adds only a small run-time cost.

Catalog record

CREATORMinh Tran, Yutong Pang, Debjyoti Paul, Laxmi Pandey, Kevin Jiang, Jinxi Guo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12501

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data Thanks: This work was done during an internship at Meta. Minh Tran, Yutong Pang, Debjyoti Paul, Laxmi Pandey, Kevin Jiang, Jinxi Guo Abstract We introduce DAS (Domain Adaptation with Synthetic data), a novel domain adaptation framework for pre-trained ASR model, designed to efficiently adapt to various language-defined domains without requiring any real data. In particular, DAS first prompts large language models (LLMs) to generate domain-specific texts before converting these texts to speech via text-to-speech technology. The synthetic data is used to fine-tune Whisper with Low-Rank Adapters (LoRAs) for targeted domains such as music, weather, and sports. We introduce a novel one-pass decoding strategy that merges predictions from multiple LoRA adapters efficiently during the auto-regressive text generation process. Experimental results show significant improvements, reducing the Word Error Rate (WER) by 10 % 10\% to 17 % 17\% across all target domains compared to the original model, with minimal performance regression in out-of-domain settings − 1 % -1\% on Librispeech test sets. We also demonstrate that DAS operates efficiently during inference, introducing an additional 9 % 9\% increase in Real Time Factor (RTF) compared to the original model when inferring with three LoRA adapters. I Introduction Current state-of-the-art speech recognition systems are trained on extensive and varied datasets. For example, the Whisper model [ 1 ] underwent training with 680,000 hours of multilingual data, encompassing a wide array of audio conditions and environments. This extensive training strategy has enabled these models to express a high level of robustness in speech recognition across diverse conditions and multiple evaluation datasets [ 1 ] . Such robustness motivates us to question whether domain-specific data distribution, especially transcript distribution over language characterizing a given topic such as music or sports, are still relevant. Are state-of-the-art speech recognition systems universally effective, or is there still benefits in tailoring these robust models to specific domains? In this paper, we explore this research question using Whisper, one of the most widely utilized and state-of-the-art automatic speech recognition (ASR) models. In particular, we introduce a domain adaptation framework, which we refer to as DAS . The framework is designed to efficiently adapt Whisper [ 1 ] to various language-defined domains, i.e., speech with corresponding transcripts characterizing a given topic or genre. DAS is characterized by two key features: 1) it does not require any real data (audio or text) for adaptation to a target domain, thereby eliminating the substantial costs of human annotations and enhancing scalability, and 2) it maintains effective adaptation across different domains without compromising its generalizability in out-of-domain conditions. To achieve the first goal, we prompt large language models (LLMs) to generate texts that are relevant to a target domain. Subsequently, we employ text-to-speech to convert the generated text into corresponding audio data, and use this data to fine-tune the Whisper model specifically for that domain. To fulfill the second goal, we explore the use of a single encoder with multiple decoders, with each decoder dedicated to a specific domain. To mitigate the additional computational costs associated with multiple decoders, we employ LoRA (Low-Rank Adaptation) adapters. Additionally, we propose a novel one-pass decoding strategy that efficiently merge predictions from multiple LoRA adapters during each step of the auto-regressive text generation process. Experimental results demonstrate that our model consistently achieves a reduction ranging from 10\% to 17\% in the Word Error Rate metric (WER) over the original Whisper model on real speech across three target domains: music, weather/datetime, and sports. Notably, the model exhibit minimal performance regression in out-of-domain settings on multiple public benchmarks while introducing a modest latency increase of 9\% in the Real Time Factor (RTF) metric compared to the original model. In summary, our contributions include: • We provide a novel and thorough analysis on the effect of adapting ASR systems to language-defined domains using only synthetic audio/text data. • We propose DAS , a novel domain adaptation framework for speech recognition systems, that effectively and efficiently adapt a base model to a set of target domains without sacrificing generalizabilty. • We provide extensive experiments and ablation study to demonstrate the effectiveness of DAS . II Related work Fig. 1: An overview of the proposed domain adaptation framework. We first use LLM to generate texts related to a particular domain of interest ( e.g., music) and then use text-to-speech (TTS) to generate synthetic speech. We then fine-tune the decoder of Whisper with the generated synthetic data using Low-rank Adaptation (LoRA) [ 2 ] . Finally, we propose DAS to effectively generate predictions with multiple LoRAs to enhance in-domain speech recognition performance with minimal regression in out-of-domain settings. Existing work on domain adaptation for ASR mainly focus on unsupervised domain adaptation settings, where both source and target domain speech are available but word transcripts are only available in the source domain. Sun et al. [ 3 ] proposed using domain adversarial learning with Gradient Reversal Layer [ 4 ] on deep neural networks to handle noise robustness or speech with different genres. Hsu et al. [ 5 ] propose a data augmentation method based on latent features extracted from variational autoencoders learned from both source and target domain speech. Meng et al. [ 6 ] extends Teacher/Student (T/S) learning to large-scale unsupervised domain adaptation of an end-to-end (E2E) model by using teacher’s token posteriors as soft labels and one-best predictions for decoder guidance. Samarakoon et al. [ 7 ] explore domain adaptation for ASR in low-resource setting by adding domain labels to the label sequences, using factorized hidden layer adaptation and a novel domain-specific gating mechanism. Sim et al. [ 8 ] propose using factorized hidden layer for LSTM. Sim et al. [ 9 ] present an extensive study of different parameter-efficient fine-tuning methods for domain adaptation, and show the superior performance of Residual Adapters [ 10 ] and Low-rank Adapters (LoRA) [ 2 ] . Zheng et al. [ 11 ] explore the use of synthetic speech to enhance recognition performance on out-of-vocabulary words. In contrast, our work focuses on domain adaptation, aiming to increase the likelihood of specific vocabulary distributions being accurately predicted. Most relevant to our work, Huang et al. [ 12 ] fine-tune BART on a seed set of in-domain texts to generate domain-relevant texts. They use TTS-generated speech from these texts to augment the LibriSpeech dataset and train an RNN-T based model [ 13 ] from scratch, demonstrating significant improvement in in-domain ASR performance. Compared to existing work, our main contributions are: 1) DAS does not assume any available in-domain text or real audio data, allowing scalability; 2) unlike [ 12 ] , we focus on adapting a pre-trained ASR model and use only synthetic speech for adaptation rather than treating it as augmentation data alongside real speech data; and 3) we are the first to explore strategies to avoid performance regression for out-of-domain speech for ASR systems. III Method Problem Formulation. Given a pre-trained ASR model \mathcal{M} and a set of language-defined domains \{d_{1},\dots,d_{k}\} with corresponding real speech test sets \{\mathcal{D}_{1},\dots,\mathcal{D}_{k}\} , our goal is to produce a new model \mathcal{M^{\prime}} from \mathcal{M} such that \mathcal{M^{\prime}} achieves better performance (lower Word-Error-Rate) than \mathcal{M} across all in-domain test sets

Read the complete text