Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation

Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu Ho · 2026

Read Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation by Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu Ho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16920.txt.

Introduction

Yamashita and co-authors adapt speech recognition to new domains with synthetic data. Recognizers often degrade on domain specific speech because in domain data is scarce. Their framework has two parts. First, a language model pipeline creates varied domain text, filtered to balance diversity and term coverage. Second, phonetic respelling uses model written pseudo spellings to add pronunciation variety before synthesis. Across four domain datasets, the authors report drops in word error rate. They argue that term coverage plus realistic pronunciation change makes synthetic speech more useful.

Catalog record

CREATORNatsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu Ho
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24232
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16920

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu Ho Abstract End-to-end automatic speech recognition often degrades on domain-specific data due to scarce in-domain resources. We propose a synthetic-data-based domain adaptation framework with two contributions: (1) a large language model (LLM)-based text augmentation pipeline with a filtering strategy that balances lexical diversity, perplexity, and domain-term coverage, and (2) phonetic respelling augmentation (PRA), a novel method that introduces pronunciation variability through LLM-generated orthographic pseudo-spellings. Unlike conventional acoustic-level methods such as SpecAugment, PRA provides phonetic diversity before speech synthesis, enabling synthetic speech to better approximate real-world variability. Experimental results across four domain-specific datasets demonstrate consistent reductions in word error rate, confirming that combining domain-specific lexical coverage with realistic pronunciation variation significantly improves ASR robustness. † † address: Hitachi, Ltd. 1 Introduction End-to-end automatic speech recognition (ASR) systems have achieved remarkable progress in recent years, but they still suffer substantial performance degradation when applied to domain-specific data that differs from the training distribution [ 1 ] . Since collecting large amounts of target-domain text and speech can be costly, recent studies have explored generating domain-specific text using large language models (LLMs) and converting it into synthetic speech via text-to-speech (TTS) as a cost-effective approach for ASR domain adaptation [ 30 , 31 ] . However, the existing synthetic-data approaches face two key limitations: (1) insufficient domain-specific lexical diversity—these studies have primarily focused on increasing the amount of text without explicitly optimizing for domain-aware lexical diversity and coverage; and (2) lack of natural phonetic variability—synthetic speech generated via TTS lacks the pronunciation variations, errors, and idiosyncrasies found in real speech [ 2 ] . Existing acoustic-level augmentation methods (e.g., SpecAugment [ 21 ] ) mask parts of the spectrograms rather than introduce pronunciation variants and, when applied to uniformly rendered synthetic speech, can be detrimental in some setups [ 31 ] . (a) (b) Figure 1 : Proposed methods overview. Italics denote placeholders. To address these limitations, we propose a robust ASR domain adaptation framework that relies solely on synthetic data. Our approach makes two key contributions: (1) it enhances domain-specific lexical diversity via an LLM-based text augmentation pipeline, equipped with a novel filtering strategy that jointly maximizes type–token ratio (TTR), perplexity, and domain-specific vocabulary coverage; and (2) it introduces phonetic respelling augmentation (PRA), a novel method that leverages LLMs to generate orthographic pseudo-spellings reflecting realistic pronunciation variability. Unlike SpecAugment, which modifies acoustic features after synthesis, PRA injects phonetic diversity directly at the text stage, enabling synthetic speech to capture natural variations such as pronunciation errors, substitutions, and idiosyncrasies, while remaining fully compatible with standard TTS systems. Experimental results show that our LLM-based text augmentation pipeline improves recognition accuracy across multiple domain-specific datasets, and PRA yields additional gains. Example prompts, generated texts, filtering code, and audio samples with multiple TTS systems are available on our project page. 1 1 1 https://natsuooo.github.io/llm-asr-augmentation/ 2 Related work 2.1 Text filtering for ASR domain adaptation Recent studies have explored the use of LLM-generated text for downstream ASR domain adaptation [ 30 , 31 ] but the quality of the text has not been sufficiently discussed. Some previous studies have focused on filtering out high-perplexity sentences, aiming to retain only fluent texts [ 20 , 26 ] . However, prioritizing low-perplexity sentences may risk excluding specialized or technical expressions necessary for domain adaptation. Other approaches employ vocabulary coverage maximization (VCM) to ensure lexical diversity by maximizing the number of unique words included in the training set [ 11 ] . Nevertheless, VCM may lead to the selection of numerous irrelevant or meaningless words generated by LLMs, which can hinder model learning rather than enhance it. In contrast, we adopt a normalized tri-objective selection that jointly balances TTR, perplexity, and domain-term coverage, resulting in texts that are both diverse and domain-specific. 2.2 Phonological Tasks and LLMs Traditional grapheme-to-phoneme (G2P) models rely heavily on pronunciation dictionaries and often fail to generalize to domain-specific words [ 7 ] . Recently, LLMs have been explored for phonological tasks such as G2P conversion [ 28 , 23 ] . While LLMs can capture broader contextual and linguistic cues than conventional models, prior work has shown that their performance remains inferior to specialized G2P systems or human annotators. Moreover, approaches that employ complex phoneme representations (e.g., IPA [ 4 ] ) often degrade text-to-speech (TTS) quality due to the difficulty of accurately rendering fine-grained phonetic symbols [ 32 ] . In contrast, our approach uses LLMs to generate alphabetic respellings that reflect actual pronunciation, thus avoiding the limitations of dictionary-based G2P and specialized phonetic symbols while enabling diverse and realistic variations. 3 Proposed method Our framework has two key components: an LLM-based text augmentation pipeline (Figure 1 (a)) and PRA (Figure 1 (b)). Synthesized speech generated from both components is used to fine-tune the ASR model. 3.1 LLM-based text augmentation pipeline Unlike DAS [ 30 ] , which generates only the required amount of text, our proposed pipeline first over-generates a large pool of candidate sentences through multi-stage LLM-based augmentation (steps 1–5), and then applies a novel filtering process (step 6) to select the most relevant and diverse subset. This over-generation and filtering is a key novelty, allowing more effective control over lexical diversity and domain coverage. 1. Domain seed : To ensure that the generated sentences are relevant to the target domain, our pipeline begins by conditioning each prompt on domain-specific information, referred to as a domain seed (e.g., air traffic control). 2. Context seed : Scenarios are generated as context seeds, and texts are created for each scenario to increase diversity. In many production settings, stakeholders provide an operational lexicon (e.g., user names, product names, call signs) obtained from chat logs, websites, or manuals, and systems must recognize these terms with high recall [ 16 , 32 , 10 , 22 ] . Therefore, we also explore using these terms as context seeds to generate texts containing them. 3. Multilingual prompting : For each context, prompts are constructed in multiple languages (e.g., English, Japanese, Chinese), and the outputs are translated back into the target language to enrich linguistic diversity. To ensure that domain-specific terms are preserved during translation, we add an instruction such as “Keep the term in English” to the prompts in the previous step. 4. Paraphrasing : LLM-based paraphrasing generates alternative expressions for each sentence. 5. Multiple LLMs : Multiple LLMs trained on different data are used in Steps 1–4, and their outputs are combined. 6. Filtering : While the above steps yield a large pool of candidate sentences, synthesizing speech and fine-tuning an ASR model on the entire set is inefficient. To address this, we introduce a novel filtering step based on three key heuristics: (1) maximizing TTR to promote lexical diversity, (2) maximizing perplexity to encourage technical and in-domain words, and (3) weighting domain-specific terms to enhance coverage of underrepresented vocabulary. We compute the combined score S(s) for each candidate sentence s as a weighted sum of TTR gain, perplexity, and the normalized count of domain-specific terms, as follows: \displaystyle S(s) \displaystyle=\!\alpha\,\frac{|\mathrm{Vocab}(s)\!\setminus\!V|}{|s|}+\beta\,\!\exp\left(\!-\frac{1}{|s|}\sum_{i=1}^{|s|}\log p(w_{i}\!\mid\!w_{<i})\right) \displaystyle\quad+\gamma\,\frac{|\{w_{i}\in s\mid w_{i}\in\mathcal{D}\}|}{|s|}. (1) where |\mathrm{Vocab}(s)| is the number of unique words

Read the complete text