LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context
Natsuo, Masaaki, Hiroaki, Yohei · 2025
Read LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context by Natsuo, Masaaki, Hiroaki, Yohei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17410.txt.
Introduction
Natsuo Yamashita and three co-authors improved in 2025 the cleanup step that fixes speech recognition errors with a language model, focusing on rare words. Such fixers often mangle uncommon or technical terms because training data barely contain them. The team generated synthetic training speech errors rich in rare words to teach the fixer. They also fed it the recognizer's top guesses together with sound-based context.
The phonetic hints guard against over-correction, where a right rare word gets replaced by a common one. Tests in English and Japanese showed better rare-word repair and lower overall error rates. The authors present the method as a practical upgrade for domain speech systems, such as medical or legal dictation. It keeps the language model's fluency while respecting what was actually said.
Catalog record
| CREATOR | Natsuo, Masaaki, Hiroaki, Yohei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17410
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context Natsuo, Masaaki, Hiroaki, Yohei Abstract Generative error correction (GER) with large language models (LLMs) has emerged as an effective post-processing approach to improve automatic speech recognition (ASR) performance. However, it often struggles with rare or domain-specific words due to limited training data. Furthermore, existing LLM-based GER approaches primarily rely on textual information, neglecting phonetic cues, which leads to over-correction. To address these issues, we propose a novel LLM-based GER approach that targets rare words and incorporates phonetic information. First, we generate synthetic data to contain rare words for fine-tuning the GER model. Second, we integrate ASR’s N-best hypotheses along with phonetic context to mitigate over-correction. Experimental results show that our method not only improves the correction of rare words but also reduces the WER and CER across both English and Japanese datasets. Yamashita Yamamoto Kokubo Kawaguchi Hitachi, Litd.Japan keywords speech recognition, error correction, large language model, keywords biasing † † email: {natsuo.yamashita.gh, maaaki.yamamoto.af, hiroaki.kokubo.dz, yohei.kawaguchi.xk}@hitachi.com Figure 1: Overview of the proposed methods of synthetic data generation from rare words and GER with phonetic context. 1 Introduction Automatic speech recognition (ASR) technology has achieved remarkable progress over the past decades, driven by advancements in deep learning techniques [ 1 ] . However, these systems often produce transcription errors, particularly due to background noise, speaker accents, different speaker styles, and domain-specific terms. To address this issue, researchers have explored various generative error correction (GER) methods, utilizing pre-trained language models (LMs) [ 2 , 3 , 4 ] and large language models (LLMs) [ 5 ] . These approaches aim to enhance ASR transcription quality by correcting grammatical, syntactical, or semantic errors. Recent studies [ 6 , 7 , 8 , 9 , 10 , 11 ] have demonstrated the effectiveness of LLMs in processing N-best hypotheses from ASR outputs to generate refined transcriptions. While these methods have shown promising results, they struggle with handling rare words that are absent from their training data [ 12 , 13 ] . This limitation arises because existing GER models often rely on pre-trained datasets that lack sufficient examples of specialized terminology or rare word errors, and obtaining domain-specific error data for fine-tuning remains a challenge. Another critical limitation of existing methods is their reliance solely on textual information while ignoring phonetic cues [ 14 ] . This often results in over-correction, where unnecessary changes are made, shifting the transcription toward formal written language and reducing the fidelity of the spoken input [ 6 ] . For instance, models may produce semantically plausible outputs that fail to preserve the original pronunciation or intent, leading to phonetically incorrect results and compromising the overall quality of ASR systems. To overcome these challenges, this study investigates an LLM-based GER approach that specifically targets rare words and incorporates phonetic information. First, we propose generating transcripts from rare words using an LLM and synthesizing speech from them to prepare error-pair data. By generating multiple transcripts and synthesizing speech with multiple speakers, we ensure a variety of error patterns in the generated data, which are then used to fine-tune the LLM. This approach enables the model to learn from various errors, allowing it to handle both common and rare words that are not part of its initial training data. In contrast to previous studies [ 6 , 10 , 11 , 15 ] that replace words in hypotheses or rescore hypotheses to limit the introduction of new words and avoid over-correction, our method focuses on the accurate correction of errors, even when they are not present in the hypotheses. Second, inspired by the previous work on phone-conditioned LM [ 2 ] , we incorporate phonetic information into the LLM-based GER process to improve both phonetic and semantic correctness. Unlike the previous work [ 14 ] , which integrated acoustic features into the ASR decoding process, our approach focuses on utilizing phonetic context derived from ASR hypotheses. This approach is significantly easier to integrate with the latest LLMs and ASR models that are continuously updated, especially given that the internal structures of some models, such as ChatGPT [ 16 ] , are not publicly disclosed. By leveraging an LLM to predict simplified phonetic readings of text, we provide additional phonetic context as input to our correction model. This enables the system to incorporate both textual and phonetic features, resulting in more accurate corrections that better align with the original spoken input. Experimental results demonstrate that our approach improves the correction of rare words, and reduces the word error rate (WER) and character error rate (CER) across both English and Japanese datasets. 2 Related Work 2.1 Error correction for rare words In many practical scenarios, a rare word list can be obtained in advance from sources such as user names, meeting chat logs, websites, manuals, or even words registered by users [ 17 ] . Several studies [ 18 , 19 ] have explored the use of rare word lists for error correction, but the error-pair data is limited within the training data. A recent study proposed an RAG-based keyword error correction using LLMs by searching for similar entries in rare word lists [ 20 ] , but correcting non-keyword errors remains a challenge. In this paper, we propose generating diverse error-pair data that includes rare words, enabling the error correction of both rare and non-rare word errors. 2.2 Phonetic notation In the field of linguistics, the International Phonetic Alphabet (IPA) [ 21 ] has been widely used to accurately represent the pronunciation of all languages around the world. However, it is often perceived as complex, particularly by non-native speakers, due to the specialized linguistic symbols [ 22 ] . In the field of text-to-speech (TTS), to represent phonetic information, ARPAbet for English and romanized Kana for Japanese are commonly used by using graphene-to-phoneme (G2P) tools [ 2 , 23 , 24 ] . A very recent study investigated LLM-based grapheme-to-phoneme conversion and reported that the performance of an LLM alone is significantly lower than that of an approach combining an LLM and a G2P dictionary [ 25 ] . This finding suggests that current LLMs struggle to effectively handle specialized phonetic representations. To address this issue, we propose an LLM-based approach using simplified phonetic representation to enable LLMs to better comprehend phonetic information. 3 Methodology Figure 1 illustrates our proposed methods. First, we propose training a GER model using synthetic data generated by gerative models based on a given rare word list. Additionally, to prevent over-correction caused by ignoring phonetic information, we propose explicitly providing the GER model not only N-best hypotheses, but also 1-best phonetic representations. 3.1 Synthetic data generation from rare words Given a set of W biasning words \left\{\mbox{\boldmath$w_{n}$}\right\}_{n=1}^{W} , we generate T synthesized transcripts \left\{\mbox{\boldmath$t$}^{(w)}_{n}\right\}_{n=1}^{T} that include each rare word. To create these transcripts, we prompt an LLM with instructions, optionally providing contextual information when available. For instance, the instruction might read: “Provide 5 different English sentences in various contexts that include the term w_{n} , which is a medical term.” . Subsequently, using text-to-speech models, we generate S synthetic spoken utterances \left\{\mbox{\boldmath$u$}^{(w,t)}_{n}\right\}_{n=1}^{S} for each synthetic transcript. Here, multiple transcripts are generated for each rare word