GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems

Amin Robatian, Mohammad Hajipour, Mohammad Reza Peyghan, Fatemeh Rajabi, Sajjad Amini *, Shahrokh Ghaemmaghami, Iman Gholampour · 2025

Read GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems by Amin Robatian, Mohammad Hajipour, Mohammad Reza Peyghan, Fatemeh Rajabi, Sajjad Amini *, Shahrokh Ghaemmaghami, Iman Gholampour (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10734.txt.

Introduction

Amin Robatian and co-authors present GEC-RAG, a method that fixes speech recognition errors by having a large language model learn from past mistakes, in a 2025 paper. Speech systems falter in fields and languages with little training data, such as Persian, and their word error rates climb. The new method treats the speech system as a closed box, as cloud services require, and never retrains it.

The team first built a store of past outputs paired with the true text, covering both the top guess and the five best guesses. For each new output, the system pulls out stored cases with similar wording and shows them to the language model as worked examples of the fixes needed. Error rates in Persian dropped sharply with this help. The authors note the same store-and-retrieve trick can adapt speech systems to new fields without touching the model itself.

Catalog record

CREATORAmin Robatian, Mohammad Hajipour, Mohammad Reza Peyghan, Fatemeh Rajabi, Sajjad Amini *, Shahrokh Ghaemmaghami, Iman Gholampour
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems Amin Robatian, Mohammad Hajipour, Mohammad Reza Peyghan, Fatemeh Rajabi, Sajjad Amini *, Shahrokh Ghaemmaghami, Iman Gholampour Abstract Automatic Speech Recognition (ASR) systems have demonstrated remarkable performance across various applications. However, limited data and the unique language features of specific domains, such as low-resource languages, significantly degrade their performance and lead to higher Word Error Rates (WER). In this study, we propose Generative Error Correction via Retrieval-Augmented Generation (GEC-RAG), a novel approach designed to improve ASR accuracy for low-resource domains, like Persian. Our approach treats the ASR system as a black-box, a common practice in cloud-based services, and proposes a Retrieval-Augmented Generation (RAG) approach within the In-Context Learning (ICL) scheme to enhance the quality of ASR predictions. By constructing a knowledge base that pairs ASR predictions (1-best and 5-best hypotheses) with their corresponding ground truths, GEC-RAG retrieves lexically similar examples to the ASR transcription using the Term Frequency-Inverse Document Frequency (TF-IDF) measure. This process provides relevant error patterns of the system alongside the ASR transcription to the Generative Large Language Model (LLM), enabling targeted corrections. Our results demonstrate that this strategy significantly reduces WER in Persian and highlights a potential for domain adaptation and low-resource scenarios. This research underscores the effectiveness of using RAG in enhancing ASR systems without requiring direct model modification or fine-tuning, making it adaptable to any domain by simply updating the transcription knowledge base with domain-specific data. I Introduction Automatic Speech Recognition (ASR) systems are designed to transform speech signals into sequences of words, supporting applications such as text-based communication and device control [ 1 ] . In recent advancements, ASR technology has seen significant progress, transitioning from traditional Hidden Markov Model (HMM)-based frameworks [ 2 ] to cutting-edge end-to-end (E2E) models like wav2vec 2.0 [ 3 ] and Whisper [ 4 ] . However, due to the lack of data in low-resource languages or specific domains, these systems face challenges and errors. The ongoing presence of ASR errors has increased the need to develop techniques for automatically detecting and correcting errors in ASR systems [ 1 ] . Given the limitations of paired data for training ASR systems, the use of Language Models (LMs) trained on datasets several orders of magnitude larger has been investigated to improve ASR performance [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . These approaches are typically classified into two categories: language model fusion methods and Error Correction (EC) techniques as a post-processing step. In this paper, we focus on methods that treat the ASR system as a black-box, an approach that is particularly significant given the increasing use of cloud-based ASR services through APIs. Therefore, we will concentrate on EC methods, which offer a practical solution for improving ASR accuracy without requiring modifications to the core ASR system. The study by Shin et al. [ 6 ] employs the BERT model to rescore N-best list hypotheses generated by ASR systems, significantly improving recognition accuracy through enhanced contextual understanding. Similarly, Guo et al. [ 7 ] propose a spelling correction model for end-to-end ASR systems, effectively addressing transcription and spelling errors as a post-processing step. Hrinchuk et al. [ 9 ] present a transformer-based sequence-to-sequence model for ASR error correction, demonstrating its ability to learn error patterns and generate refined transcriptions. Furthermore, Dutta et al. [ 13 ] showcase the effectiveness of BART in ASR error correction by utilizing diverse data augmentation techniques during fine-tuning, leading to significant WER reductions, with further improvements achieved through alignment and rescoring strategies. In addition, Ma et al. [ 15 ] introduce an N-best T5 model fine-tuned from a pre-trained T5 language model, which leverages ASR N-best lists to improve transcription accuracy. By transferring knowledge from the language model and extracting richer information from the ASR decoding space, this method outperforms a strong Conformer-Transducer baseline. Recent advancements in generative Large Language Models (LLMs) have introduced a transformative paradigm for Generative Error Correction (GER) in automatic speech recognition, aiming to predict accurate transcriptions from the decoded N-best hypotheses through the application of generative LLMs [ 16 , 17 , 18 , 19 , 20 , 21 , 22 ] . In the studies by Ma et al. [ 16 ] and [ 17 ] , large language models, particularly ChatGPT, are leveraged for ASR error correction, showing that generative models can significantly enhance transcription accuracy by utilizing their deep language comprehension, without requiring task-specific fine-tuning. Li et al. [ 18 ] fine-tune a multilingual LLM to correct 1-best ASR hypotheses in over 100 languages, demonstrating that the approach not only boosts ASR performance but also improves error correction in low-resource languages via knowledge transfer between languages with shared writing systems. Hu et al. [ 19 ] introduce ClozeGER, an innovative ASR error correction paradigm using SpeechGPT, a multimodal LLM that incorporates source speech to refine correction accuracy, and reformulates GER as a cloze test to minimize redundancy and simplify the process. Li et al. [ 20 ] present LA-RAG, a novel Retrieval-Augmented Generation (RAG) approach, which enhances ASR accuracy by integrating token-level speech data stores and a speech-to-speech retrieval mechanism. Pusateri et al. [ 21 ] propose a RAG-like approach for correcting entity name errors in ASR systems, particularly for low-frequency entities. Their method involves indexing relevant entities in a vector database and utilizing an LLM to integrate the retrieved entities into the correction process, thereby enhancing ASR outputs. [ 22 ] introduces DARAG, which fine-tunes LLaMA-2 on augmented datasets containing synthetic errors and incorporates named entity retrieval to improve ASR error correction. In this study, we introduce Generative Error Correction via Retrieval-Augmented Generation (GEC-RAG), a novel approach designed to enhance the performance of ASR systems, particularly for low-resource languages. The key innovation of GEC-RAG lies in leveraging a knowledge base that stores ground truth data alongside ASR predictions, including 1-best and 5-best hypotheses. This knowledge base functions as a resource for retrieving examples based on lexical similarity, utilizing the TF-IDF [ 23 ] approach to identify relevant matches, which are then used for In-Context Learning (ICL) with the GPT-4o model. By explicitly exposing the model to the typical error patterns of the ASR system, GEC-RAG enables precise and effective error correction without requiring any modifications to the ASR system itself. This approach is particularly well-suited to the challenges of low-resource language scenarios, offering a scalable, adaptable, and efficient solution for reducing Word Error Rate (WER) while maintaining compatibility with black-box ASR services. Our contributions are as follows: • We propose a novel retrieval-based approach for ASR error correction that operates externally to the ASR system. • We demonstrate the effectiveness of in-context learning with GPT-4o for correcting ASR errors in Persian, a low-resource language. • We show how our method achieves significant improvements in WER and is particularly advantageous for domain adaptation. The remainder of this paper is organized as follows: Section II provides a comprehensive overview of the proposed methodology, detailing the design and construction of the knowledge base, the retrieval process in the RAG framework, and the integration of in-context learning using GPT-4o to address

Read the complete text