Customizing Speech Recognition Model with Large Language Model Feedback
Shaoshi Ling, Guoli Ye · 2025
Read Customizing Speech Recognition Model with Large Language Model Feedback by Shaoshi Ling, Guoli Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11091.txt.
Introduction
Shaoshi Ling and Guoli Ye use feedback from a large language model to tune a speech recognition system in a 2025 paper. Recognizers handle plain speech well but stumble on rare names and on new domains. Language models, trained on vast text, often know those names and contexts better.
Their method needs no labeled speech for the new domain. The language model scores the recognizer's draft transcripts given context, and those scores act as rewards in a learning loop that fine-tunes the recognizer. The approach cut errors on named entities by about a fifth compared with standard self-training. The paper shows a path to adapting speech tools where labeling budgets are thin.
Catalog record
| CREATOR | Shaoshi Ling, Guoli Ye |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11091
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Customizing Speech Recognition Model with Large Language Model Feedback Shaoshi Ling, Guoli Ye Abstract Automatic speech recognition (ASR) systems have achieved strong performance on general transcription tasks. However, they continue to struggle with recognizing rare named entities and adapting to domain mismatches. In contrast, large language models (LLMs), trained on massive internet-scale datasets, are often more effective across a wide range of domains. In this work, we propose a reinforcement learning based approach for unsupervised domain adaptation, leveraging unlabeled data to enhance transcription quality—particularly the named entities affected by domain mismatch—through feedback from a LLM. Given contextual information, our framework employs a LLM as the reward model to score the hypotheses from the ASR model. These scores serve as reward signals to fine-tune the ASR model via reinforcement learning. Our method achieves a 21% improvement on entity word error rate over conventional self-training methods. I Introduction Automatic speech recognition (ASR) systems have made significant progress in recent years, driven by advances in neural network architectures and the availability of increasingly large amounts of labeled data [ 1 ] . However, the performance of ASR systems often degrade significantly when the target domain or test conditions differ from those seen during training. And this domain mismatch between training and testing conditions occurs commonly in the real world application and poses a major challenge. One straightforward approach is to collect labeled data from the target domain and use it for adapting the pre-trained source model. However, manually annotating large volumes of data for each new domain is both costly and time-consuming. This highlights the need for the methods that can leverage unlabeled data to enable effective source-to-target domain transfer [ 2 ] . The existing efforts to address this problem primarily focus on leveraging unlabeled target domain data to enhance the ASR performance. This approach is generally known as unsupervised domain adaptation and has been widely explored in speech processing communities. Among various techniques, the most widely used one is known as self-training [ 3 ] , which typically involves two stages. First, a pre-trained model generates pseudo-labels for the target-domain data. Then, these pseudo-labeled samples, often accompanied by confidence scores, are used to adapt the model. The confidence scores are typically estimated using the pseudo posterior probabilities produced by the softmax function in ASR models [ 4 ] . However, these scores can be unreliable, especially in previously unseen domains. In contrast, large language models (LLMs) such as GPT4 [ 5 ] , Deepseek [ 6 ] , Phi [ 7 ] trained on internet-scale corpora, demonstrate remarkable generalization capabilities across a wide range of domains. These models have demonstrated strong performance on tasks involving commonsense reasoning, in-context learning, and even zero-shot adaptation. Such capabilities suggest that LLMs could serve as valuable resources for enhancing the performance of ASR models, particularly in low-resource or domain-mismatched settings. Recent studies have explored integrating LLMs with ASR systems through approaches such as re-ranking hypotheses [ 8 , 9 , 10 , 11 ] or error correction during post-processing [ 12 , 13 , 14 ] . Inspired by reinforcement learning (RL) from AI feedback [ 15 , 16 ] , we propose an unsupervised domain adaptation of ASR systems using online LLMs feedback. Rather than relying on a separately trained reward model tailored for the ASR task, our method bypasses this requirement by directly extracting evaluation scores from an LLM. Specifically, given hypothesis of spoken utterance in the target domain along with contextual information (e.g., domain metadata, description of the targeted named entities or preceding dialogue), we use an LLM to evaluate the quality of those hypotheses by computing the sum of log-probability scores in the LLMs. These scores are then used to generate reward signals, which guide the fine-tuning of the ASR model using state-of-the-art RL algorithms like DPO [ 17 , 16 ] and GRPO [ 18 ] . Unlike traditional self-training techniques, our method does not require confidence score. Instead, it leverages the LLMs’ implicit understanding of language and context to produce reliable rewards, thereby promoting more accurate and contextually appropriate transcriptions in target domains. Our experiments demonstrate that our RL framework can substantially improve transcription quality in domain-mismatched scenarios, particularly for named entities and context-sensitive phrases. Compared to standard self-training domain adaptation methods, our approach achieves up to 21% relative improvement on entity word error rate (EWER), showcasing the effectiveness of LLMs feedback in guiding ASR model adaptation. Fig. 1: The Overview II Related Work II-A Unsupervised Domain Adaptation in ASR Since obtaining ground-truth speech transcriptions in the target domain is often prohibitively expensive, many existing approaches leverage out-of-domain data to bootstrap improved models for the target domain [ 19 , 20 ] . Other methods includes simulating the target domain speech [ 21 ] , domain-invariant representation learning [ 22 ] , domain adversarial learning [ 20 ] and teacher-student learning for efficient adaptation [ 23 , 24 ] . More recently, self-supervised pre-trained models have also been used for pseudo-labelling to achieve unsupervised adaptation [ 25 , 26 ] . To address the inherent uncertainty in ASR decoding, a complementary line of research focuses on confidence estimation and data selection, which have been studied for decades. Recent efforts focus on performing pseudo-labeling denoising technique, such as uncertainty driven self-training [ 25 ] , mask prediction [ 27 ] , and leveraging internal model signals—e.g., analyzing the self-attention matrix to evaluate pseudo-label quality [ 28 ] , or partitioning attention weights [ 29 ] to better fit the model on pseudo-label data. II-B Reinforcement Learning in ASR In the field of ASR, several reinforcement learning approaches have been proposed to bridging the gap between the training and testing phases. Self-critical sequence training [ 30 ] utilizes a word error rate based reward to reduce the reliance on teacher forcing during training. Others [ 31 ] employed policy gradient method to train the S2S ASR model, treating sequence generation as a sequential decision-making process under the RL framework. And the work [ 32 ] introduced a framework that incorporates user feedback through hypothesis selection, allowing the model to refine its predictions based on human preference. While prior works have primarily relied on either human feedback or WER for designing reward signals, our approach explores the use of feedback from LLMs. To the best of our knowledge, this is the first work to apply RL in the contextual customization task for ASR. This opens up new possibilities for adapting ASR systems to diverse domains by utilizing powerful LLMs not merely in post-processing stages such as re-ranking [ 8 , 9 , 10 , 11 ] or error correction [ 12 , 13 , 14 ] , but as an integral part of the training process. III Method III-A Adaptation Framework Given an utterance audio X , an ASR model can be viewed as a policy \pi that maps the audio input into the corresponding transcription Y , represented as \pi(Y|X) . We define a reward function r(Y|X) that assigns a feedback score to the transcription hypothesis. For each audio X, we can generate n candidate hypotheses: a_{1},a_{2},\ldots,a_{n} . As shown in Figure 1 , our framework followed the general RL pipeline