Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems

Bo Ren, Yu Shi, Jinyu Li · 2025

Read Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems by Bo Ren, Yu Shi, Jinyu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06252.txt.

Introduction

Bo Ren, Yu Shi and Jinyu Li present a lightweight prompt biasing method for contextual speech recognition in 2025. End-to-end recognizers still stumble on rare and domain-specific names and terms that matter most to users.

Their method trains a prompt biasing model to decide when to focus on listed entities, within a unified multitask setup. A filtering step drops irrelevant entities so the prompt stays lean. On in-house domain data, entity error rates fell sharply versus a shallow fusion baseline, for both small and large entity lists. The gain comes with no change to the model structure, they note.

Catalog record

CREATORBo Ren, Yu Shi, Jinyu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems Bo Ren, Yu Shi, Jinyu Li Abstract End-to-End Automatic Speech Recognition (ASR) has advanced significantly yet still struggles with rare and domain-specific entities. This paper introduces a simple yet efficient prompt-based biasing technique for contextualized ASR, enhancing recognition accuracy by leverage a unified multi-task learning framework. The approach comprises two key components: a prompt biasing model which is trained to determine when to focus on entities in prompt, and a entity filtering mechanism which efficiently filters out irrelevant entities. Our method significantly enhances ASR accuracy on entities, achieving a relative 30.7% and 18.0% reduction in Entity Word Error Rate compared to the baseline model with shallow fusion on in-house domain dataset with small and large entity lists, respectively. The primary advantage of this method lies in its efficiency and simplicity without any structure change, making it lightweight and highly efficient. I Introduction In recent years, End-to-End (E2E) automatic speech recognition (ASR) systems have made significant progress thanks to advances in deep learning models [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . However, these systems still face challenges in accurately recognizing rare words and domain-specific terms. To enhance ASR performance, researchers have proposed various methods [ 11 , 12 , 13 , 14 , 15 ] to improve the performance on specific contexts. Contextual biasing techniques leverage external information to boost recognition accuracy, particularly when handling rare or specialized entities. Typically, the external information is provided to the ASR system during decoding in the form of an words/entities list, which is often referred as biasing list. Extensive research has been conducted on contextual biasing techniques for E2E ASR systems, which can be broadly classified into two major categories. Shallow fusion methods [ 11 , 12 ] integrate external language models (LMs) with ASR systems during decoding by weighting LM scores, allowing the system to prioritize contextually relevant terms. Further advancements in shallow fusion include sub-word regularization, pre-training, grapheme-to-grapheme pronunciation learning, and deep integration with neural network language models [ 16 , 17 ] . However, a fundamental limitation of shallow fusion is its reliance on post-contextual boosting, which requires the model to generate the correct expected prefix terms in the candidate pool without access to external contextual information. To address these limitations, deep biasing methods directly incorporate contextual information into E2E ASR models, enabling joint optimization. A notable example is the Contextual Listen, Attend and Spell (CLAS) system, which integrates ASR components with contextual embeddings [ 13 ] . The CLAS system has been further enhanced to include phonetic information, leveraging pronunciation knowledge for improved recognition of rare words [ 18 , 19 ] . Other approaches have explored intermediate biasing loss and attention mechanisms to improve contextual modeling in CLAS systems [ 20 ] . However, these methods often rely on additional encoders to embed contextual information, introducing computational overhead and complexity. Moreover, attention mechanisms may struggle to scale effectively with large biasing lists, a common scenario in real-world applications [ 13 ] . Transformer-based architectures have recently become the backbone of state-of-the-art E2E ASR systems [ 21 ] . These models consistently outperform traditional Recurrent Neural Network (RNN) approaches and are now widely used in both research and industry [ 1 , 22 , 23 , 24 , 25 , 2 , 26 ] . However, integrating contextual biasing effectively into Transformer models remains a challenging problem, with direct impact on user experience and commercial deployment. There is also increasing interest in utilizing large language models (LLMs) to enhance contextual biasing in ASR [ 27 , 28 ] . Although these methods have demonstrated promising results, they often require significant computational resources and add architectural complexity. As LLM-based solutions are still emerging and not yet widely adopted in production ASR systems, our work focuses on practical Transformer-based E2E ASR models, leaving LLM integration for future research. In this work, we propose a straightforward and effective method for contextual biasing in Transformer-based ASR systems using a multi-task learning framework. Our approach, called Prompt Biasing , leverages the Transformer’s cross-attention mechanism to introduce contextual information as a prompt to the decoder. By employing a unified multi-task learning setup with dedicated task tokens, the model can efficiently handle both biasing and non-biasing scenarios without requiring any architectural changes. Additionally, we introduce an efficient entity filtering strategy that rely on the same model during decoding, enabling robust performance even with large biasing lists. Experimental results demonstrate that our lightweight Prompt Biasing model consistently outperforms the baseline with shallow fusion on domain-specific datasets, achieving a relative reduction in Entity Word Error Rate (EWER) of 30.7% and 18.0% for small and large biasing lists, respectively. (a) Standard Transformer ASR training (b) Unified multi-task training for contextual biasing (c) Multi-task token format for Prompt Biasing training Fig. 1: Overview of the proposed Prompt Biasing method. (a) Standard Transformer ASR training, where the model predicts the next token from previous tokens. (b) Unified multi-task training for contextual biasing, where the model predicts the next token using both previous tokens and contextual information provided as a prompt. (c) Multi-task token format for Prompt Biasing training: biasing and non-biasing tasks are indicated by special task tokens ( <hit> / <miss> ), and the biasing list is included as a prompt starting with the <sop> token. II Related Work This section provides an overview of recent advancements in Transformer-based ASR systems and contextual biasing techniques, with a particular focus on state-of-the-art developments in Transformer architectures. II-A Transformers based E2E ASR Recent advancements in Transformer-based architectures have significantly advanced the field of end-to-end ASR. The Speech-Transformer [ 23 ] first demonstrated the effectiveness of self-attention and cross-attention mechanisms for modeling long-range dependencies in speech, resulting in improved recognition accuracy and efficiency. Building on this, the Transformer-Transducer framework incorporates a Transformer encoder within a transducer model, enabling efficient streaming recognition with competitive performance [ 26 , 29 ] . The Conformer architecture further enhances Transformer models by integrating convolutional layers, which capture local acoustic features and improve robustness in challenging acoustic conditions [ 24 ] . Collectively, these innovations have established Transformer-based models as the state-of-the-art for ASR, offering superior accuracy, scalability, and computational efficiency compared to traditional RNN-based systems. More recently, Whisper [ 25 ] has emerged as a versatile end-to-end Transformer-based speech processing system, trained on large-scale multilingual data. Whisper supports not only speech recognition but also translation and language identification within a unified framework, further demonstrating the flexibility and effectiveness of Transformer architectures in speech applications. II-B Contextual biasing While shallow fusion techniques can be easily applied to Transformer-based ASR systems, their performance improvements remain limited due their post-training adjustment, and also be concerned to either under- or over- biasing issue [ 11 ] . Deep biasing techniques, such as the CLAS system [ 13 ] , have historically relied on RNN architectures. Some studies have attempted to enhance contextual biasing capabilities in Transformer-based models. For instance, certain works introduce a Tree-Constrained Pointer Generator (TCPGen) module [ 30 ] , which dynamically adjusts transcription by interpolating between the original model and TCPGen distributions [ 31 ] .

Read the complete text