Peeking Into The Future For Contextual Biasing
Ramaneswaran Selvakumar \sthanks Work done during internship at Samsung Research America, Cindy Tseng, Eesung Kim, Vijendra Raj Apsingekar, Yun Tang · 2025
Read Peeking Into The Future For Contextual Biasing by Ramaneswaran Selvakumar \sthanks Work done during internship at Samsung Research America, Cindy Tseng, Eesung Kim, Vijendra Raj Apsingekar, Yun Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17657.txt.
Introduction
Ramaneswaran Selvakumar and four co-authors teach a speech recognizer to guess names it has never heard. Voice assistants stumble exactly there: a contact called Xochitl or a street name outside the training data comes out garbled. Given a candidate list, such as the user's contacts, the model should lean toward list words when the sound is close.
Their trick makes the model predict several tokens ahead at once, not just the next sound. Peeking ahead lets it score whole candidate names against the coming audio before committing. No extra name encoder or attention layer is bolted on, so the design stays lean. On the LibriSpeech benchmark, errors on named entities fell by up to 50.34 percent relative to the plain baseline.
Catalog record
| CREATOR | Ramaneswaran Selvakumar \sthanks Work done during internship at Samsung Research America, Cindy Tseng, Eesung Kim, Vijendra Raj Apsingekar, Yun Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Peeking Into The Future For Contextual Biasing Ramaneswaran Selvakumar \sthanks Work done during internship at Samsung Research America, Cindy Tseng, Eesung Kim, Vijendra Raj Apsingekar, Yun Tang Abstract While end-to-end (E2E) automatic speech recognition (ASR) models excel at general transcription, they struggle to recognize rare or unseen named entities (e.g., contact names, locations), which are critical for downstream applications like virtual assistants. In this paper, we propose a contextual biasing method for attention based encoder decoder (AED) models using a list of candidate named entities. Instead of predicting only the next token, we simultaneously predict multiple future tokens, enabling the model to ”peek into the future” and score potential candidate entities in the entity list. Moreover, our approach leverages the multi-token prediction logits directly without requiring additional entity encoders or cross-attention layers, significantly reducing architectural complexity. Experiments on Librispeech demonstrate that our approach achieves up to 50.34% relative improvement in named entity word error rate compared to the baseline AED model. † † address: Samsung Research America, USA 1 Introduction In recent years, end-to-end (E2E) automatic speech recognition (ASR) systems have emerged as the dominant paradigm for speech recognition [ 17 , 14 ] . Unlike conventional systems that require individual components such as acoustic model (AM) and language model (LM), E2E approaches such as connectionist temporal classification (CTC) [ 5 , 6 ] and attention encoder-decoder (AED) [ 19 ] directly map acoustic features to text tokens. However, the effectiveness of these E2E approaches is inherently tied to the context present in its training data. Therefore, these models often fail to generalize to specific user contexts, as these contain information, particularly named entities that are rare or unseen during training. Named entities such as personal names or locations are so vast and diverse, that any static training dataset may not provide adequate coverage. Moreover, these named entities convey crucial semantic information to downstream natural language understanding (NLU) tasks in applications such as voice assistants. It is often impractical to retrain E2E-ASR models for every new context. Therefore, a common strategy is to utilize a ’bias list’ - a list of keywords as external contextual knowledge to bias speech recognition. A simple approach to incorporate a bias list is shallow-fusion, which works by rescoring the ASR hypothesis during inference using an external biasing component such as another language model (LM) [ 9 , 22 ] or a weighted finite state transducer (WFST) [ 23 , 3 ] . While this approach is architecturally simple, it requires a separate and often extensive tuning process to balance the scores from the main ASR model and the biasing component. Furthermore, despite this tuning effort, its contextualization capability is often limited, as the biasing is not deeply integrated into the model itself. A more promising alternative is end-to-end neural biasing [ 18 , 7 ] , which directly integrates the biasing component into the ASR model, enabling joint optimization. These methods use dedicated bias encoders to obtain bias phrase representations and implement cross-attention layers to associate bias phrases with the decoder or acoustic encoder. However, there are some limitations. It has been observed that simply adding a cross-attention layer for the bias list is not effective [ 8 ] . To overcome this, some approaches use additional branches to detect bias phrases [ 8 , 7 ] , deeper integration [ 1 , 10 ] or auxiliary loss functions [ 11 , 21 ] , but this leads to more complex architectures which are difficult to deploy. Moreover, a common issue among these approaches is entity fragmentation [ 20 ] . They treat entities not as whole units, but as a disjointed sequence of sub-word tokens from the static vocabulary. For instance, a single entity like “timaeus” maybe fragmented into a sequence “tim” “ae” “us” forcing the model to learn the unnatural dependencies within the entity itself. Main Contributions: In this paper, we propose an architecturally simple yet effective method of context biasing. First, we propose a multi-token prediction [ 4 ] (MTP) mechanism for the attention-encoder-decoder (AED) architecture. MTP enables the parallel prediction of multiple future tokens which allows the model to look-ahead and learn non-local sequence patterns. We then introduce a novel biasing mechanism that leverages this lookahead capability. Specifically, we utilize the un-normalized token logits and a learned projection to score entities in bias list. This approach does not require bias encoders or cross-attention layers, resulting in a simpler architecture. Finally, these entity scores are used as prediction logits, creating a unified search space at inference time, where at a time-step the model can either predict next token from its static vocabulary or next entity from the dynamic bias list. Moreover, by treating each entity as a single, indivisible candidate in the search space, our approach also mitigates the entity fragmentation problem. We demonstrate the effectiveness of our approach on Librispeech corpus where it reduces named entity WER by 50.34% compared to the AED baseline. Figure 1: Overview of our proposed approach 2 Attention-Based Encoder Decoder This section describes the attention-based encoder-decoder system that consists of an audio encoder and an attention-based decoder, which are extended to the proposed contextual biasing method. Audio Encoder The audio encoder comprises two convolutional layers, a linear projection layer followed by several Conformer blocks [25]. The audio encoder transforms an audio feature sequence X to T -length hidden state vectors H_{e}=\text{AudioEnc}(X)=[h^{e}_{1},...,h^{e}_{T}]\in\mathbb{R}^{T\times d} , where d represents the dimension and AudioEnc is the audio encoder. Attention-based Decoder: The decoder is an autoregressive model that estimates the probability of the token sequence y given the audio representation H_{e} . This is factorized as a product of conditional probabilities: P(y|X)=\prod_{s=1}^{S}P(y_{s}|y_{0:s-1},X). (1) At each decoding step s , the decoder network processes the previously generated tokens y_{0:s-1} and the encoder output H_{e} to produce a decoder output h^{d}_{s}=\text{Dec}(y_{0:s-1},H_{e}) . The probability distribution for the next token y_{s} is shown as below: \displaystyle\mathbf{l}_{s} \displaystyle=W_{o}h^{d}_{s}, (2) \displaystyle P(y_{s}|y_{0:s-1},X) \displaystyle=\text{Softmax}(\mathbf{l}_{s}), (3) where \mathbf{l}_{s}\in\mathbb{R}^{V} is output logits, V is the static vocabulary size, and W_{o} is the final language model projection layer. 3 Proposed Approach 3.1 Multi-Token Prediction The standard attention-based decoder from section 2 predicts tokens sequentially using P(y_{s}|y_{0:s-1},X) , limiting its ability to consider future context when making current predictions. We propose to extend this model to predict K future tokens simultaneously, enabling the model to ”peek into the future” and evaluate long-term dependencies. This lookahead capability is particularly valuable for contextual biasing, as it allows better scoring of contextual entities by considering how well they fit within the broader sequence context. To make the parallel prediction of a K -token sequence tractable, we employ a conditional independence assumption. The joint probability of the next K tokens is approximated as the product of K independent probabilities, all conditioned on the same preceding context: \displaystyle P(y_{s+1:s+K}|y_{\leq s},X) \displaystyle\approx\prod_{k=1}^{K}P_{k}(y_{s+k}|y_{\leq s},X), (4) \displaystyle\mathbf{l}^{k}_{s} \displaystyle=g_{k}(h^{d}_{s}), (5) \displaystyle P(y_{s+k}|y_{0:s-1},X) \displaystyle=\text{Softmax}(\mathbf{l}^{k}_{s}), (6) where g_{k} is the MTP module for the k -th prediction head, and k\in[1,K] . g_{k} consists of a Feed-Forward Network (FFN) block that generates a specialized representation for the k^{th} future token. To ensure parameter efficiency, all heads share the final language model projection layer to produce the vocabulary logits. Each P_{k}(y_{s+k}|y_{\leq s},X) can