DYNAC: Dynamic Vocabulary-based Non-Autoregressive Contextualization for Speech Recognition

Yui, Yosuke, Muhammad, Yifan, Chyi-Jiunn, Shinji · 2025

Read DYNAC: Dynamic Vocabulary-based Non-Autoregressive Contextualization for Speech Recognition by Yui, Yosuke, Muhammad, Yifan, Chyi-Jiunn, Shinji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00422.txt.

Introduction

Yui Sudo and co-authors present a 2025 method to help speech recognition catch rare words and names. Such context phrases often trip standard systems. A newer trick adds the phrases as fresh tokens a model can use, but the fast model type loses track of how the new tokens link to the rest. Their method, called DYNAC, fixes that link inside a fast frame.

DYNAC feeds the added tokens into mid layers of the encoder, so later steps know the context. The fix keeps the speed edge of the fast model type. On a standard test set, run time fell by 81 percent. Word error rate worsened by just a tenth of a point. Fast context help matters for live captions and voice tools that must know names, places, and jargon on the fly.

Catalog record

CREATORYui, Yosuke, Muhammad, Yifan, Chyi-Jiunn, Shinji
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00422

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DYNAC: Dynamic Vocabulary-based Non-Autoregressive Contextualization for Speech Recognition Yui, Yosuke, Muhammad, Yifan, Chyi-Jiunn, Shinji Abstract Contextual biasing (CB) improves automatic speech recognition for rare and unseen phrases. Recent studies have introduced dynamic vocabulary, which represents context phrases as expandable tokens in autoregressive (AR) models. This method improves CB accuracy but with slow inference speed. While dynamic vocabulary can be applied to non-autoregressive (NAR) models, such as connectionist temporal classification (CTC), the conditional independence assumption fails to capture dependencies between static and dynamic tokens. This paper proposes DYNAC (Dynamic Vocabulary-based NAR Contextualization), a self-conditioned CTC method that integrates dynamic vocabulary into intermediate layers. Conditioning the encoder on dynamic vocabulary, DYNAC effectively captures dependencies between static and dynamic tokens while reducing the real-time factor (RTF). Experimental results show that DYNAC reduces RTF by 81% with a 0.1-point degradation in word error rate on the LibriSpeech 960 test-clean set. Sudo Fukumoto Shakeel Peng Lin Watanabe Honda Research Institute JapanJapan Carnegie Mellon UniversityUSA keywords speech recognition, biasing, dynamic vocabulary, non-autoregressive, self-conditioned CTC † † email: [email protected], [email protected], [email protected] 1 Introduction End-to-end automatic speech recognition (E2E-ASR) [ 1 , 2 ] can be classified into autoregressive (AR) and non-autoregressive (NAR) models. AR models, such as the attention-based encoder-decoder [ 3 , 4 , 5 , 6 ] and recurrent neural network transducer (RNN-T) [ 7 , 8 ] , predict the next token based on the previous token sequence in an AR manner. In contrast, NAR models, such as connectionist temporal classification (CTC) [ 9 , 10 , 11 , 12 ] , predict all tokens simultaneously under the conditional independence assumption. This approach allows faster inference but often results in lower performance compared to AR models. Several methods have been proposed to relax the conditional independence assumption [ 13 , 14 , 15 , 16 , 17 ] . For example, self-conditioned CTC incorporates intermediate predictions as feedback to refine subsequent encoder layers, allowing the model to capture contextual dependencies [ 18 ] . Despite these advances, the performance of E2E-ASR models remains highly dependent on the training data, leading to performance inconsistencies for rare and unseen phrases. Frequent retraining to adapt to these phrases is impractical, highlighting the need for a method to contextualize models without additional retraining. Contextual biasing (CB) [ 19 , 20 , 21 , 22 , 23 ] provides an effective strategy for adapting E2E-ASR models to specific context phrases through a bias list without requiring additional retraining. Most CB methods are designed for AR models, leveraging a cross-attention layer in the decoder for accurate bias phrase recognition. Although AR-CB methods achieve accurate bias phrase recognition, their slow inference speed makes them unsuitable for latency-sensitive applications. Several CB methods have been proposed for NAR models or adapted to CTC-based NAR models to improve inference speed [ 24 , 25 , 26 ] . However, most existing CB methods represent bias phrases as sequences of subwords from a pre-defined vocabulary (referred to as static vocabulary), resulting in unnatural token patterns with low occurrence probabilities. For example, the personal name “ Raphael ” might be segmented into “ Ra ”, “ pha ”, and “ el ”. If such static token patterns are rare in the training data, their recognition accuracy decreases significantly. Several studies mitigate this issue by employing additional information, such as text-only data [ 27 , 28 ] , synthesized speech [ 29 , 30 ] , phonemes [ 31 , 32 ] , and named entity tags [ 33 ] . While these approaches improve CB performance, they introduce additional computational and operational overhead. To mitigate this issue without requiring such additional information, dynamic vocabulary expansion has been proposed [ 34 ] . This method introduces a dynamic vocabulary, where each bias phrase is represented as a dynamically expandable single token instead of being decomposed into a sequence of static tokens. For example, the bias phrase “ Raphael ” is represented as a single dynamic token [ < Raphael > ] rather than the static token sequence [“ Ra ”, “ pha ”, “ el ”]. While this method improves CB performance without relying on additional information, its evaluation has been limited to AR models (CTC/attention and CTC/RNN-T) [ 35 , 36 ] , which are computationally expensive. Although dynamic vocabulary expansion can be applied to CTC-based NAR models, the conditional independence assumption limits the ability to capture contextual dependencies between static and dynamic tokens. This paper proposes DYNAC (Dynamic Vocabulary-based NAR Contextualization), an NAR-CB method using self-conditioned CTC that incorporates dynamic vocabulary into the intermediate encoder layers. By conditioning the encoder on the dynamic vocabulary, DYNAC relaxes the conditional independence assumption. This approach allows the model to capture dependencies between static and dynamic tokens while reducing the real-time factor (RTF) compared to AR-CB methods [ 34 ] . The main contributions of this paper are as follows: • We propose DYNAC, an NAR-CB method using self-conditioned CTC that integrates dynamic vocabulary, enabling efficient CB while reducing the RTF. • We analyze the effectiveness of DYNAC by comparing the token-wise scores of the static and dynamic vocabulary to highlight its impact on recognition accuracy. • We demonstrate the effectiveness of DYNAC on the LibriSpeech 960 corpus and our in-house Japanese dataset, showing robust performance even on unseen phrases. 2 CTC-based CB with dynamic vocabulary This section describes the CTC-based CB method using dynamic vocabulary, which consists of an audio encoder, a bias encoder, and a scoring layer based on CTC. Although dynamic vocabulary expansion was originally proposed for AR-CB methods, it can be applied to CTC-based NAR models [ 34 ] . 2.1 Audio encoder We use the Conformer [ 8 ] for the audio encoder, comprising L encoder layers. The l -th encoder layer transforms its T -length d -dimensional hidden state representation input \bm{X}^{\text{in}}_{(l)}\in\mathbb{R}^{T\times d} into \bm{X}^{\text{out}}_{(l)}\in\mathbb{R}^{T\times d} as follows: \bm{X}^{\text{out}}_{(l)}=\mathrm{AudioEnc}_{(l)}(\bm{X}^{\text{in}}_{(l)}). (1) Then, the output of the l -th encoder layer \bm{X}^{\text{out}}_{(l)} is directly used as the input of the ( l +1)-th encoder layer \bm{X}^{\text{in}}_{(l+1)} as follows: \bm{X}^{\text{in}}_{(l+1)}=\bm{X}^{\text{out}}_{(l)}. (2) By iterating this process L times, the final output \bm{X}^{\text{out}}_{(L)}\in\mathbb{R}^{T\times d} is obtained. 2.2 Bias encoder The bias encoder comprises Transformer [ 37 ] layers and a mean pooling layer with a bias list \bm{B}=\{b_{1},\cdots,b_{N} }, where N represents the number of bias phrases. Each bias phrase b_{n} is represented as a sequence of static tokens (e.g., [“ Ra ”, “ pha ”, “ el ”]) in the pre-defined static vocabulary \mathcal{V}^{\text{static}} . The bias list \bm{B} is processed by the bias encoder to obtain phrase-level representations \bm{V}=[\bm{v}_{1},\cdots,\bm{v}_{N}]\in\mathbb{R}^{N\times d} as follows: \bm{V}=\mathrm{BiasEnc}(\bm{B}). (3) The bias phrases in the bias list \bm{B}=\{b_{1},\cdots,b_{N} } are incorporated into the dynamic vocabulary \mathcal{V}^{\text{dynamic}}=\{<\hskip-3.0ptb_{1}\hskip-3.0pt>,\cdots,<\hskip-3.0ptb_{N}\hskip-3.0pt>\} of size N , where each entry represents the corresponding bias phrase as a single dynamic token (e.g., [ < Raphael > ]). 2.3 CTC scoring layer with dynamic vocabulary Similar to conventional CTC, the scoring layer predicts an alignment sequence A=[a_{1},\dots,a_{T}] instead of directly predicting the output token sequence Y . Unlike conventional CTC, the scoring layer expands the static vocabulary \{\mathcal{V}^{\text{static}}\cup\{\phi\}\} of size K to include the dynamic vocabulary \mathcal{V}^{\text{dynamic}} of size N by incorporating \bm{V} as follows: P(Y\mid\bm{X}^{\text{out}}_{(L)},\bm{V})=\sum_{A\in\mathcal{B^{\text{-1}}}(Y)}P(A\mid\bm{X}^{\text{out}}_{(L)},\bm{V}), (4) where \phi represents the

Read the complete text