Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization

Luong, Khanh, Vinh, Bao, Tan, Duc · 2025

Read Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization by Luong, Khanh, Vinh, Bao, Tan, Duc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24229.txt.

Introduction

Speech recognizers hear "twenty twenty five" but readers need "2025", a cleanup step called inverse text normalization. Luong Ho and co-authors built a version in 2025 that works live, while the words are still arriving, rather than waiting for the full sentence. Doing that with little context and low delay had been little explored.

Their system starts from a pretrained language model and adds a dynamic context scheme that stretches or shrinks the chunk it reads and peeks at words just ahead. On a Vietnamese dataset it matched the accuracy of offline systems and beat earlier live ones, all at low latency. The design drops into streaming recognizers as they run.

Catalog record

CREATORLuong, Khanh, Vinh, Bao, Tan, Duc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization Luong, Khanh, Vinh, Bao, Tan, Duc Abstract Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text, enhancing both readability and usability. Despite its importance, the integration of streaming ITN within streaming ASR remains largely unexplored due to challenges in accuracy, efficiency, and adaptability, particularly in low-resource and limited-context scenarios. In this paper, we introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness. To address streaming constraints, we propose Dynamic Context-Aware during training and inference, enabling adaptive chunk size adjustments and the integration of right-context information. Experimental results demonstrate that our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset, all while maintaining low latency, ensuring seamless integration into ASR systems. Ho Le Pham Nguyen Tran Chau Zalo AIVietnam University of ScienceHo Chi Minh City, Vietnam Vietnam National UniversityHo Chi Minh City, Vietnam keywords inverse text normalization, speech recognition, streaming, pretrained language model † † email: {congluong2122001,khanhld218,phamvinh257,baonguyenduy2002,tranctan96}@gmail.com, [email protected] 1 Introduction Inverse Text Normalization (ITN) is a natural language processing task that converts spoken-language transcriptions, typically produced by automatic speech recognition (ASR) systems, into their proper written form. For example, the spoken phrase “three point five dollars” may need to be transformed into “$3.50”. ITN is essential for improving text accuracy and readability in applications like document generation, voice assistants, and machine translation. The task involves resolving linguistic variations in numbers, dates, currencies, and more, often requiring contextual understanding to handle ambiguities. Recent advancements in ITN methods emphasize end-to-end neural network-based models, typically employing sequence-to-sequence (seq2seq) architectures [ 1 , 2 , 3 ] , which directly convert spoken-form input into written-form output. However, standard seq2seq models that rely on content-based attention often face challenges, including insertion, substitution, and deletion errors [ 4 , 5 ] . The most significant drawback of these systems is their tendency to produce unrecoverable errors that, while linguistically coherent, fail to preserve the original information accurately. The hybrid approach addresses these issues by combining a neural tagging model with finite state transducer (FST) based rules [ 1 , 5 , 6 , 7 ] , leveraging the adaptability of neural models and the precision of FST. This method enhances accuracy by mitigating common errors, preserving unmodified text, and efficiently handling overlapping source and target sentences. Its modular design improves scalability, allowing for easy rule adjustments without retraining the neural model, making it highly effective for ITN tasks. However, the hybrid approach requires significant effort to label inside–outside–beginning (IOB) tags for each word in a sentence, demanding considerable time and human resources. Although seq2seq methods also face data challenges, training them in an end-to-end manner by converting target sequences to source sequences generally involves less effort compared to the intensive tagging required for hybrid methods. To address the scarcity of training data, particularly in-domain data, recent advanced approaches have begun leveraging pre-trained language models [ 3 , 4 , 5 , 6 , 8 ] or adopting data-driven strategies [ 2 , 7 , 9 ] . Integrating ITN into a streaming ASR system can be approached in two ways. The first is applying ITN after a silence period or end of speech, which has been the focus of most previous research. The second, more challenging approach, involves applying ITN in real-time as the user speaks, a scenario that has received limited attention in publications. In streaming ITN, seq2seq models face disadvantages due to their inability to operate in a streaming manner. Their encoder-decoder architecture, reliant on cross-attention, makes real-time processing inherently difficult. To date, only [ 10 ] has introduced a streaming ITN method to the community. However, their approach faces significant challenges, particularly in scenarios with limited data, such as conversational data, as their experiments are conducted in a large dataset setting. Additionally, their method relies on a fixed chunk size for processing, which does not align with variable ASR outputs. In practice, ASR systems may generate more or fewer words than the predefined chunk size of a streaming ITN model, resulting in cumulative unprocessed words that negatively impact the real-time experience. This fixed chunk size limitation is particularly problematic for tasks like punctuation restoration. For instance, when processing the phrase “are you ok peter” with a chunk size of three, the output may incorrectly become “Are you OK? peter” instead of the intended “Are you OK, Peter?”. In this paper, we propose a novel approach for streaming ITN tasks that achieves results comparable to non-streaming methods, even with limited datasets, while also addressing the challenge of insufficient surrounding context in chunk-based processing of complex sentences. Specifically: 1. Streaming Pretrained Language Model: We are the first to introduce a streaming ITN approach that leverages a pretrained language model. By modifying the architecture of the pretrained model, we enable real-time processing while taking advantage of the pretrained weights. 2. Dynamic Context-Awared Training and Inference: We propose a novel method to enhance the model’s robustness across varying context sizes during both training and inference. This approach allows the model to process variable chunk sizes and right context lengths, unlike the fixed chunk size used in [ 11 ] . This flexibility enables instant inverse normalization during speech, thus improving performance and user experience. 2 Streaming Inverse Text Normalization With Pretrained Language Models 2.1 Pre-trained Language Models Bidirectional Encoder Representations from Transformers (BERT) [ 12 ] leverages the Transformer architecture to learn deep and bidirectional representations across all layers. During pre-training, BERT is trained on a massive, unlabeled text corpus using two unsupervised objectives: Masked Language Modeling and Next Sentence Prediction. These tasks enable the model to learn rich contextual representations. Subsequently, the pre-trained BERT model is fine-tuned on task-specific labeled datasets. This involves initializing the model with the pre-trained parameters and updating them to optimize performance on downstream tasks such as Question Answering and Natural Language Inference. In the context of Vietnamese language modeling, PhoBERT [ 13 ] , a BERT-based model trained on a 20GB word-level Vietnamese corpus, has achieved state-of-the-art performance. It excels in diverse Vietnamese NLP tasks, including word-level tasks like Part-of-Speech tagging, Dependency parsing, and Named-Entity Recognition, as well as the more complex Natural Language Inference task. PhoBERT, with 135 million parameters, leverages 12 encoder layers, 12 attention heads, and a 768 hidden size, crucially incorporating a Byte-Pair Encoding tokenizer optimized for the nuances of the Vietnamese language. Given our focus on Vietnamese language processing, we employ PhoBERT as the pre-trained foundation for the sequential IOB tagging problem, which is subsequently adapted to accommodate streaming requirements. 2.2 Weighted Finite-State Transducers Traditional rule-based approaches to ITN often struggle with the complexity and variability of natural language. To address these limitations, Weighted Finite-State Transducers (WFST) [ 14 ] have emerged as a powerful and flexible framework. WFST enables the representation of complex linguistic transformations as a network of states and transitions, where weights can encode probabilities or costs associated with different normalization options. This allows for efficient and robust ITN, capable of handling ambiguities and variations in spoken language. 2.3 Model Architecture Figure 1: Streaming Pretrained Language Model for Inverse Text Normalization.

Read the complete text