TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition

Minh N. H. Nguyen, Anh Nguyen Tran, Dung Truong Dinh, Nam Van Vo · 2025

Read TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition by Minh N. H. Nguyen, Anh Nguyen Tran, Dung Truong Dinh, Nam Van Vo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05983.txt.

Introduction

Recognizing speech that flips between Vietnamese and English mid-sentence is the challenge taken up in a 2025 paper led by Tran Nguyen Anh with three co-authors. Code-switching strains general speech recognizers, and this language pair is especially tricky because the two phonologies differ sharply yet some sounds are easily confused. The authors answer with TSPC, a two-stage phoneme-centric architecture.

TSPC routes recognition through an extended Vietnamese phoneme set, an intermediate alphabet of sounds shared across both languages, while staying light on computing resources. In experiments it consistently beats existing baselines, including PhoWhisper-base, reaching a much lower word error rate with less training. The two-stage design also allows phoneme adaptation and language conversion to clean up hard mixed-language stretches.

Catalog record

CREATORMinh N. H. Nguyen, Anh Nguyen Tran, Dung Truong Dinh, Nam Van Vo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TSPC: A Two-Stage Phoneme-Centric Architecture for code-switching Vietnamese-English Speech Recognition Minh N. H. Nguyen, Anh Nguyen Tran, Dung Truong Dinh, Nam Van Vo Abstract Code-switching (CS) presents a significant challenge for general Auto-Speech Recognition (ASR) systems. Existing methods often fail to capture the subtle phonological shifts inherent in CS scenarios. The challenge is particularly difficult for language pairs like Vietnamese and English, where both distinct phonological features and the ambiguity arising from similar sound recognition are present. In this paper, we propose a novel architecture for Vietnamese-English CS ASR, a Two-Stage Phoneme-Centric model (TSPC). The TSPC employs a phoneme-centric approach, built upon an extended Vietnamese phoneme set as an intermediate representation to facilitate mixed-lingual modeling. Experimental results demonstrate that TSPC consistently outperforms existing baselines, including PhoWhisper-base, in Vietnamese-English CS ASR, achieving a significantly lower word error rate of 19.9% with reduced training resources. Furthermore, the phonetic-based two-stage architecture enables phoneme adaptation and language conversion to enhance ASR performance in complex CS Vietnamese-English ASR scenarios. † † address: ⋆ Vietnam - Korea University of Information and Communication Technology, The University of Danang, Vietnam † † address: Vietnam - Korea University of Information and Communication Technology, The University of Danang, Vietnam ⋆ Equal Contribution 1 Introduction Automatic Speech Recognition (ASR) has gained significant attention in recent years and proliferated as an emerging interaction method between humans and machines. Despite the continuous development of multilingual ASR approaches in the globalization era, code-switching remains extremely challenging, where speakers randomly shift between languages within a conversation. Existing multilingual ASR approaches are still limited due to the inability to capture the unique pronunciation nuances of each language. To investigate this limitation, we conducted a comparative analysis of multilingual and monolingual models’ performance in recognizing the mix of Vietnamese and English languages, the illustration is presented in Table 1 . † † nicematrix-placeholder: NiceTabular (nicematrix) Table 1: The results shown by monolingual and multilingual models, where the English words in bold are incorrect prediction. Recent End-to-End ASR approaches often overlook inherent phonological structures, leading to significant performance degradation in the mix of language scenarios. Hence, several approaches have been proposed to enhance the traditional ASR architectures, such as the work in Deep context [ 4 ] , which introduced a bias-encoder for embedding contextual phrases into dimensional representations. However, the approach encounters limitations when processing a large number of context phrases since the model struggles to differentiate highly correlated phrases within extensive corpora. Other approaches integrate a separate Language Identification (LID) module [ 5 , 6 ] into the main architecture to identify both primary and secondary languages, while the effectiveness of the LID component relies heavily on a large volume of code-switched data, which poses a significant challenge for low-resource languages. Additionally, [ 7 ] proposed AdaCS, employing a bias attention module to improve the generalization for unseen language mixes without retraining the model, especially in Vietnamese scenarios. In most of the ASR approaches, linguistic factors such as tone, which plays a crucial role in ASR, are often overlooked, especially for tonal languages like Vietnamese, since Vietnamese relies on pitch variations to express distinct word meanings, a characteristic absent in non-tonal languages like English. Hence, tonal dependency, coupled with substantial phonological overlap between Vietnamese and English (e.g., “ lít ” vs. “ list ”), poses significant challenges for ASR systems in CS scenarios. As a result, tone-insensitive ASR models frequently face recognition errors and fail to differentiate phonetically similar lexical items. Addressing tone and language-specific phonological features is crucial to enhance ASR performance in multilingual and code-switched environments. In this work, we propose the TSPC model, a novel framework for Vietnamese-English CS ASR. In particular, the proposed model adopts a phoneme-centric approach, leveraging extended Vietnamese phonemes as a linguistic bridge for mixed-lingual modeling by comprising two stages: a Speech-to-Phone (S2P) module that represents acoustic input to phoneme sequences, and a Phone-to-Text (P2T) module that translates phoneme sequences into text. 2 Phoneme-Centric Model 2.1 Extended Vietnamese Phonemes with English Conversion In code-switched scenarios, phoneme recognition is the fundamental linguistic representation that bridges the acoustic signal and text sequences [ 8 , 9 ] . The phoneme-level representation provides an intermediate representation to enhance the generalization of ASR models across languages and dialects. The problem becomes increasingly critical in many languages in the globalization era. Representation at the phonetic level enables the identification of linguistic intersections, revealing cross-linguistic similarities in phonetic notation and pronunciation, especially in naturalistic contexts where languages mix, such as Vietnamese and English. Despite existing differences in phonological systems across languages, comparative analysis in [ 10 ] shows that both Vietnamese and English utilize a range of similar consonants and vowels, such as ([p], [b], [m], [n], [i]). The overlapping issue that appears in code-switched scenarios can cause recognition errors for acoustic models that struggle to differentiate between highly similar phonemes. Moreover, Vietnamese is a highly tonal language, characterized by short, simple syllables and six distinct lexical tones [ 11 ] that determine the meaning of the words. Particularly, tone acts as a primary component and significantly influences Vietnamese speakers when speaking non-tonal languages like English, e.g.: English diphthong “eI” is replaced with the Vietnamese syllable “ây” . † † nicematrix-placeholder: NiceTabular (nicematrix) Table 2: Comparative analysis of English words, an English word is separated to prefix and postfix parts, where Vietnamese syllable (vi-syllable) is mapped with IPA, and converted to Vietnamese phoneme (phone). We perform a comprehensive comparative validation to examine the overlapping phonemes in the context of Vietnamese-English CS based on sound similarities extracted based on the pre-trained Vietnamese ASR model, as shown in Fig. 1 . From this study, we obtained preliminary results for converting the pronunciation of English words into Vietnamese syllables, as illustrated in the Table 2 . Subsequently, we utilized existing phonetic vocabularies and conversion modules to convert represented Vietnamese syllables into precise phonetic representations. For instance, the English word “ a ” is mapped with the Vietnamese syllable “ ây ”, which was then converted to the phonetic form “ @ - 0 iz ”. The phoneme conversion methodology demonstrates a robust method based on the intricate phonetic interactions and adaptations observed in cross-linguistic contexts. Figure 1: T-SNE visualization of English words in blue and Vietnamese syllables in red. Figure 2: Two-stage phoneme-centric model (TSPC), where predicted phoneme sequence represents the input for P2T model. 2.2 Two-stage Model Development The proposed two-stage ASR model utilizes phonemic representation by combining two foundation models, such as Speech-to-Phone (S2P) and Phone-to-Text (P2T) that are independently pre-trained, then integrated and fine-tuned for code-switched scenarios. Speech-to-Phone: For the S2P model, we adopt the well-established Sequence-to-Sequence (Seq2Seq) paradigm [ 12 ] for speech-to-text tasks due to its efficient architecture for learning complex acoustic features and mapping variable-length input speech to a sequence of phonemes. Specifically, the S2P model employs a pre-trained Encoder built on large-scale Vietnamese datasets that can capture rich acoustic feature extraction capabilities. Following the acoustic Encoder, different variants of Decoders are validated for generating phoneme sequences. Throughout our comprehensive experiments in our prior work in [ 13 ] , we can figure out the most suitable Decoder for phoneme recognition as shown in the in Table 3 . † † nicematrix-placeholder:

Read the complete text