Building English ASR model with regional language support

Purvi Agrawal , Vikas Joshi , Bharati Patidar, Ankur Gupta, Rupesh Kumar Mehta · 2025

Read Building English ASR model with regional language support by Purvi Agrawal , Vikas Joshi , Bharati Patidar, Ankur Gupta, Rupesh Kumar Mehta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07522.txt.

Introduction

Purvi Agrawal and co-authors build an English speech reader that also hears Hindi in a 2025 paper. Users in India oft mix Hindi asks into English speech tasks. A lone English model stumbles on those asks and loses trust.

Their model shares deep layers across both tongues, with split heads per tongue. Self attention weighs which head to trust for each sound stretch. Text models from English and from Hindi in Latin script are blended too. Word errors fell hard on Hindi tests and a bit on English. Mixed speech homes can thus use one reader for both tongues.

Catalog record

CREATORPurvi Agrawal , Vikas Joshi , Bharati Patidar, Ankur Gupta, Rupesh Kumar Mehta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Building English ASR model with regional language support Purvi Agrawal , Vikas Joshi , Bharati Patidar, Ankur Gupta, Rupesh Kumar Mehta Abstract In this paper, we present a novel approach to developing an English Automatic Speech Recognition (ASR) system that can effectively handle Hindi queries, without compromising its performance on English. We propose a novel acoustic model (AM), referred to as SplitHead with Attention (SHA) model, features shared hidden layers across languages and language-specific projection layers combined via a self-attention mechanism. This mechanism estimates the weight for each language based on input data and weighs the corresponding language-specific projection layers accordingly. Additionally, we propose a language modeling approach that interpolates n-gram models from both English and transliterated Hindi text corpora. Our results demonstrate the effectiveness of our approach, with a 69.3 % 69.3\% and 5.7 % 5.7\% relative reduction in word error rate on Hindi and English test sets respectively when compared to a monolingual English model. † † address: Microsoft Corporation Email: [email protected], {vijoshi, bpatidar, angup, rupeshme }@microsoft.com 1 Introduction We analysed the queries on our Indian English (en-IN) automatic speech recognition (ASR) system and observed the following: • About one-third of the queries were entirely in regional languages. For example, “Kesariya ka bhakti gaana dikhaiye” is one such query, where both the entity and language of conversation are in Hindi. Note that the Hindi words are represented in Latin characters for wider readability. The English translation of this query is “show Kesariya’s bhakti song”. • Most entertainment queries had regional language movie or song names. • More than 30\% of the call center queries involved code-switching to regional languages for all utterances. It is becoming increasingly evident that users may interact in regional languages, even if they have selected English as their preferred language. While current voice assistants can recognize entities from different languages, they still face challenges when the entire query, including the language of conversation, is in another language. Some voice assistants now support bilingual queries, allowing users to interact interchangeably in English and Hindi without explicitly setting the language of the conversation. This system is achieved by using two monolingual models in tandem and selecting the output language based on the decision of the language identifier (LID) [ 1 ] . However, using such a system much higher computation cost than monolingual models, as both monolingual models must produce recognition outputs, increasing the cost of recognition for the end user. Additionally, the performance of these systems is sensitive to the accuracy of the LID, as the monolingual models perform well in their respective languages but suffer greatly in other languages. The aim of this work is not to build a truly bilingual model that can recognize both languages equally well. Instead, the goal is to improve the performance of the Indian English (en-IN) ASR model on Hindi (hi-IN) queries, bringing it reasonably close to the performance of the Hindi model on Hindi queries. This is to be achieved without increasing the computational costs or reducing its performance on English queries. This model is referred to as the enhanced en-IN ASR model. It is more pragmatic and commercially viable, as it can simply replace the existing monolingual en-IN ASR model without affecting the accuracy or computational cost for existing users. In this work, we propose a novel approach to build an en-IN ASR model that improves performance on Hindi queries without regressing on English queries. Our approach includes a split-head with attention (SHA) acoustic model (AM), which has shared hidden layers and two language-specific projection layers. The output of the language-specific projection layers are weighed by corresponding weights obtained from an attention model and then added to obtain posterior probabilities over the labels, chenones in this case [ 2 ] . It is important to note that in our approach, both the outputs from language-specific projection layers must correspond to the same set of chenones in order to be added together. This differs from a Shared Hidden Layer modeling approach [ 3 ] , where each language can have its own set of output labels. The language model (LM) is obtained by interpolating an en-IN n-gram LM with the one obtained from transliterated Hindi text via domain adaptation approach. Additionally, we propose to use contextual rather than word-based transliteration approach. With these innovations in AM and LM, our enhanced en-IN ASR model shows 69.3\% and 5.7\% word error rate relative (WERR) reduction over a conventional en-IN model on Hindi and English test sets, respectively. 2 Relation to prior work In the acoustic modeling , transfer learning, multilingual and multi-dialect modeling are widely used in hybrid ASR systems [ 4 , 5 , 6 , 7 , 8 , 9 , 3 , 10 , 11 , 12 ] . Transfer learning methods [ 7 , 8 ] leverage a well trained AM from high-resource language to bootstrap the target AM. A natural extension is to train multilingual model [ 9 , 3 , 10 ] , by using data from multiple languages to build robust seed model. However, such a model has inferior performance compared to their monolingual counterparts, as reported in [ 13 , 14 ] . The approach of multiple monolingual models in tandem and selecting an output based on LID results into a costly system for the end user [ 15 ] . Hence, our goal in this paper is to design an enhanced AM in hybrid framework without regressing on primary language and increase in user cost. In LM, similar to work discussed in [ 16 ] , a core aspect of our approach is the transliteration of regional language data to the Latin script. However, in order to keep our LMs lightweight, fast and easily modifiable, we do not build neural LMs, and instead, focus on interpolation-based techniques. Such techniques [ 17 , 18 , 19 , 20 , 21 ] have been around for a while, but to the best of our knowledge, it has not been used to support a secondary language (Hindi, in this case) while not regressing on the primary language (English, in this case). There are some approaches that focus purely on the problem of code-mixing, such as [ 22 , 23 , 24 ] . Some of these approaches require code-mixed training data or additional information such as language at run-time. It is important to note that our goal is different from these with our goal being to build a good English LM with the additional support of Hindi, not necessarily interspersed with each other or occurring in the same utterance. The rest of the paper is organized as follows. Section 3 discusses the proposed AM in detail, followed by proposed LM in Section 4 . Section 5 discusses the experiments and results, followed by Summary in the last Section. 3 Proposed AM: SplitHead with Attention (SHA) In this section, we will discuss the model architecture, data preparation and training strategy for our proposed approach. 3.1 Model architecture We introduce Splithead with Attention (SHA) model, a novel architecture designed for multilingual and code-mixed speech recognition. It can be broadly divided into two parts: (a) SplitHead model : The SplitHead AM features language-specific projection/output layers and

Read the complete text