Building Robust and Scalable Multilingual ASR for Indian Languages
Arjun Gangwar, Kaousheik Jayakumar, S. Umesh · 2025
Read Building Robust and Scalable Multilingual ASR for Indian Languages by Arjun Gangwar, Kaousheik Jayakumar, S. Umesh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15418.txt.
Introduction
Arjun Gangwar, Kaousheik Jayakumar, and S. Umesh describe multilingual speech recognition systems for Indian languages built for a shared challenge. Systems must predict both the language and dialect of each utterance among eight languages and thirty-three dialects. Both challenge tracks restrict extra data, requiring largely from scratch systems in one track. Their core idea uses a multi decoder design.
A phonemic common label set serves as an intermediate representation between audio and text. That step improves performance in phonemic space over the baseline. Further methods preserve the gain when converting back to normal scripts. Their systems beat the baseline for three languages on error rates. They also achieve the highest language and dialect identification accuracy among all participants.
Catalog record
| CREATOR | Arjun Gangwar, Kaousheik Jayakumar, S. Umesh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15418
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building Robust and Scalable Multilingual ASR for Indian Languages Arjun Gangwar, Kaousheik Jayakumar, S. Umesh Abstract This paper describes the systems developed by SPRING Lab, Indian Institute of Technology Madras, for the ASRU MADASR 2.0 challenge. The systems developed focuses on adapting ASR systems to improve in predicting the language and dialect of the utterance among 8 languages across 33 dialects. We participated in Track 1 and Track 2, which restricts the use of additional data and develop from-the-scratch multilingual systems. We presented a novel training approach using Multi-Decoder architecture with phonemic Common Label Set (CLS) as intermediate representation. It improved the performance over the baseline (in the CLS space). We also discuss various methods used to retain the gain obtained in the phonemic space while converting them back to the corresponding grapheme representations. Our systems beat the baseline in 3 languages (Track 2) in terms of WER/CER and achieved the highest language ID and dialect ID accuracy among all participating teams (Track 2). I Introduction The ASRU MADASR 2.0 1 1 1 https://sites.google.com/view/respinasrchallenge2025/home challenge presents a multilingual, multi-dialect dataset spanning over 8 low-resource Indian languages. The challenge provides a dataset of 1200 hours - 150 hours per language - and is evaluated over 4 tracks. The ASR systems developed are evaluated on hidden test sets with metrics such as word error rate (WER), character error rate (CER), language ID accuracy (LID accuracy) and dialect ID accuracy (DID accuracy). These are the 4 tracks with different levels of restrictions. • Track 1 allows only the use of given 30 hours small subset per language with no no external data or models. • Track 2 allows only the use of given 120 hours large subset per language with no external data or models. • Track 3 - an extension of track 1 with the usage of any external data or pre-trained models (open-sourced). • Track 4 - an extension of track 2 with the usage of any external data or pre-trained models (open-sourced). This challenge is among the first works that open sources a labelled language ID and dialect ID tokens inclusive dataset to develop ASR systems for Indian languages. This paper presents the ASR systems developed by the SPRING Lab, IIT Madras for tracks 1 and 2. We mainly focus on leveraging the phonemic similarities among Indian languages through a common label set [ 4 ] and discuss ways to retain the gains from the ASR while converting back from the CLS space to corresponding graphemic notations. II Common Label Set A very strong correlation exists between phonemes and graphemes of Indian languages, which eases the (grapheme-to-phoneme) G2P conversion. Moreover, the strong phonemic and graphemic similarities between many Indian languages stem from their common roots in related language families. For instance, among the languages used in this challenge, Bhojpuri, Magahi, Marathi, and Chhattisgarhi use the same Devanagari script. Whereas Telugu and Kannada belong to the same Dravidian language family. We exploit these facts by converting the graphemic representations into a common phonemic space using the unified parser [ 1 ] . A standard set of labels are given to phonetically similar speech sounds among different Indian languages known as the Common Label Set. Examples of CLS representation across different languages is shown in Fig 1 . Reconstructing text in the native script from CLS representations is inherently challenging due to linguistic phenomena like schwa deletion, geminate correction, and the intricacies of syllable segmentation. This paper discuss a few text-to-text machine transliteration (MT) approaches to retain the gains obtained by the CLS ASR. III Multilingual ASR models We use the ESPnet toolkit [ 6 ] for all our experiments. Track 1 models are trained on the small dataset (approx. 240 hours), and Track 2 models are trained on the large dataset (approx. 1200 hours). We use 80-dimensional log-Mel Spectrogram as speech features for all our experiments. Fig. 1: CLS representations of different languages The Mel Spectrogram are computed using 80 Mel filter banks, with hop length of 10ms and window size of 25ms. All audio files are in 16KHz. We use unified parser to convert native text to CLS. For all our models, language ID and dialect ID information is passed as special token <LID\_DID> in the beginnning of both the CLS and native script text. III-A Baseline The baseline system follows a standard encoder-decoder architecture, comprising a Conformer [ 3 ] encoder and a Transformer decoder [ 5 ] . It takes log-Mel spectrograms as input to the encoder and generates output in the native script. A special token <LID> is prepended to the target text to indicate language identity. The baseline model does not predict dialects and is trained using a hybrid CTC-Attention loss [ 7 ] . III-B Approach 1: Cascaded ASR and MT models In this approach, we employ a cascaded system comprising two models. The first model performs speech recognition in the CLS (Common Label Set) space. It takes log-Mel spectrograms as input and generates transcriptions in the CLS format. The second model handles machine transliteration by converting the CLS transcriptions into the target language’s native script. Both the models are trained separately. For speech recognition, we use an encoder-decoder architecture with a Conformer encoder and a Transformer decoder. The model is trained using a hybrid CTC-Attention loss function. The machine transliteration model also follows an encoder-decoder architecture, utilising transformer layers for both the encoder and decoder, and is trained using only the attention-based cross-entropy loss. III-C Approach 2: Multi-Decoder model The multi-decoder architecture [ 2 ] consists of two sub-networks as shown in the Fig 2 : an ASR sub-network and an MT (machine transliteration) sub-network. The ASR sub-network employs a Conformer encoder followed by a Transformer decoder, while the MT sub-network uses a lightweight Transformer encoder and a Transformer decoder. The ASR sub-network takes log-Mel spectrograms as input and generates hidden states through its decoder. These hidden states are passed directly to the encoder of the MT sub-network. The MT decoder then produces the final output in the target language’s native script. The ASR sub-network is trained using a hybrid CTC-Attention loss, where the CTC loss is computed with respect to the CLS (Common Label Set) transcription. The MT sub-network is trained using an attention-based cross-entropy loss. Both the sub-nets are jointly trained. Fig. 2: Multi-decoder architecture Additionally, the MT decoder incorporates cross-attention over the speech encoder outputs, allowing it to access acoustic information directly. This auxiliary speech context helps correct errors in the ASR output during transliteration. An optional hierarchical encoder can also be inserted on top of the speech encoder, comprising additional Transformer layers and enabling an auxiliary CTC loss computed with native script transcriptions. This hierarchical setup has been shown to improve transliteration performance. The multi-decoder architecture retains the benefits of a cascaded system such as the ability to perform beam search and re-scoring on intermediate ASR outputs while reducing the error propagation typically associated with cascaded pipelines. III-C 1 From Scratch training In this approach, the weights of both sub-networks are randomly initialized. We observed that the ASR sub-network requires significantly more training time compared to the MT sub-network. Since the MT task is relatively easier in this setup, the