LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
Jianan Chen, Xiaoxue Gao, Kawahara Tatsuya and Nancy F. Chen · 2026
Read LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families by Jianan Chen, Xiaoxue Gao, Kawahara Tatsuya and Nancy F. Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20042.txt.
Introduction
Chen and co-authors benchmark large speech language models on low-resource speech recognition. Most past benchmarks stress high-resource languages, where these models already do well. Real use demands support for languages with little training data. LoASR-Bench covers 25 languages from 9 language families. It includes both Latin and non-Latin scripts for fair comparison. The tests expose limits of current speech models across these languages. The authors aim to guide deployment of such systems in truly multilingual settings.
Catalog record
| CREATOR | Jianan Chen, Xiaoxue Gao, Kawahara Tatsuya and Nancy F. Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18389 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20042
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families Jianan Chen, Xiaoxue Gao, Kawahara Tatsuya and Nancy F. Chen Abstract Large language models (LLMs) have driven substantial advances in speech language models (SpeechLMs), yielding strong performance in automatic speech recognition (ASR) under high-resource conditions. However, existing benchmarks predominantly focus on high-resource languages, leaving the ASR behavior of SpeechLMs in low-resource languages insufficiently understood. This gap is critical, as practical ASR systems must reliably support low-resource languages and generalize across diverse language families, and it directly hinders the deployment of SpeechLM-based ASR in real-world multilingual scenarios. As a result, it is essential to evaluate SpeechLMs on low-resource languages to ensure their generalizability across different language families. To address this problem, we propose LoASR-Bench , a comprehensive benchmark designed to evaluate lo w-resource a utomatic s peech r ecognition ( ASR ) of the latest SpeechLMs across diverse language families. LoASR-Bench comprises 25 languages from 9 language families, featuring both Latin and non-Latin scripts, enabling cross-linguistic and cross-script assessment of ASR performance of current SpeechLMs. Experimental results highlight the limitations of the latest SpeechLMs in handling real-world low-resource languages. I Introduction The recent emergence of large language models (LLMs) has significantly propelled the development of speech language models (SpeechLMs) [ 1 , 2 , 3 ] . These SpeechLMs typically leverage large-scale multimodal and multilingual pre-training to achieve strong performance across a wide range of speech-related tasks. With this rapid progress, benchmarks play a crucial role in systematically evaluating and comparing these SpeechLMs [ 4 , 6 , 5 ] . Existing benchmarks primarily assess model performance on tasks such as language identification, automatic speech recognition (ASR), and speech translation. A common approach to applying LLMs to speech tasks is to construct cascaded systems, where speech is first transcribed with ASR models. The transcripts are then passed into LLMs for text-to-text generation [ 8 , 9 , 10 ] . In this work, we concentrate on ASR, aiming to provide a focused and in-depth evaluation of SpeechLMs in this foundational and widely deployed task. Although cascaded approaches are practical for high-resource languages, performance often degrades in low-resource settings due to error propagation and limited training data [ 12 , 11 , 13 ] . To address these limitations, recent SpeechLMs have increasingly adopted end-to-end (E2E) architectures that are jointly trained on speech and text within a unified framework [ 14 , 15 , 16 , 17 , 20 , 19 , 18 ] . In addition, self-supervised learning approaches such as XLSR-53 [ 8 , 21 ] have shown strong potential in learning universal speech representations across languages. SpeechLMs are increasingly trained in multilingual settings to facilitate cross-lingual knowledge transfer [ 22 , 23 , 24 ] . Such transfer is particularly beneficial for low-resource languages, where both speech and text data are limited. By leveraging shared phonetic and semantic features across languages, SpeechLMs can improve ASR performance in low-resource scenarios [ 27 , 26 , 25 ] . This multilingual training strategy enables models to learn more universal representations. Numerous benchmarks have been developed to evaluate ASR systems from different perspectives. Early benchmarks focused primarily on English and other high-resource languages [ 32 , 29 , 31 , 28 , 33 , 36 , 30 , 34 , 35 ] . More recent efforts have expanded to multilingual settings, with benchmarks such as CommonVoice [ 37 ] and FLEURS [ 38 ] . However, these benchmarks typically treat languages independently and lack systematic analysis of typologically diverse language-family characteristics (i.e., Dravidian, Uralic, and Turkic) or script diversity (Latin vs. non-Latin). As a result, they provide limited insight into how ASR models generalize across languages. Fig. 1: Overview of LoASR-Bench covering low-resource languages from nine language families. To address these problems, we introduce LoASR-Bench , a benchmark designed to provide a comprehensive evaluation of Lo w-Resource A utomatic S peech R ecognition. LoASR-Bench encompasses 25 languages across 9 typologically diverse language families (Dravidian, Romance, Uralic, Isolate, Japonic, Sino-Tibetan, Indo-Aryan, Turkic, and Koreanic), including both Latin and non-Latin scripts. To the best of our knowledge, LoASR-Bench is the first benchmark to systematically compare the performance of SpeechLMs across different language families and script types. By introducing diversity across language families and scripts, LoASR-Bench enables a more realistic evaluation of SpeechLMs’ performance on low-resource languages. We evaluate state-of-the-art SpeechLMs, including XLSR-53, Whisper, and Qwen models, and demonstrate that current models encounter unique challenges in low-resource languages that are absent in resource-rich ones. Consequently, these gaps risk overestimating the generalizability of SpeechLMs to real-world low-resource languages. We find that models perform substantially better on Latin-script languages (i.e., Romance family) compared to non-Latin scripts (i.e., Dravidian and Indo-Aryan families). Notably, we observe that model size and multilingual pre-training significantly impact performance. Our main contributions include: • We present LoASR-Bench , a new benchmark designed specifically for low-resource ASR across diverse language families, covering 25 languages from 9 language families. • We conduct a systematic evaluation of different SpeechLMs, revealing significant performance variations on low-resource languages. • We perform comprehensive cross-family and cross-script comparisons, showing that languages with Latin scripts achieve lower error rates than non-Latin languages. • We conduct both language-aware and language-unaware inference for SpeechLMs and observe that identifying languages is crucial for ASR performance. • We propose to fine-tune XLSR-53, Whisper, and Qwen models across 25 target languages, covering several low-resource languages that lack publicly available checkpoints. This fine-tuning effort establishes a unified and reproducible set of SpeechLMs for low-resource ASR, with training code and checkpoints to be released upon acceptance. II LoASR-Bench To address the lack of systematic evaluation for low-resource ASR, we introduce LoASR-Bench , a comprehensive multilingual benchmark designed to evaluate SpeechLMs across 25 low-resource languages spanning 9 language families, as illustrated in Fig. 1 . TABLE I: ASR performance across language families and models. CER% is reported for Latin-script languages, and WER% for non-Latin scripts. Train (h) denotes training data duration in hours. Best results for each language or average row are shown in bold . Lower is better. Language Family Language (ISO) Train (h) xlsr-53 Whisper-M Whisper-L Qwen2-Audio Qwen2-Audio ft Qwen3-Omni Dravidian (WER) : non-Latin script Tamil (ta) 233 0.36 0.52 0.48 3.05 0.63 0.31 Telugu (te) 1 1.00 0.33 0.34 4.80 1.13 0.71 Malayalam (ml) 1 0.36 0.56 0.49 4.73 0.46 0.85 Avg. – 0.57 0.47 0.44 4.19 0.74 0.62 Romance (CER) : Latin script Portuguese (pt) 208 0.21 0.21 1.86 3.26 0.26 0.07 Spanish (es) 549 0.05 0.04 0.04 2.20 0.04 0.03 French (fr) 989 0.11 0.21 0.20 2.60 0.21 0.06 Italian (it) 353 0.25 0.18 0.20 0.19 0.30 0.06 Catalan (ca) 2670 0.07 0.17 0.15 2.40 0.10 0.17 Galician (gl) 48 0.08 0.15 0.95 1.75 0.03 0.10 Romanian (ro) 20 0.48 0.82 0.76 3.31 0.81 0.12 Avg. – 0.18 0.25 0.59 2.24 0.25 0.09 Uralic (CER) : Latin script Estonian (et) 46 0.24 0.12 0.14 0.75 0.43 0.34 Finnish (fi) 12 0.25 0.17 0.19 1.22 0.90 0.86 Hungarian (hu) 92 0.12 0.09 0.20 2.30 0.57 0.08 Avg. – 0.20 0.13 0.18 1.42 0.63 0.43 Isolate (CER) : Latin script Basque (eu) 220 0.45 0.06 0.04 3.55 0.06 0.25 Avg. – 0.45