BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language

Muhammad Ali · 2026

Read BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language by Muhammad Ali (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03504.txt.

Introduction

Ali presents BaltiVoice, the first public speech resource for the Balti language. Balti is a Tibetic language spoken in Gilgit-Baltistan, Pakistan. No public data for speech recognition in Balti existed before this work. The corpus holds 16.8 hours of read speech. It contains 10,060 checked utterances in native Nastaliq script, drawn from Mozilla Common Voice recordings.

A fine-tuned Whisper-small model anchors the recognition system. On a speaker-disjoint validation set, it reaches a word error rate near 25 percent. Its character error rate is near 8 percent. The zero-shot baseline, by contrast, errs on nearly every word. A smaller Whisper-base model scores far worse on the same data. That gap shows model capacity matters in this low-resource setting. The dataset, the tuned model, and a live demo are public on HuggingFace. Other low-resource language teams can copy the recipe.

Catalog record

CREATORMuhammad Ali
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language Muhammad Ali Abstract We present BaltiVoice, a 16.8-hour read-speech corpus for Balti (ISO 639-3: \fontspec_if_language:nT ENG \addfontfeature Language=Englishbft ), a Tibetic language spoken in Gilgit-Baltistan, Pakistan, with no prior publicly available ASR resources. The corpus contains 10,060 validated utterances in native Nastaliq script, derived from Mozilla Common Voice recordings. Fine-tuning OpenAI Whisper-small yields a Word Error Rate (WER) of 26.74% and a Character Error Rate (CER) of 8.67% on the 538-utterance speaker-disjoint validation set, down from a zero-shot baseline of 159.19% WER and 152.52% CER. A Whisper-base fine-tuned on the same data achieves 44.54% WER and 15.61% CER, confirming that model capacity matters for this low-resource setting. The dataset, fine-tuned model, and a live transcription demo are publicly available on HuggingFace. \fontspec_if_language:nTENG\addfontfeatureLanguage=English1 \fontspec_if_language:nTENG\addfontfeatureLanguage=English1 \fontspec_if_language:nT ENG \addfontfeature Language=English1 Dataset: \fontspec_if_language:nT ENG \addfontfeature Language=Englishhttps://huggingface.co/datasets/mohdali1/baltivoice-asr Model: \fontspec_if_language:nT ENG \addfontfeature Language=Englishhttps://huggingface.co/mohdali1/whisper-small-balti Demo: \fontspec_if_language:nT ENG \addfontfeature Language=Englishhttps://huggingface.co/spaces/mohdali1/baltivoice-demo Code: \fontspec_if_language:nT ENG \addfontfeature Language=Englishhttps://github.com/mohdali-dev/BaltiVoice-ASR \fontspec_if_language:nT ENG \addfontfeature Language=English1 Introduction Balti is spoken by roughly 400,000 people in the Gilgit-Baltistan region of Pakistan and parts of Ladakh, India. It belongs to the Tibetan branch of the Sino-Tibetan family and is written in a Nastaliq-based script adapted from Urdu. Despite having a distinct phonology and grammar, Balti has almost no presence in NLP or speech research. There is no published ASR system, no annotated speech corpus in any public repository, and no Balti entry in major multilingual benchmarks. This gap matters practically and scientifically. Speakers cannot use voice interfaces or dictation tools in their own language. Researchers have no baseline to measure progress against. This paper addresses both problems by releasing a corpus, reproducible baselines, and a benchmark that future work can build on. Our contributions are: \fontspec_if_language:nT ENG \addfontfeature Language=English• BaltiVoice corpus : 10,060 utterances, 16.8 hours of speech, native Nastaliq transcriptions, train/validation split, released under CC0 on HuggingFace. \fontspec_if_language:nT ENG \addfontfeature Language=English• Baseline models : fine-tuned Whisper-small (26.74% WER) and Whisper-base (44.54% WER), both publicly released on HuggingFace. \fontspec_if_language:nT ENG \addfontfeature Language=English• A reproducible training pipeline with code, a Colab notebook, and a live Gradio demo. A 27% WER means roughly one word in four is wrong. That is too high for dictation, but enough to establish a measurable starting point for a language that previously had none. Future work aims to reduce this error rate through extended training and data augmentation. BaltiVoice establishes the first public benchmark for Balti ASR, providing a corpus, two baseline models, and a reproducible pipeline that future work can build on and improve. \fontspec_if_language:nT ENG \addfontfeature Language=English2 Related Work Low-resource ASR. Fine-tuning large pretrained models on small target-language datasets has produced consistent results across under-resourced languages. 2 showed that XLS-R fine-tuned on 10 minutes of labeled speech outperforms supervised systems trained on hundreds of hours in some settings. 6 , trained on 680,000 hours of web audio, covers 99 languages but not Balti, and shows near-random behavior on languages outside its training distribution. Whisper fine-tuning for low-resource languages. Several groups have adapted Whisper to languages absent from its pretraining. 9 fine-tuned Whisper on Yoruba and Swahili with 5–20 hours of data, achieving WERs between 28% and 45%. 3 reported similar ranges for Hindi dialects. These results suggest that 15–20 hours of labeled audio is enough for Whisper fine-tuning to produce usable, if imperfect, transcription. Tibetan language resources. Tibetan, the closest well-resourced relative to Balti, has received limited but growing NLP attention. 5 released a Tibetan TTS dataset, and 8 built an ASR system for standard Lhasa Tibetan using deep neural networks. No published work addresses Balti specifically. Mozilla Common Voice. Common Voice ( 1 ) hosts community-contributed read-speech data in over 100 languages. Balti was added in 2023. As of our data collection, the Balti subset contained 10,547 recorded clips, of which 10,060 passed the platform’s validation threshold. Keyword spotting for low-resource languages. 7 survey keyword spotting approaches for Urdu, noting that Nastaliq-scripted languages face unique challenges in token boundary detection that transfer learning only partially addresses. This directly motivates our choice to treat Balti ASR as a transfer learning problem from Urdu-adjacent script representations. \fontspec_if_language:nT ENG \addfontfeature Language=English3 Dataset \fontspec_if_language:nT ENG \addfontfeature Language=English3.1 Source and Collection BaltiVoice is derived from the Mozilla Common Voice Balti ( \fontspec_if_language:nT ENG \addfontfeature Language=Englishbft ) dataset. Volunteers recorded themselves reading Balti sentences aloud; other volunteers validated each recording by voting it as correct or incorrect. We used only validated recordings. \fontspec_if_language:nT ENG \addfontfeature Language=English3.2 Statistics Table summarizes the corpus statistics. Property Value Language Balti (bft) Script Nastaliq (Arabic) Total utterances 10,060 Total duration 16.8 hours Mean clip duration 6.00 seconds Mean words per utterance 10.12 Mean characters per utterance 48.80 Total speakers 136 Train speakers 122 Dev speakers 14 (disjoint) Gender (female) 1,246 utterances Gender (undisclosed) 1,030 utterances Age (twenties) 3,758 utterances Age (thirties) 3,697 utterances Age (teens) 97 utterances Train split 9,519 Validation split 538 (speaker-disjoint) Audio format 16 kHz mono WAV \fontspec_if_language:nT ENG \addfontfeature Language=English \fontspec_if_language:nT ENG \addfontfeature Language=English \fontspec_if_language:nT ENG \addfontfeature Language=English Hyperparameter Value Optimizer AdamW Learning rate 1\times 10^{-5} Warmup steps 100 Effective batch size 16 (8 \times 2 accum.) Max steps 1,000 Precision fp16 Gradient checkpointing Enabled Best model selection Min. validation WER Hardware NVIDIA Tesla T4 Training time 1h 54m \fontspec_if_language:nT ENG \addfontfeature Language=English Model Params WER (%) CER (%) Whisper-small (zero-shot) 244M 159.19 152.52 Whisper-base (fine-tuned) 74M 44.54 15.61 Whisper-small (fine-tuned) 244M 26.74 8.67 \fontspec_if_language:nT ENG \addfontfeature Language=English \fontspec_if_language:nT ENG \addfontfeature Language=English \fontspec_if_language:nT ENG \addfontfeature Language=English0 table Qualitative examples from the validation set. The model struggles with morphological endings and consonant clusters but captures core lexical items. Type Example (Reference vs. Prediction) Correct Ref: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduہئی لے بُوژھا یا کھچیم پو مہ زیرفو سی Pred: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduہئی لے بُوژھا یا کھچیم پو مہ زیرفو سی (9-word utterance, fully matched.) Partial Error Ref: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduزیربا نہ تا درے ترانگمو یودپی، کھیانگ شیو پژے ن٘ا بید Pred: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduزیربا نہ تا درے ترانگمو یودپی، کھیانگ شیو پژے ن٘ا بیاس (Final characters substituted: bīd \to biyās . Common morphological error.) Substitution Ref: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduغدیانگ چھودے دانشؔرگہ کھن کُن نہ دیرے Pred: arab \addfontfeature Script=Arabic \fontspec_if_language:nT URD \addfontfeature Language=Urduغدیانگ چھودے دانشؔرگہ کھن کُن نہ دیر (Final vowel/ezafe dropped: dīre \to dīr . Phonetic simplification.) Most errors are single-character substitutions at word endings, consistent with the model learning lexical patterns but making morphological mistakes in an agglutinative language. Whole-word deletion or insertion errors are less frequent. These patterns reflect two Balti-specific challenges. First, Balti morphology is agglutinative, meaning suffixes attach to stems to mark tense, case, and agreement, so a single character error at a word boundary changes grammatical meaning entirely. Second, Nastaliq Unicode ambiguity between visually similar characters (e.g., Arabic Yeh vs. Farsi Yeh) produces substitution errors that WER counts equally to semantic errors, likely inflating the reported rate. \fontspec_if_language:nT ENG \addfontfeature Language=English4 Discussion A 26.74% WER is not good enough for dictation or accessibility tools. A Balti speaker would need to correct roughly one in four words. For narrower tasks like keyword spotting or topic detection, where exact transcription is less critical,

Read the complete text