Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages

V.S.D.S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger · 2026

Read Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages by V.S.D.S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18204.txt.

Introduction

Akavarapu, Daniel, and Jäger analyze speech recognition for Archi and Rutul, two hard East Caucasian languages with very little data. They gather existing recordings into clean resources of about 50 minutes for Archi and 1 hour 20 minutes for Rutul. For wav2vec2, heuristic output setup plus a phoneme vocabulary built for each language matches or beats Whisper. Error checks show accuracy tracks training frequency closely, so many errors blamed on sound complexity may just reflect scarce data.

Catalog record

CREATORV.S.D.S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48584
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages V.S.D.S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger Abstract We present a phoneme-level analysis of automatic speech recognition (ASR) for two low-resourced and phonologically complex East Caucasian languages, Archi and Rutul, based on curated and standardized speech–transcript resources totaling approximately 50 minutes and 1 hour 20 minutes of audio, respectively. Existing recordings and transcriptions are consolidated and processed into a form suitable for ASR training and evaluation. We evaluate several state-of-the-art audio and audio–language models, including wav2vec2, Whisper, and Qwen2-Audio. For wav2vec2, we introduce a language-specific phoneme vocabulary with heuristic output-layer initialization, which yields consistent improvements and achieves performance comparable to or exceeding Whisper in these extremely low-resource settings. Beyond standard word and character error rates, we conduct a detailed phoneme-level error analysis. We find that phoneme recognition accuracy strongly correlates with training frequency, exhibiting a characteristic sigmoid-shaped learning curve. For Archi, this relationship partially breaks for Whisper, pointing to model-specific generalization effects beyond what is predicted by training frequency. Overall, our results indicate that many errors attributed to phonological complexity are better explained by data scarcity. These findings demonstrate the value of phoneme-level evaluation for understanding ASR behavior in low-resource, typologically complex languages. 1 Introduction Figure 1: Phoneme-level F1-score as a function of (log) training frequency for one model–language pair, illustrating a characteristic sigmoid-shaped learning trend. Rutul and Archi are two East Caucasian languages with exceptionally rich sound systems that pose major challenges for modern automatic speech recognition (ASR). Archi, in particular, exhibits an unusually complex phonological system with 16 vowel phonemes and—depending on analytical assumptions—between 73 and 81 consonant phonemes, making it one of the largest non-click consonant inventories described to date. Rutul, represented here by its Kina variety, likewise features a large consonant inventory and special articulations such as pharyngealization. Both languages are highly endangered: Archi has only a few thousand speakers and is considered among the most severely endangered languages of Russia ( 19 ; 9 ) , while Rutul is classified as definitely endangered with an estimated \sim 30,000 speakers ( 1 ) . Precise counts are difficult due to the geographic dispersion of speaker communities across Daghestan. Crucially, there are no established ASR benchmarks or standardized resources for either language. Prior ASR research has largely focused on word- or character-level evaluation and has rarely examined phoneme-level behavior in typologically extreme languages, with only limited exceptions ( 23 , e.g.,) . As a result, it remains unclear whether errors attributed to phonological complexity reflect intrinsic difficulty or simply data scarcity. To address these gaps, we curate and standardize speech–transcript resources for Archi and Kina Rutul, consolidating material from linguistic documentation projects ( 20 ; 1 ) into a form suitable for ASR training and evaluation. The two corpora differ markedly in recording conditions: Kina Rutul consists primarily of spontaneous speech recorded in relatively noisy environments, whereas the Archi data comprise read speech produced by trained speakers under controlled conditions. Despite these differences, we observe broadly comparable trends across the two languages. We fine-tune and evaluate several state-of-the-art ASR models on these resources, building on recent work showing that useful phonetic recognizers can be obtained from minutes of speech when combined with multilingual pretraining ( 6 ) . By benchmarking multiple architectures under identical conditions, we assess how current ASR systems handle languages with extreme phonological inventories. Our phoneme-level analysis reveals a consistent pattern across most models and both languages: phoneme recognition accuracy, measured by F1 score, increases sigmoidally with the logarithm of training frequency (Figure 1 ). Performance is near zero for very rare phonemes, rises sharply once sufficient examples are observed, and saturates for frequent ones. This mirrors frequency effects reported in cognitive and statistical models of language processing, where logistic functions of log-frequency capture performance trends ( 16 ) . Contributions We summarize our main contributions as follows: 1. We curate speech–transcript resources for Archi and Kina Rutul, enabling systematic ASR benchmarking for two East Caucasian languages previously lacking such resources. 2. We benchmark multiple state-of-the-art ASR architectures including our introduction, a heuristic initialization trick by averaging of CTC based architecture, under extremely low-resource conditions, highlighting the advantages of speech-specialized models. 3. We provide a detailed phoneme-level error analysis, revealing a robust sigmoid-shaped relationship between phoneme recognition accuracy and training frequency. 2 Related work Approaches to low-resource ASR primarily rely on cross-lingual transfer through multilingual pretraining, often using shared phonemic representations such as IPA or standardized phoneme sets derived from high-resource languages ( 31 ; 22 ; 32 ; 29 ) . Another common strategy is augmenting end-to-end ASR systems with external n -gram language models to improve decoding in data-scarce settings ( 36 ; 14 ; 23 ) . More recently, heuristic initialization of model layers—particularly output embeddings for unseen phonemes—has been explored to better transfer phonetic knowledge across languages ( 37 ) . Parallel to these developments, several works combine pretrained speech encoders with Large Language Models (LLMs) ( 7 ; 5 ) , either keeping the language model frozen ( 11 ) or adapting it using parameter-efficient techniques such as Low-Rank Adaptation (LoRA) ( 17 ; 12 ) . Despite extensive linguistic documentation, computational resources and ASR studies for East Caucasian languages remain limited. There is no prior ASR work for Archi or Rutul, and only isolated efforts exist for other languages in the family, typically with very small datasets and without phoneme-level evaluation ( 24 ) . Finally, while ASR performance is sometimes analyzed by broad phonological features/categories (e.g., tone, nasality, vowel length) ( 25 ) or such features are utilized in building ASR systems ( 3 ) , detailed phoneme-level analyses remain rare, especially for phonologically complex and low-resource languages. 3 Datasets We work with manually transcribed speech data for two endangered East Caucasian languages from the Lezgic branch: Archi and Rutul. Our contribution lies in curating, consolidating, and standardizing existing speech–transcript resources into a form suitable for ASR training and evaluation. The resulting benchmark consists of: Archi: approximately 45 minutes of training data (545 sentences) and 7 minutes of test data (100 sentences), derived from materials documented in 20 . The recordings contain speech which was read out under controlled conditions. Kina Rutul: approximately 75 minutes of training data (1,388 sentences) and 7 minutes of test data (90 sentences), based on documentation work reported in 1 . The recordings contains spotaneous speech often recorded in noisy environments. Lang. Split Size Vocabulary Sizes Words Phonemes Composites Archi Train 545 / 45m 1445 85 50 (59%) Test 100 / 7m 394 70 37 (53%) Rutul Train 1388 / 75m 4866 78 40 (51%) Test 90 / 7m 441 58 21 (36%) Table 1: Dataset statistics — split-wise number of sentences, total length in minutes (m), number of unique words, phones and composite (complex) phones. Detailed split-wise statistics are provided in Table 1 . Composite phonemes are those with a diacritic such as Q , w , ’ or :. All possible phonemes per language are listed in the Appendix A . The original annotations differ substantially across the two corpora. Kina Rutul recordings are aligned with sentence-level annotations in Praat TextGrid format, while Archi recordings are aligned

Read the complete text