Where Are We At with Automatic Speech Recognition for the Bambara Language?
Seydou Diallo, Yacouba Diarra, Mamadou K. KEITA, Panga Azazia Kamaté, Adam Bouno Kampo, Aboubacar Ouattara · 2026
Read Where Are We At with Automatic Speech Recognition for the Bambara Language? by Seydou Diallo, Yacouba Diarra, Mamadou K. KEITA, Panga Azazia Kamaté, Adam Bouno Kampo, Aboubacar Ouattara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09785.txt.
Introduction
Seydou Diallo and five co-authors offer the first standard benchmark for testing auto speech hearing in the Bambara language, using one hour of pro taped Malian charter text. Their 2026 paper designs the set as a set model under near-optimal sound and language settings, then tests thirty-seven models ranging from Bambara-trained systems to large-scale store models.
Current scores remain far below use standards even in this narrow formal domain. The top system misses nearly half of words, while the best letter error rate, set by another model, sits near an eighth. Several known multi-tongue models exceed a hundred percent word error rate. Multi-tongue pre-training and scaling alone prove too little for rare tongues, and these best-case figures stay new against real settings. The benchmark and a board support open future tests.
Catalog record
| CREATOR | Seydou Diallo, Yacouba Diarra, Mamadou K. KEITA, Panga Azazia Kamaté, Adam Bouno Kampo, Aboubacar Ouattara |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Where Are We At with Automatic Speech Recognition for the Bambara Language? Seydou Diallo, Yacouba Diarra, Mamadou K. KEITA, Panga Azazia Kamaté, Adam Bouno Kampo, Aboubacar Ouattara Abstract This paper introduces the first standardized benchmark for evaluating Automatic Speech Recognition (ASR) in the Bambara language, utilizing one hour of professionally recorded Malian constitutional text. Designed as a controlled reference set under near-optimal acoustic and linguistic conditions, the benchmark was used to evaluate 37 models, ranging from Bambara-trained systems to large-scale commercial models. Our findings reveal that current ASR performance remains significantly below deployment standards in a narrow formal domain; the top-performing system in terms of Word Error Rate (WER) achieved 46.76% and the best Character Error Rate (CER) of 13.00% was set by another model, while several prominent multilingual models exceeded 100% WER. These results suggest that multilingual pre-training and model scaling alone are insufficient for underrepresented languages. Furthermore, because this dataset represents a best-case scenario of the most simplified and formal form of spoken Bambara, these figures are yet to be tested against practical, real-world settings. We provide the benchmark and an accompanying public leaderboard to facilitate transparent evaluation and future research in Bambara speech technology. 1 Introduction Figure 1: Models combined performance on Bambara Benchmark. Lower is better. Automatic Speech Recognition (ASR) for Bambara has seen growing interest in the past three years. Since the 2022 release of Jeli-ASR 2 , the first open ASR dataset for the language, numerous models and datasets have emerged from both research labs and community initiatives. However, this rapid growth raises concerns about quality and usability, concerns that cannot be addressed without standardized evaluation. Quality, when it comes to low resource African languages, is the object of strong debates among the African NLP community due to the variety of dialects, writing systems, and standards 4 , but also the complexity of the contact phenomenon between African languages and western languages, namely code switching. As the Word Error Rate (WER) is only relevant when we have already defined and assessed the quality of the evaluation set, Whatever quality means for one, some researchers recommend defaulting to human evaluation by native speakers ( 7 ; 9 ). However, this process is time consuming and expensive, furthermore edit distance metrics like WER or Character Error Rate (CER) remain insightful on a curated and standardized benchmark. However, no such benchmark existed for evaluating Bambara ASR models, most openly released models 1 1 1 hf.co/facebook ; facebookresearch/omnilingual-asr ; hf.co/asr-africa ; hf.co/MALIBA-AI ; hf.co/RobotsMali ; hf.co/djelia report values for WER and CER on internal test sets. To address this issue and offer a reference test set , we publish the first Bambara ASR benchmark and leaderboard backed with experts validated transcriptions. As more data collection initiatives for African languages emerge, often with strict rules to capture simplified language and context, such as no slang, no code-switching, no background noise etc, we have designed this first benchmark to represent an equally "pure" version of the Bambara language. Relatively poor evaluation results of models trained on more modern and accessible Bambara (see section 3 ) raise questions about the representativeness and usability of simplified language for real-world applications where natural data often include noise, informal terms, and code-switching. Therefore, we anticipate that this benchmark will be among the most difficult test sets for current Bambara ASR systems, covering a specialized and highly formal domain, and we argue for its interpretation as a reference test set for pure Bambara . 2 Characteristics of the Benchmark This first version of the evaluation set consists of a 1 hour recording of a professionally translated version of the Malian constitution, translated and recorded by the Direction Nationale de l’Education Non Formelle et des Langues Nationales (DNENF-LN) 2 2 2 DNENF-LN is the government founded organization in charge of literacy training and official documents translation in all the 13 national languages of Mali: https://dnenfln.ml/ under studio conditions, featuring one unique adult male voice. With the premier legal text of Mali as topic, the dataset features a highly formal and diverse vocabulary that unpacks many aspects of the organization of Malian society, laws, institutions, rights and responsibilities, all written in the Bambara latin script using standard orthography and without code switching . The dataset also has an important representation of numbers, as the constitution contains 191 articles as of July 2023, 160 of which are clearly spelled out in the recording, specifically in ordinal forms. We ran manual segmentation and audio-text alignment using the Audacity software 1 . Then we performed a final quality assurance step wherein the aligned utterances were reviewed to correct divergences resulting from the corpus’ read speech (READ) nature, specifically addressing instances where the speaker paraphrased or interpreted the text rather than providing a literal recital. This process resulted in 500 variable-length audio utterances ranging from 600 ms to 46 seconds , with a mean duration of 7.57 seconds . With this variability the benchmark aims to test models’ capabilities on both short and long form transcription. We calculated Signal-to-noise Ratio (SNR) as an estimate of the acoustic purity of the benchmark (a higher value is best). We used the same Voice-activity-detection based implementation and classification thresholds as 3 but we calculated SNR on the segmented utterances, because the original recording features transition music and longer silences that would hinder the accuracy of the estimation as any non-speech segment is considered for estimating the Noise Power in this implementation ( 3 ; 10 ). Note that we still kept 8 of these silent/music segments in the final benchmark to test the robustness of the models, especially the tendency to "hallucinate" tokens as the silence becomes lengthy when there is in fact no speech. Table 1 shows the SNR distribution the 492 remaining speech segments. SNR Category Threshold (dB) Recordings Medium SNR [5,15) 5 High SNR [15,25) 109 Very high SNR \geq 25 378 Total Audios 492 Table 1: Distribution of Audio utterances by Signal-to-noise Ratio Category. We note that 99% of the utterances are classified as relatively noise-free. This is an important point for interpreting our results: this benchmark represents near-optimal acoustic conditions 3 3 3 In future versions, we will collect data in various domains under different recording conditions, trying to maximize diversity and real world representativeness instead of purity . Any production deployment would face significantly more challenging audio quality, so these results should be interpreted with caution given the specialized and nature of the reference test set and its acoustic purity. 3 Leaderboard and Results of Open Bambara ASR Models We evaluated 37 publicly available ASR models on our benchmark, including monolingual ASR models, multilingual models with Bambara support, and large-scale commercial ASR systems. Table 2 presents the complete leaderboard ranked by a weighted average score of WER and CER (50% WER + 50% CER). This equal weighting reflects a neutral stance that does not privilege either word-level or character-level accuracy, treating both as equally informative for assessing transcription quality. We acknowledge that optimal weighting may depend on downstream application requirements for instance, applications sensitive to semantic accuracy may prioritize WER, while those tolerant of word boundary errors may favor CER. To address this, our public leaderboard allows users