Whispering in Amharic: Fine-tuning Whisper for Low-resource Language

Dawit Ketema Gete, Bedru Yimam Ahmed, Tadesse Destaw Belay, Yohannes Ayana Ejigu, Sukairaj Hafiz Imam, Alemu Belay Tessema, Mohammed Oumer Adem, Tadesse Amare Belay, Robert Geislinger, Umma Aliyu Musa, Martin Semmann, Shamsuddeen Hassan Muhammad, Henning Schreiber, Seid Muhie Yimam · 2025

Read Whispering in Amharic: Fine-tuning Whisper for Low-resource Language by Dawit Ketema Gete, Bedru Yimam Ahmed, Tadesse Destaw Belay, Yohannes Ayana Ejigu, Sukairaj Hafiz Imam, Alemu Belay Tessema, Mohammed Oumer Adem, Tadesse Amare Belay, Robert Geislinger, Umma Aliyu Musa, Martin Semmann, Shamsuddeen Hassan Muhammad, Henning Schreiber, Seid Muhie Yimam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18485.txt.

Introduction

Dawit Ketema Gete and a large team tune a speech recognition model for Amharic in this 2025 paper. The base model hears this Ethiopian language poorly because its training barely included it. The team fine-tuned it on public speech collections plus a new local dataset.

The best model trained on a blend: new Amharic data alone taught too little, but mixing it with an existing multilingual set let the model specialize while keeping general skill. Normalizing Amharic words that sound alike but differ in spelling lifted scores further on two standard metrics. The authors distill lessons on data mixing for other low-resource languages whose speakers automated transcription still underserves.

Catalog record

CREATORDawit Ketema Gete, Bedru Yimam Ahmed, Tadesse Destaw Belay, Yohannes Ayana Ejigu, Sukairaj Hafiz Imam, Alemu Belay Tessema, Mohammed Oumer Adem, Tadesse Amare Belay, Robert Geislinger, Umma Aliyu Musa, Martin Semmann, Shamsuddeen Hassan Muhammad, Henning Schreiber, Seid Muhie Yimam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Whispering in Amharic: Fine-tuning Whisper for Low-resource Language Dawit Ketema Gete, Bedru Yimam Ahmed, Tadesse Destaw Belay, Yohannes Ayana Ejigu, Sukairaj Hafiz Imam, Alemu Belay Tessema, Mohammed Oumer Adem, Tadesse Amare Belay, Robert Geislinger, Umma Aliyu Musa, Martin Semmann, Shamsuddeen Hassan Muhammad, Henning Schreiber, Seid Muhie Yimam Abstract This work explores fine-tuning OpenAI’s Whisper automatic speech recognition (ASR) model for Amharic, a low-resource language, to improve transcription accuracy. While the foundational Whisper model struggles with Amharic due to limited representation in its training data, we fine-tune it using datasets like Mozilla Common Voice, FLEURS, and the BDU-speech dataset. The best-performing model, Whisper-small-am, significantly improves when fine-tuned on a mix of existing FLEURS data and new, unseen Amharic datasets. Training solely on new data leads to poor performance, but combining it with FLEURS data reinforces the model, enabling better specialization in Amharic. We also demonstrate that normalizing Amharic homophones significantly enhances Word Error Rate (WER) and Bilingual Evaluation Understudy (BLEU) scores. This study underscores the importance of fine-tuning strategies and dataset composition for improving ASR in low-resource languages, providing insights for future Amharic speech recognition research. 1 Introduction Speech is one of the most fundamental and natural forms of human communication, enabling the exchange of ideas, emotions, and information across people, cultures, and generations. The transcription of speech into text, a process known as speech-to-text (STT) , has evolved significantly over time. Historically, this task was performed by humans, such as stenographers or transcribers, who manually convert the spoken language into written form. However, computational technologies, which are machine-based STT systems, have emerged and are revolutionizing the field. Early attempts at Automated Speech Recognition (ASR) in the mid-20th century had relied on rule-based systems and limited vocabularies ( Jurafsky and Martin 2000 ) . Over decades, advancement in machine learning, particularly deep learning, has enabled the development of more accurate and robust STT systems than rule-based systems, capable of handling diverse languages, accents, and domains ( Hinton et al. 2012 ) . ASR is a machine learning technology designed to convert spoken language into written text, facilitating seamless communication between humans and machines Saksamudre et al. 2015 ; Kheddar et al. 2024 . This technology has become a cornerstone of modern voice-driven systems, enabling a wide range of applications across various sectors. In healthcare, ASR is used for transcribing medical records and assisting in diagnostics, while in marketing, it powers voice-activated customer service tools and personalized advertising. In education, ASR supports language learning and accessibility tools for students with disabilities. Additionally, ASR plays a critical role in cultural preservation by transcribing and archiving oral histories and in military applications for real-time communication and command systems Kumar 2024 ; Yin et al. 2024 . The versatility of ASR underscores its importance in bridging the gap between spoken language and digital technology, making it an indispensable tool in today’s increasingly voice-driven world. Recent developments in multilingual ASR systems such as Whisper - an ASR system developed by OpenAI 1 1 1 https://openai.com/index/whisper/ trained on 680K hours of multilingual and multitask supervised data collected from the web, present opportunities to enhance transcription capabilities for low resource languages Radford et al. 2023 . However, these models often face challenges when dealing with low-resource languages such as Amharic Polat et al. 2024 ; Yu et al. 2021 . Various strategies have been proposed to optimize Whisper’s performance Kummervold et al. 2024 ; Timmel et al. 2024 ; Li et al. 2024a , achieving notable improvements in transcription accuracy and adaptability for low-resource languages other than Amharic. Based on Whisper’s improvement approaches for other low-resource languages, our study focuses on fine-tuning the Whisper model specifically for Amharic ASR and explores the capability, fine-tuning strategy, and evaluation mechanisms that lead to effective results. Our approach utilizes a comprehensive and diverse dataset, incorporating publicly available speech corpora such as Mozilla Common Voice Foundation 2024 , Google/FLEURS Goyal et al. 2021 , and the BDU Speech Corpus Assfaw et al. 2022 . The main contributions of this work are three folds: 1) we extensively fine-tune and evaluate various versions of models fine-tuned on the Whisper small ASR model, and 2) we investigated the dynamic of fine-tuning with mixed datasets to reinforce the model’s specialization on Amharic. 3) We investigated the impact of homophone normalization in the evaluation of the whisper-small model, which can be a good point to consider while evaluating other models and also for fine-tuning as a further exploration. 2 Related works In recent years, the rise of multi-modal large language models (MM LLMs) has further expanded the capabilities of artificial intelligence to understand human language. These models, such as OpenAI’s GPT-4 and Google’s Gemini, integrate text, audio, and visual data to perform complex tasks, ranging from text to audiovisual understanding Caffagni et al. 2024 . In the era of speech-related LLMs, Whisper by OpenAI has set a new benchmark in STT and TTS tasks as an SOTA model trained on a massive dataset of multilingual and multitask supervised data ( Radford et al. 2023 ) . The multilingual nature of the whisper model creates the ability to generalize across low-resource languages, which makes it valuable for underrepresented linguistic communities and a best candidate to adopt its capability by fine-tuning with a more refined language-specific dataset. STT in Low-resource languages Amharic is one of the languages with progress made to provide techno-linguistic tools, datasets, and research for downstream NLP tasks Tonja et al. 2023 . However, there is an insufficient research focus in speech-related tasks like TTS and STT along with resources, which hinders the development of accurate and reliable ASR systems. The lack of these datasets and tools limits the access to technology-driven opportunities. In the context of LLMs, low-resource languages are often underrepresented in training datasets, leading to suboptimal performance and biased outcomes. This underscores the need for fine-tuning pre-trained LLMs like Whisper to adapt them to the unique phonetic, syntactic, orthographic, and semantic characteristics of the languages ( Zhong et al. 2024 ; Hangya et al. 2022 ) . Whisper and Low-Resource Languages OpenAI’s Whisper, a Transformer-based multilingual ASR model trained on 680k hours of diverse audio data Radford et al. 2023 , has demonstrated state-of-the-art zero-shot performance across numerous languages. However, its efficacy diminishes for low-resource languages like Amharic, which are underrepresented in its training corpus. Recent studies highlight that fine-tuning Whisper on language-specific data mitigates this limitation. For instance, Polat et al. 2024 applied Low-Rank Adaptation (LoRA) to optimize Whisper for Turkish, achieving parameter-efficient adaptation with minimal computational overhead. Similarly, Li et al. 2024b reduced the Word Error Rate (WER) for Kazakh by over 10% through dynamic data augmentation and model quantization. Singh and Bhatt 2024 improved Hindi ASR performance by integrating transfer learning with Whisper’s pre-trained encoder-decoder architecture. These efforts underscore the adaptability of Whisper to linguistically diverse, low-resource settings. Advancements in research on low resource languages have focused on addressing data scarcity and linguistic complexity. Ejigu and Asfaw 2024 curated a foundational ASR dataset of 128 hours of Amharic speech, enabling targeted fine-tuning and data augmentation. Building on this, multilingual acoustic modeling approaches leveraging phonetically related languages reduced WER from 23.23% to 21.52% Teferra et al. 2024 . Furthermore, Adnew and

Read the complete text