Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data

Hillary Mutisya, John Mugane · 2026

Read Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data by Hillary Mutisya, John Mugane (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11378.txt.

Introduction

Hillary Mutisya and John Mugane reach state-of-the-art Swahili speech recognition with remarkably little labeled data. They continue to pretrain a speech model before fine-tuning it. Their recipe runs pseudo-labeled continued pretraining on unlabeled audio. It then runs supervised fine-tuning on twenty thousand labeled samples. The result cuts word error rate by eighty-two percent relative to the baseline. It beats the best previously reported system by sixty-one percent. This is a transferable methodology for other low-resource languages.

Catalog record

CREATORHillary Mutisya, John Mugane
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19587
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data Hillary Mutisya, John Mugane Abstract We investigate continued pretraining (CPT) for adapting wav2vec2-bert-2.0 to Swahili automatic speech recognition (ASR). Our approach combines unlabeled audio with limited labeled data through pseudo-labeled CPT followed by supervised finetuning. With 20,000 labeled samples, we achieve 3.24% WER on Common Voice Swahili—an 82% relative improvement over the baseline. This result surpasses the best previously reported academic system (8.3% WER from XLS-R) by 61% relative improvement. We provide concrete data requirements and a replicable methodology applicable to other low-resource languages. Keywords: Automatic Speech Recognition, Low-Resource Languages, Continued Pretraining, Swahili, African Languages 1 Introduction 1.1 The Data Challenge Swahili, despite being one of Africa’s most widely spoken languages with over 100 million speakers, faces a fundamental challenge shared by virtually all African languages: the scarcity of high-quality labeled speech data. While high-resource languages like English benefit from tens of thousands of hours of professionally transcribed speech, Swahili and other African languages must achieve competitive ASR with substantially less labeled data. This data scarcity creates a critical barrier to developing speech technology that serves African language communities. Self-supervised foundation models like wav2vec 2.0 and wav2vec2-bert-2.0 have transformed this landscape by enabling learning from unlabeled audio. However, an important question remains: how can we most effectively combine readily available unlabeled audio with limited labeled data to achieve high-quality ASR for low-resource languages? 1.2 The CPT Hypothesis Continued pretraining (CPT) offers a potential solution by adapting foundation models to target languages using unlabeled audio before supervised finetuning on labeled data. The hypothesis is straightforward: if we can leverage abundant unlabeled audio, we might achieve strong performance with far less labeled data than traditional supervised approaches require. However, recent work shows mixed results for CPT, particularly for low-resource scenarios. Some studies report substantial improvements, while others observe performance degradation or minimal benefits. For Swahili specifically—a language with moderate resources and some representation in foundation model pretraining—the value proposition of CPT remains empirically unvalidated. 1.3 Research Questions This study addresses three fundamental questions: Q1: Does continued pretraining on pseudo-labeled unlabeled data improve Swahili ASR performance when combined with limited labeled data? We systematically compare models trained with and without CPT to isolate its contribution. Q2: How does the amount of labeled data interact with continued pretraining effectiveness? We test both minimal (5K samples) and modest (20K samples) labeled data scales to understand whether CPT benefits vary with supervision level. Q3: What level of performance can be achieved by optimally combining unlabeled and labeled data? We establish concrete WER benchmarks that demonstrate the feasibility of high-quality Swahili ASR with limited resources. 1.4 Key Contributions Our research makes the following contributions: 1. First systematic evaluation of pseudo-labeled CPT for Swahili : We demonstrate that continued pretraining on unlabeled audio labeled with a baseline model, followed by supervised finetuning, achieves state-of-the-art results. 2. New state-of-the-art for Swahili ASR : We achieve 3.24% WER on Common Voice Swahili with just 20K labeled samples—a 61% relative improvement over the best published academic baseline (8.3% from XLS-R [ 3 ] ). 3. Concrete data requirements for deployment : We establish that \sim 20K labeled samples ( \sim 11 hours) combined with unlabeled audio suffices for high-quality Swahili ASR suitable for many practical applications. 4. Practical methodology for low-resource languages : We provide a replicable training pipeline that combines readily-available unlabeled audio with modest labeled datasets, offering a path forward for other underserved languages. 1.5 Organization The remainder of this paper is organized as follows: Section 2 reviews related work on continued pretraining and Swahili ASR. Section 3 presents our experimental methodology including the pseudo-labeling approach. Section 4 reports results for Swahili across both data scales. Section 5 discusses implications and provides practical guidelines. Section 6 concludes with future directions. 2 Related Work 2.1 Self-Supervised Models The development of self-supervised learning for speech has revolutionized ASR for low-resource languages. Wav2vec 2.0 [ 4 ] pioneered contrastive learning on unlabeled audio, learning representations by distinguishing masked speech frames from distractors. XLS-R [ 3 ] scaled this to 128 languages using 436,000 hours of unlabeled speech, though African language representation remained limited to only 3 languages. Wav2vec2-bert-2.0 [ 11 ] represents the current state-of-the-art, combining wav2vec 2.0 and WavLM architectures with training on 4.5 million hours across 104 languages. Crucially for our work, this model includes Swahili in its pretraining data, providing a strong baseline that our continued pretraining approach builds upon. 2.2 Continued Pretraining Continued pretraining extends self-supervised learning to adapt pretrained models for specific domains or languages. For domain adaptation, Hsu et al. [ 6 ] demonstrated that CPT on in-domain unlabeled data improves subsequent finetuning, even when finetuning data comes from a different domain. Attia et al. [ 2 ] achieved 10% WER reduction for classroom ASR through CPT on classroom audio. For language adaptation, results have been more mixed. Nowakowski et al. [ 9 ] found CPT “clearly the most effective way” to adapt XLSR-53 to Ainu (a critically endangered language with < 5 hours labeled data), achieving 40% relative WER reduction. However, other studies report minimal benefits or degradation, particularly when target languages have some foundation model representation or when CPT data quality is poor. Pseudo-labeling for CPT has emerged as a practical approach when purely self-supervised objectives are unavailable or underperform. The baseline foundation model generates transcriptions for unlabeled audio, which are then used as targets for continued pretraining. Success depends critically on baseline model quality—high-quality pseudo-labels enable effective adaptation, while noisy labels can introduce systematic errors. 2.3 Swahili ASR Benchmarks Published Swahili ASR benchmarks establish the performance landscape our work improves upon. Academic systems include XLS-R finetuned models achieving 8.3% WER on Common Voice [ 3 ] , SpeechBrain wav2vec2 models achieving 9-12% WER on Common Voice [ 10 ] , and various open-source implementations reporting 8-10% WER. Our 3.24% WER represents a substantial improvement over these academic benchmarks, demonstrating the effectiveness of continued pretraining for low-resource Swahili ASR. 2.4 Positioning of This Work Our work makes several unique contributions to existing literature: 1. First systematic CPT evaluation for Swahili : While XLS-R and other models finetune on Swahili, no prior work systematically evaluates continued pretraining’s contribution using controlled experiments with pseudo-labeled data. 2. Readily-available unlabeled data : We demonstrate that unlabeled audio, when pseudo-labeled by a competent baseline model, provides effective CPT data—a scalable approach applicable to many languages. 3. Concrete data requirements : Rather than reporting performance at arbitrary data scales, we establish specific labeled data requirements (5K and 20K samples) needed to achieve defined quality levels when combined with CPT. 3 Methodology 3.1 Experimental Design Our experimental design evaluates the effectiveness of continued pretraining at two labeled data scales (5K and 20K samples). CPT Pipeline: 1. Train a labeling model on available labeled data 2. Generate pseudo-labels for unlabeled audio using the labeling model 3. Perform continued pretraining on the pseudo-labeled data 4. Supervised finetune on labeled data Comparison Baseline: To quantify CPT’s contribution, we separately trained a baseline model by finetuning wav2vec2-bert-2.0 directly on 50K labeled samples without any continued pretraining. This baseline achieves 17.71% WER and serves as a reference point for evaluating the CPT approach.

Read the complete text