HARNESS: Lightweight Distilled Arabic Speech Foundation Models
2026
Read HARNESS: Lightweight Distilled Arabic Speech Foundation Models (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14186.txt.
Introduction
Sukhadia and Chowdhury present HARNESS, a family of Arabic-focused speech models trained from scratch with iterative self-distillation. A large bilingual Arabic-English teacher passes knowledge down to compact student models that keep Arabic sound and expressive detail. The students trade accuracy for efficiency on speech recognition, dialect identification, and emotion recognition, beating HuBERT and XLS-R on Arabic tasks while staying strong under heavy structural cuts.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28114 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14186
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HARNESS: Lightweight Distilled Arabic Speech Foundation Models Abstract Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweight student variants that offer strong accuracy-efficiency trade-offs on Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER). Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models while preserving Arabic-relevant acoustic and paralinguistic representations. We further study PCA-based compression of the teacher supervision signal to better match the capacity of shallow and thin students. Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks, while the compressed models remain competitive under substantial structural reduction. These results position HArnESS as a practical and accessible Arabic-centric SSL foundation for real-world speech applications. Keywords: Self-supervised model, Distillation, Benchmark resources, Arabic downstream tasks HARNESS: Lightweight Distilled Arabic Speech Foundation Models Vrunda N. Sukhadia 1∗ , Shammur Absar Chowdhury 2 1 Amazon India, 2 Qatar Computing Research Institute, HBKU, Qatar [email protected], [email protected] Abstract content * * footnotetext: This work was carried out at QCRI. 1. Introduction Self-supervised learning (SSL) has transformed speech processing by learning transferable representations from large amounts of unlabeled audio. Large SSL models capture rich acoustic and linguistic structure and have shown strong performance across a wide range of speech tasks 5 ; 4 ; 8 ; 18 ; 12 ; 26 . These models can be used either as fixed feature extractors or fine-tuned with limited labeled data, making them especially attractive in low-resource settings. The effectiveness of SSL models, however, depends heavily on the scale, diversity, and balance of the pretraining data. Multilingual SSL models such as XLS-R 7 have shown clear advantages for low-resource languages compared with monolingual models trained on high-resource languages such as English 23 . At the same time, recent evidence suggests that multilingual models may disproportionately favor languages with greater pretraining coverage, which can limit gains for underrepresented languages 24 . This motivates a closer study of language-focused SSL models that better reflect the linguistic and acoustic properties of a target language. Arabic is a particularly challenging case for speech modeling. It is spoken across 22 countries and exhibits substantial dialectal diversity, with wide varieties differing in phonetics, morphology, and lexical usage. In addition, Arabic speech often includes influences from other languages, including English and French 3 . This diversity makes Arabic speech processing difficult for generic multilingual models, which may not fully capture dialect-sensitive and culturally grounded speech patterns. These challenges motivate Arabic-centric SSL modeling that can better represent spoken Arabic while remaining robust to variation across regions and speaking styles. At the same time, training and deploying language-focused SSL models remains expensive. Large-scale pretraining requires substantial compute, long training times, and broad unlabeled speech collections. These costs also make deployment difficult in practical and resource-constrained environments, where model size, memory use, and latency matter. Model compression is therefore essential for making such systems more accessible and usable. Knowledge distillation has emerged as an effective approach for compressing large speech models while preserving much of their performance. In this setting, a smaller student model learns from a larger teacher model, leading to lower memory usage and faster inference with limited degradation in downstream quality. Prior work, including DistillHuBERT 9 , FitHuBERT 16 , DPHuBERT 22 , SKILL 27 , and related methods 6 ; 10 , has explored task-agnostic distillation for HuBERT-style models. However, such work has focused primarily on general-purpose compression, with limited attention to Arabic-centric SSL trained from scratch and systematically distilled into lightweight models. While prior studies have applied self-supervised speech models such as HuBERT and wav2vec 2.0 to Arabic, Arabic-centric SSL remains under-explored, especially in the setting of large-scale training from scratch and deployment-oriented compression. In this work, we focus on both aspects. We introduce H uBERT-based Ar abic a n d E nglish S elf- S upervised Speech ( HArnESS ), an Arabic-centric SSL model family trained from scratch on large-scale bilingual Arabic-English speech, and we study iterative self-distillation to build compact student models that retain strong performance on ASR, SER, and DID. We adopt bilingual Arabic-English pretraining for two reasons. First, English corpora provide additional acoustic and phonetic diversity at scale, which can stabilize representation learning when Arabic resources are comparatively limited and heterogeneous. Second, Arabic speech in real-world settings often includes borrowed English words and code-switching, particularly in conversational and media domains. Our goal is therefore not to weaken Arabic-centric modeling, but to combine Arabic-focused coverage with the regularization benefits of broader bilingual pretraining. Following the HuBERT training paradigm, we first train a large teacher model, HArnESS-L, with 24 encoder layers through iterative self-distillation. We then transfer its knowledge to smaller students, yielding HArnESS-S, a shallow variant, and HArnESS-ST, a shallow (S) and thin (T) variant. In addition, we investigate low-rank approximation of the teacher supervision signal to simplify the distillation target space and improve knowledge transfer to compact students. We evaluate HArnESS-L, HArnESS-S, and HArnESS-ST on three downstream tasks spanning content, dialectal, and paralinguistic information, namely ASR, DID, and SER. We compare them against HuBERT-Large, trained primarily on English, and XLS-R, a multilingual SSL model 7 . Our results show that Arabic-centric pretraining combined with iterative distillation provides an effective balance between task performance and model efficiency. Figure 1 : Overview of the iterative self-distillation framework used to build the HArnESS model family. Our main contributions are as follows: 1. We introduce HArnESS, an Arabic-centric SSL model family trained from scratch, consisting of HArnESS-L (large), HArnESS-S (shallow), and HArnESS-ST (shallow and thin). 2. We study iterative self-distillation as a strategy for compressing Arabic-centric SSL models into lightweight deployment-oriented students. 3. We investigate compact supervision through low-rank approximation of the teacher signal and analyze its effect on student performance. 4. We benchmark the HArnESS family on ASR, DID, and SER, covering content, speaker-related, and paralinguistic speech tasks. 5. We publicly release the distilled models and benchmark resources to support future research. 1 1 1 https://huggingface.co/QCRI/distillHarness 2. HArnESS Models 2.1. HArnESS Model Figure 1 illustrates the HArnESS training pipeline, which follows a HuBERT-style iterative self-distillation procedure. At iteration i , we train a model \mathbf{M}_{i} using discrete pseudo-labels produced from the previous iteration model \mathbf{M}_{i-1} . The core idea is masked-prediction pretraining: a subset of time frames is masked, and the model is optimized to predict the corresponding pseudo-labels. Iterative self-distillation pipeline. Given an input utterance x , we first obtain frame-level embeddings from \mathbf{M}_{i-1} and convert them into discrete targets via clustering (described below), yielding a pseudo-label sequence z^{(i-1)}=\{z^{(i-1)}_{t}\}_{t=1}^{T} with z^{(i-1)}_{t}\in\{1,\dots,K\} . We then train \mathbf{M}_{i} to predict these targets from contextualized frame representations, where some frames are replaced by a mask token (or masked spans), forcing the model to use broader context. Training regimen and compression schedule. We perform multiple iterations of refinement. In the first two iterations, we keep the model architecture unchanged to encourage progressively stronger acoustic abstractions. Starting from the third iteration, we distill and compress the model to obtain efficient variants. Specifically, we explore three compression axes: (a) reducing Transformer depth ( d ) to