AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages

Joshua Sakthivel Raju, Sanjay S, Jaskaran Singh Walia, Srinivas Raghav, Vukosi Marivate · 2025

Read AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages by Joshua Sakthivel Raju, Sanjay S, Jaskaran Singh Walia, Srinivas Raghav, Vukosi Marivate (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18020.txt.

Introduction

Joshua Sakthivel Raju and four co-authors shrink a multilingual AI model in a 2025 paper. Big language models are hard to run where computing power is scarce. Shrinking them often hurts languages with little training data. The team focused on five African languages.

Their student model learns from a larger teacher in two ways. It copies the teacher's answers and also matches its attention patterns. The result, AfroXLMR-Comet, is over eighty five percent smaller. Languages covered include Swahili, Hausa, Igbo, Yoruba, and Kinyarwanda. The small model kept most of the teacher's accuracy in tests. The authors offer the method as a path to usable AI on modest hardware. That matters for schools and services across Africa.

Catalog record

CREATORJoshua Sakthivel Raju, Sanjay S, Jaskaran Singh Walia, Srinivas Raghav, Vukosi Marivate
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages Joshua Sakthivel Raju, Sanjay S, Jaskaran Singh Walia, Srinivas Raghav, Vukosi Marivate Abstract Language model compression through knowledge distillation has emerged as a promising approach for deploying large language models in resource-constrained environments. However, existing methods often struggle to maintain performance when distilling multilingual models, especially for low-resource languages. In this paper, we present a novel hybrid distillation approach that combines traditional knowledge distillation with a simplified attention matching mechanism, specifically designed for multilingual contexts. Our method introduces an extremely compact student model architecture, significantly smaller than conventional multilingual models. We evaluate our approach on five African languages: Kinyarwanda, Swahili, Hausa, Igbo, and Yoruba. The distilled student model– AfroXLMR-Comet –successfully captures both the output distribution and internal attention patterns of a larger teacher model (AfroXLMR-Large) while reducing the model size by over 85%. Experimental results demonstrate that our hybrid approach achieves competitive performance compared to the teacher model, maintaining an accuracy within 85% of the original model’s performance while requiring substantially fewer computational resources. Our work provides a practical framework for deploying efficient multilingual models in resource-constrained environments, particularly benefiting applications involving African languages. 1 Introduction Large language models (LLMs) have become a pillar of modern Natural Language Processing (NLP), achieving state-of-the-art results across various tasks ( Devlin 2018 ; Liu et al. 2019 ; Lewis 2019 ) . Their performance only continues to improve with the expansion of computational power, the availability of vast datasets, and the scaling of model architectures ( Kaplan et al. 2020 ; Brown et al. 2020 ; Touvron et al. 2023 ) . But, despite their success, a significant portion of the world’s languages, particularly low-resource languages (LRLs), remain underrepresented in NLP research. These languages, numbering in the thousands, lack the necessary linguistic resources and data required for traditional statistical methods to be effectively applied ( Singh 2008 ; Cieri et al. 2016 ; Tsvetkov 2017 ) . The challenges associated with LRLs are substantial and multifaceted. These languages often suffer from a lack of computational tools, limited digital presence, and insufficient educational infrastructure, which hinders their incorporation into modern NLP systems. While advancements in NLP for LRLs present immense potential, especially in regions such as Africa and India, where over 2.5 billion people speak these languages, the barriers are high. Addressing these challenges not only promises economic and cultural benefits but also plays a vital role in preserving linguistic heritage and improving societal outcomes, such as aiding in emergency response and enhancing educational and cultural exchanges ( Tsvetkov 2017 ) . 1.1 Knowledge Distillation Knowledge distillation (KD) refers to the process where a smaller model learns from a larger one, transferring knowledge from a teacher model to a student model to enhance the student’s performance. The central concept is that the student model emulates the teacher model, using the teacher’s insights to achieve competitive or even superior results. A typical KD framework includes three main components: knowledge, a distillation algorithm, and the teacher-student model architecture. The key challenge is to effectively convey knowledge from the large teacher model to the smaller student model while ensuring the student retains or improves its performance ( Buciluǎ et al. 2006 ; Ba and Caruana 2014 ; Hinton 2015 ; Urban et al. 2017 ) . Response-based knowledge distillation focuses on aligning the final output (logits) of the student model with the teacher’s logits, typically using soft targets and temperature scaling to capture the "dark knowledge" from the teacher. This approach has been widely applied in tasks like image classification, but it is limited by only utilizing the output of the last layer, potentially missing intermediate-level supervision that could benefit deeper networks ( Hinton 2015 ; Ba and Caruana 2014 ) . To address this, feature-based knowledge distillation extends the method by transferring feature maps from intermediate layers of the teacher model to the student model, enhancing representation learning. Techniques like attention maps and activation boundaries are used to match features between layers, but challenges persist in selecting the right layers and handling size differences between layers ( Romero et al. 2015 ; Zagoruyko and Komodakis 2017 ) . In this work, we propose a novel hybrid distillation framework that integrates knowledge distillation and attention matching to improve multilingual model compression. Existing approaches often focus on either response-based distillation, which transfers knowledge through soft output distributions, or feature-based distillation, which aligns internal representations. Our method integrates both, enabling a more comprehensive transfer of knowledge from teacher to student models. Additionally, we introduce a highly compact multilingual student model with a significantly smaller hidden dimension, optimized for low-resource African languages. Our contributions can be summarized as follows: • Hybrid Distillation Framework: We propose a novel distillation approach that integrates knowledge distillation with attention matching, enabling the student model to learn both the output distribution and internal attention patterns of the teacher model. • Compact Multilingual Model: We introduce an extremely compact multilingual architecture with a hidden dimension of 256, significantly smaller than existing models (typically 768 or higher), while maintaining reasonable performance. • Simplified Attention Matching: Our mean-pooled attention matching mechanism effectively transfers knowledge between teacher and student models while reducing computational overhead compared to complex relation-based methods. • Evaluation on African Languages: We conduct a systematic evaluation of our hybrid distillation approach on five African languages—Kinyarwanda ( rw ), Swahili ( sw ), Hausa ( ha ), Igbo ( ig ), and Yoruba ( yo )—addressing gaps in model compression research for low-resource languages. • Empirical Analysis: We analyze trade-offs between model size, computational efficiency, and performance, demonstrating that effective knowledge transfer is possible despite substantial architectural differences between teacher and student models. 2 Related Work Alabi et al. 2022 proposed AfroXLMR, a multilingual pre-trained language model specifically adapted for African languages through multilingual adaptive fine-tuning (MAFT). Their approach has shown to enhance the performance of pre-trained models, such as XLM-R and AfriBERTa, on a variety of tasks for African languages by fine-tuning on monolingual texts from 17 high-resource African languages, alongside three widely spoken high-resource languages in Africa. One of the key innovations is the reduction of model size by removing tokens for non-African writing scripts from the embedding layer, which decreases the model size by approximately 50%. The resulting model not only performs competitively with language-adaptive fine-tuning (LAFT) on individual languages but also improves the cross-lingual transfer Thangaraj et al. 2024 abilities of models like XLM-R, while requiring significantly less disk space. This makes the AfroXLMR approach more efficient for practical deployment on tasks such as Named Entity Recognition (NER), topic classification, and sentiment analysis, especially in low-resource African languages. In MiniLM ( Wang et al. 2020 ) , deep self-attention distillation is used to compress pre-trained Transformers by transferring knowledge from the teacher model to the student model. MiniLMv2 ( Wang et al. 2021 ) builds on this by introducing multi-head self-attention relation distillation for task-agnostic compression, where attention relations are defined as the scaled dot-product between query, key, and value pairs within the self-attention module. Unlike previous methods, MiniLMv2 allows the student model to have a different number of attention heads than the teacher, which removes the constraint of matching attention head numbers.

Read the complete text