Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation
Jan Christian Blaise Cruz and Alham Fikri Aji · 2025
Read Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation by Jan Christian Blaise Cruz and Alham Fikri Aji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12660.txt.
Introduction
Jan Christian Blaise Cruz and Alham Fikri Aji present a language technology study from 2025 on making smaller models for languages with few resources. Large models that cover many languages are costly to run. The authors test whether simple teaching from a large model to a small one can keep quality while cutting size, using Tagalog as the test case.
Their small single-language models matched strong rival systems on a range of benchmark tasks while running with far less cost. Extra steps that improved how the small model learned the target language also helped. Tests and breakdowns show where the gains come from. Readers get a practical path for serving low-resource languages without huge models.
Catalog record
| CREATOR | Jan Christian Blaise Cruz and Alham Fikri Aji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12660
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation Jan Christian Blaise Cruz and Alham Fikri Aji Abstract In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method. 1 Introduction To curb the detrimental effects of pretraining with very little pretraining data in a low-resource language, most works opt to use pretrained Massively Multilingual Transformers (MMTs) such as mBERT Devlin et al. 2019 and mDeBERTa He et al. 2021b ; He et al. 2021a instead. However, this comes with a number of tradeoffs. Finetuning in only one language causes negative interference in a model that compresses many languages within a limited parameter budget Berend 2022 ; Lee and Hwang 2023 . This would mean that an MMT, in theory, would perform worse than using a transformer pretrained in one specific language Cruz and Cheng 2022 ; Pfeiffer et al. 2022 . Additionally, MMTs are unnecessarily costly as most researchers who use them are only interested in one language among many – this is most especially the case in low-resource language research communities that also suffer from a lack of computational resources Alabi et al. 2022 ; Ansell et al. 2023 . In this work, we propose the use of simple knowledge distillation to extract robust and efficient single-language pretrained transformers from an MMT. We study a number of intermediate steps that improve the distillation method, such as target-language conditioning and student initialization. We then compare the performance of our extracted models on strong baselines on a variety of benchmark tasks and perform ablations and analyses to pinpoint the sources of strong performance from our simple method. 2 Methodology 2.1 Distillation To simplify the study, we limit ourselves to one type of MMT – mBERT ( bert-base-multilingual-cased ) Devlin et al. 2019 – and one language (Tagalog) for both distillation and task finetuning. In the interest of resource-scarce research settings, the proposed method is very simple and computationally cheap: we take a pretrained mBERT and freeze its weights. We then construct a blank student transformer with a modified architecture and use teacher-student model distillation Hinton et al. 2015 using masked language modeling (MLM) as the main objective. No further tricks, post-processing, or augmentations are done after distillation. We use OSCAR’s Tagalog split Ortiz Suárez et al. 2019 as the training corpus for knowledge distillation. Mathematically, we optimize our distillation loss as a mix of the weighted sum of the Kullback-Leibler (KL) divergence and the MLM loss between the student and teacher’s output logits: \begin{split}L_{\textnormal{distil}}&=\alpha_{\textnormal{KL}}\textnormal{KL}(out_{student}||out_{teacher})+\\ &=\alpha_{\textnormal{MLM}}L_{\textnormal{MLM}}(out_{student},out_{teacher})\end{split} (1) where \alpha_{kl} and \alpha_{mlm} represent the weights of the divergence and the MLM loss respectively to the final distillation loss. For our experiments, we use cross entropy as our MLM loss. Note that we also apply a temperature parameter to cool down the logits of the student and teacher and encourage diversity in outputs. This gives us a distilled version of the pretrained mBERT but without the risk of negative interference caused by parameter sharing between multiple languages in the model during downstream finetuning. We produce two distilled models this way which we refer to as dBERT Base and dBERT Tiny , depending on the hyperparameters used. Hyperparameter choices used for distillation are listed on Table 1 . We run distillation for a total of three epochs on the training dataset. Teacher Base Tiny Hidden Dim 768 768 312 Intermediate Size 3072 3072 1200 Layers 12 6 4 Attention Heads 12 12 12 Max Positions 512 512 512 Table 1: Student vs Teacher hyperparameters. We reduce the hidden dimensionality, feedforward intermediate size, and the number of layers. The number of attention heads and max number of positions (tokens) are kept the same. 2.2 Downstream Finetuning To measure the performance of the distilled model on downstream tasks, we finetune on several benchmarks in Tagalog: • TLUnified NER Miranda 2023 – NER classification dataset developed using the TLUnified Cruz and Cheng 2022 corpus. • Hatespeech Filipino Cabasag et al. 2019 – a text classification dataset on hatespeech mined from election tweets in Tagalog. • NewsPH NLI Cruz et al. 2021 – an entailment dataset created using news articles in Tagalog. We measure accuracy for the hate speech classification and NLI tasks and measure F1 for the NER task. We compare the performance of our models with mBERT (as the teacher), Tagalog-RoBERTa Cruz and Cheng 2022 (to compare against a full model trained on Tagalog), DistilmBERT Sanh et al. 2020 (a full distilled version of mBERT retaining all the languages supported), and from-scratch training (where a blank model is directly tuned on the downstream task). TLUnified NER Hatespeech NewsPH NLI Avg. F1 Runtime Accuracy Runtime Accuracy Runtime Speedup From Scratch 0.4818 71s 0.7382 617s 0.5392 25819s Tagalog RoBERTa 0.8939 66s 0.7767 606s 0.9406 25798 mBERT 0.8925 70s 0.7543 618s 0.9318 25811s DistilmBERT 0.8818 44s 0.7372 366s 0.9172 15316s 1.68x dBERT Base (Ours) 0.8074 44s 0.7729 309s 0.9188 13006s 1.97x dBERT Tiny (Ours) 0.6085 31s 0.7261 107s 0.8328 4917s 5.23x Table 2: Main Results. Accuracy refers to evaluation accuracy on the test set. Runtime refers to the total amount of time (in seconds) that it takes to finetune on the task dataset (rounded down). Avg. Speed refers to the factor by which the distilled models are faster compared to mBERT (averaged across the three tasks). 3 Results and Discussion A summary of the results can be found on Table 2 . We can see that our models perform strongly across the three benchmark tasks. For the hate speech classification and NLI tasks, our dBERT Base model outperforms its teacher mBERT as well as the distilled DistilmBERT version with an almost 2x speedup in terms of training time. This shows that the method, albeit simple, works well to produce general-use transformers for these tasks. Performance lags slightly behind on NER, which we assume is a harder task for an extracted model as there are a lot of named entities in the vocabulary from other languages that are not completely removed and present a significant amount of negative interference. We investigate these behaviors further in ablations. The dBERT Tiny variant showed strong results that came close to the baselines on hate speech classification but lags behind the other models in all other tasks. We hypothesize that this is due to the size of the model not having enough capacity to fully capture the teacher’s representation of the target language given that the source representation space is extremely large due to the presence of other languages. Unsurprisingly, RoBERTa Tagalog performs the best in all three tasks given that it is a full-sized BERT-type model that is trained solely in Tagalog. The mBERT and DistilmBERT models are likewise strong performers but are much slower