AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text

Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam · 2025

Read AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text by Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18247.txt.

Introduction

Tadesse Destaw Belay and co-authors adapted a multilingual language model to African social media, a domain far from the religious and formal texts that dominate training data for many African languages. They built AfriSocial, a large corpus of social media and news text across the languages. They then continued pre-training the AfroXLMR model on it in two ways: adapting to the domain generally, and adapting to one task's unlabeled data before training another.

Domain adaptation improved scores by one to thirty points across sentiment, emotion, and hate speech tasks in nineteen languages. Task adaptation also transferred, with sentiment data lifting fine-grained emotion classification by up to fifteen points. Combining both gave the best results. The team releases the corpus and models openly.

Catalog record

CREATORTadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18247

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam Abstract Language models built from various sources are the foundation of today’s NLP progress. However, for many low-resource languages, the diversity of domains is often limited, more biased to a religious domain, which impacts their performance when evaluated on distant and rapidly evolving domains such as social media. Domain adaptive pre-training (DAPT) and task-adaptive pre-training (TAPT) are popular techniques to reduce this bias through continual pre-training for BERT-based models, but they have not been explored for African multilingual encoders. In this paper, we explore DAPT and TAPT continual pre-training approaches for African languages social media domain. We introduce AfriSocial , a large-scale social media and news domain corpus for continual pre-training on several African languages. Leveraging AfriSocial, we show that DAPT consistently improves performance (from 1% to 30% F1 score) on three subjective tasks: sentiment analysis, multi-label emotion, and hate speech classification, covering 19 languages. Similarly, leveraging TAPT on the data from one task enhances performance on other related tasks. For example, training with unlabeled sentiment data (source) for a fine-grained emotion classification task (target) improves the baseline results by an F1 score ranging from 0.55% to 15.11%. Combining these two methods (i.e. DAPT + TAPT) further improves the overall performance. The data and model resources are available at HuggingFace 1 1 1 https://huggingface.co/tadesse/AfroXLMR-Social . 1 Introduction Pre-trained language models (PLMs) are initially trained on vast and diverse corpora, including encyclopedias and web content Conneau et al. (2020) ; Chiang et al. (2022) . Subsequently, these pre-trained models are used in supervised training for a specific Natural Language Processing (NLP) task by further finetuning. Fine-tuned PLMs achieve strong performance across many tasks and datasets from various sources Shi et al. (2025) . However, this raises a question: Do PLMs function universally across domains, or does continual training of PLMs with domain-specific data offer better performance? Figure 1: Continual pre-training illustration. A general-purpose pretrained model, such as AfroXLMR, is first adapted to a social domain, resulting in AfroXLMR-Social . This model then undergoes Cross-task TAPT using sentiment analysis, emotion, and hate speech data without the labels for further fine-tuning. While some studies have shown the benefit of continual pre-training on a domain-specific unlabeled data Lee et al. (2019) ; Gururangan et al. (2020) , one domain may not be generalizable to other domains and languages. Moreover, it is unknown how the benefit of continual pre-training may vary with factors like the amount of unlabeled corpus, the source domain itself, the evaluation task, the resource richness of the target languages, and the trained target model Gururangan et al. (2020) . This raises the question of whether pre-training on a corpus more directly tied to the task can further improve performance. This work addresses these questions by continual domain adaptive pre-training (DAPT) and task adaptive pre-training (TAPT) on the downstream subjective NLP tasks in a low-resource language setup. We consider the social media domain (X) and News for a continual pre-training from a high-performing multilingual baseline model, AfroXLMR Alabi et al. (2022) . AfroXLMR is a continually pre-trained model for African languages based on XLM-RoBERTa Conneau et al. (2020) . We explore subjective NLP task results from baseline (the base model result), DAPT, TAPT, and DAPT + TAPT, on a smaller but directly domain- and task-relevant unlabeled corpus. The results show that DAPT and TAPT highly benefit from similar source NLP tasks. Figure 1 illustrates the general high-level continual pre-training strategies. In summary, our contributions are: • We present AfriSocial , a new quality domain-specific corpus for 14 African languages, collected from the social domain (X) and news. • We perform a through analysis of domain and task adaptive continual pre-training across subjective NLP tasks for low-resource languages. • We achieve state-of-the-art results in the evaluated NLP tasks and publicly making AfriSocial social-domain corpus and AfroXLMR-Social pretrained models for developing low-resource languages. Dataset name Task name # lang. Data Sources # Classes AfriSenti Sentiment analysis 14 News, social media 3 AfriEmo Emotion analysis 17 News, social media 6 AfriHate Hate speech detection 15 News, social media 3 AfriSocial (new domain specific corpus) Unlabeled 14 X and news - Table 1: List of subjective NLP task evaluation datasets. Social media sources include posts/comments from YouTube and X. The AfriSenti dataset class labels are positive, negative, and neutral. The AfriEmo dataset labels are six basic emotions (anger, disgust, fear, joy, sadness, and surprise) in a multi-label annotation (an instance may have none, one, two, some, or all targeted emotion labels). AfriHate labels are abuse, hate, and neutral. 2 Related Work Domain in NLP Language models (LMs) pretrained on text from various sources are the foundation of today’s NLP. Domain adaptation in NLP refers to enhancing the performance of a model using similar domain data (target domain) by leveraging knowledge from an existing domain (source domain) Ramponi and Plank (2020) . Domain refers to different implicit clusters of text representations in pretrained LMs, such as news articles, social media posts, medical texts, or legal documents Aharoni and Goldberg (2020) ; Shi et al. (2025) . Each domain has its unique characteristics, vocabulary, and writing style, which can affect the performance of NLP models when applied to new or unseen domains. Therefore, a similar domain means the text source from which the pre-trained model was made, similar to the target NLP task data source. Continual Pre-training of LMs There are several techniques for the downstream NLP task improvements, such as general-purpose pre-training Wang et al. (2023) , language-adaptive continual pre-training Alabi et al. (2022) , domain-adaptive pre-training Gururangan et al. (2020) , task-adaptive pre-training Alabi et al. (2022) , and data augmentation Zhang et al. (2024) . Prior works have shown the benefit of continual pre-training using a domain-specific corpus Gururangan et al. (2020) and training LMs in a specific domain from scratch Huang et al. (2020) . However, continual pre-training is arguably more cost-effective than training from scratch, since it is a continuous pre-training from the existing base language model. Domain Adaptive Pre-training (DAPT) Domain Adaptive pre-training (DAPT) is straightforward, continuing pre-training a model on a corpus of unlabeled domain-specific text Aharoni and Goldberg (2020) . DAPT techniques handle discrepancies between the source (pre-training) and target (fine-tuning) domains. Traditional fine-tuning often yields suboptimal results when pre-trained models encounter data that diverges significantly from their training data. In this regard, DAPT techniques reduce this mismatch by aligning data distributions, ensuring the model can generalize better in a better setup. Lee et al. (2019) considers a single domain at a time and uses a language model pretrained on a smaller and less diverse corpus than the most varied and multilingual language models. Task Adaptive Pre-training (TAPT) Task-adaptive pre-training (TAPT) refers to pre-training on the unlabeled training set for a similar task-specific data Gu et al. (2024) . Compared to DAPT, TAPT solely leverages the training data of the similar downstream task for continuous pre-training. It uses a far smaller pre-training corpus that is much more task-relevant (assuming that the training set represents aspects of

Read the complete text