AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages

Kosei Uemura, Miaoran Zhang, David Ifeoluwa Adelani · 2025

Read AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages by Kosei Uemura, Miaoran Zhang, David Ifeoluwa Adelani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23896.txt.

Introduction

Kosei Uemura and two co-authors give African languages their own corner of the text embedding benchmark world. Embeddings power retrieval and help stop language models from hallucinating, but the multilingual benchmark still underrepresents African languages, and its few tasks are recycled from translation sets. The authors answer with AfriMTEB, covering fifty-nine languages, fourteen tasks, and thirty-eight datasets, including six new ones.

The new datasets span fourteen to fifty-six languages each and add fresh task types: hate speech detection, intent detection, and emotion classification. Alongside the benchmark, the team presents AfriE5, an embedding model adapted from the mE5 model by cross-lingual contrastive distillation toward African languages. In their evaluation, AfriE5 reached state-of-the-art results on the new benchmark, beating rivals including Gemini embeddings and mE5. The work gives builders a fair test and a strong starting model for African language applications.

Catalog record

CREATORKosei Uemura, Miaoran Zhang, David Ifeoluwa Adelani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23896

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages Kosei Uemura, Miaoran Zhang, David Ifeoluwa Adelani Abstract Text embeddings are an essential building component of several NLP tasks such as retrieval-augmented generation which is crucial for preventing hallucinations in LLMs. Despite the recent release of massively multilingual MTEB (MMTEB), African languages remain underrepresented, with existing tasks often repurposed from translation benchmarks such as FLORES clustering or SIB-200. In this paper, we introduce AfriMTEB—a regional expansion of MMTEB covering 59 languages, 14 tasks, and 38 datasets, including six newly added datasets. Unlike many MMTEB datasets that include fewer than five languages, the new additions span 14 to 56 African languages and introduce entirely new tasks, such as hate speech detection, intent detection, and emotion classification, which were not previously covered. Complementing this, we present AfriE5, an adaptation of the instruction-tuned mE5 model to African languages through cross-lingual contrastive distillation. Our evaluation shows that AfriE5 achieves state-of-the-art performance, outperforming strong baselines such as Gemini-Embeddings and mE5. 1 Introduction Text embeddings are core building blocks for NLP systems in information retrieval, clustering, semantic similarity, and classification ( Gao et al. 2022 ; Feng et al. 2022 ) . However, evaluations on diverse tasks are often limited to a few high resource languages such as English ( Muennighoff et al. 2023 ) or Chinese ( Xiao et al. 2024 ) . Many under-represented languages are excluded due to lack of datasets or non-discoverability of community-created benchmarks ( Ojo et al. 2025 ) . Figure 1: AfriMTEB (Full): Model size vs. mean performance. Parameter counts (billions, log scale) are shown on the x -axis, and mean scores across AfriMTEB (Full) tasks on the y -axis. AfriE5-large-instruct (red) achieves the best overall performance (64.6) despite having far fewer parameters than most 7–8B models. Figure 2: Overview of AfriMTEB and AfriMTEB-Lite. The Full suite spans 59 languages and 38 datasets across 7 families; the Lite suite provides uniform coverage for 9 languages and 13 datasets. While recent text embedding benchmarks have improved language coverage in recent years, such as MMTEB ( Enevoldsen et al. 2025 ) , African languages remain under-represented, where many of the tasks covered are either based on massively evaluation of translation datasets ( NLLB-Team et al. 2022 ; Federmann et al. 2022 ) , and the tasks derived or repurposed from translation benchmarks such as Belebele ( Bandarkar et al. 2024 ) and SIB-200 ( Adelani et al. 2024 ) . As a result, the quality of text embeddings for languages in the African region remains unknown, as the region has few standardized tools for comparing models across tasks and languages ( Alabi et al. 2025 ) In this paper, we introduce AfriMTEB , a regional extension of MTEB tailored to African languages and tasks. Figure 2 shows the full suite that covers 59 languages and 38 datasets spanning Bitext mining, Classification (single, pair and multi-label), Semantic Text Similarity, Retrieval, Clustering, and Reranking. To support fast-and-compute friendly evaluation, we also provide AfriMTEB-Lite , a compact suite over nine geographically-diverse languages (Amharic, Oromo, Igbo, Yoruba, Hausa, Swahili, Kinyarwanda, Xhosa, Zulu) constructed by selecting, within each task family, the maximally overlapping tasks such that every task includes all nine languages, ensuring uniform language coverage across the suite. In addition to benchmarking, we develop AfriE5-Large-Instruct adapting strong embedding models to African languages. Starting from the instruction-tuned mE5-Large-Instruct ( Wang et al. 2024a ; Wang et al. 2024c ) , we leveraged cross-lingual contrastive distillation. We construct supervision by translating MNLI/SNLI datasets ( Williams et al. 2018 ; Bowman et al. 2015 ) in African languages with NLLB-200 (3.3B) ( NLLB-Team et al. 2022 ) , followed by automatic filtering by SSA-COMET (an African COMET-based QE metric) ( Li et al. 2025 ) , and expanding each example into multiple sources or target directions to encourage cross-lingual alignment. We then use the receipe of BGE Re-ranker v2 m3 ( Chen et al. 2024 ) for text embedding model training. We evaluate widely used baselines, including BGE-M3, E5 variants (e.g., e5-mistral-7b-instruct), Qwen embeddings, Gemini embedding-001, and Embedding Gemma ( Chen et al. 2024 ; Wang et al. 2024c ; Zhang et al. 2025 ; Lee et al. 2025 ) . On AfriMTEB-Lite , our adapted model AfriE5-Large-Instruct , trained only on nine African languages, achieves an average score of 63.7, surpassing me5-large-instruct (62.0) and Gemini embedding-001 (63.1). Remarkably, despite being tuned with data and languages aligned to the AfriMTEB-Lite , our model leveraged cross-lingual transfer to generalize to the broader AfriMTEB benchmark covering 59 languages and 38 datasets, where it also delivers the best performance with an average score of 62.4, ahead of me5-large-instruct (61.3) and Gemini embedding-001 (60.6) as shown in Figure 1 . These results highlight that targeted cross-lingual adaptation on a carefully selected subset of languages can transfer effectively to a much larger set, yielding consistent improvements across task families while preserving the backbone’s general utility. 2 AfriMTEB Benchmark Task Number of Languages Task Family Datasets (AfriMTEB Full) in MMTEB in AfriMTEB Bitext Mining Btxt Flores, NTREX, BibleNLP, NollySenti, Tatoeba 59 59 NLI Pr Clf XNLI, AfriXNLI 1 15 Topic class. Clf SIB200Classification, SIB200_14Classes 0 56 News Topic class. Clf MasakhaNEWS, TswanaNews, SiswatiNews, SwahiliNews 16 17 IsiZuluNews, KinNews Sentiment Clf NaijaSenti, AfriSenti, MultilingualSentiment 11 12 Hate speech Clf AfriHate 0 14 LID Clf LanguageClassification, SouthAfricanLangClassification 0 18 Clf AfriSentiLangClassification Intent Clf MassiveIntent, InjongoIntent 3 16 Scenario Clf MassiveScenario 0 3 Emotion Multi. Clf EmotionAnalysisPlus 0 14 Semantic relatedness STS SemRel24STS 7 7 Retrieval Rtrvl Belebele, MIRACL, MIRACLRetrievalHardNegatives, 31 31 MrTidy, XQuAD, XM3600T2I Clustering Clust SIB200ClusteringFast, MasakhaNEWSClusteringP2P 58 58 MasakhaNEWSClusteringS2S Reranking Rrnk MIRACLReranking 0 2 Table 1: Tasks, task families and datasets included in AfriMTEB (Full) . The task families are Bitext Mining (Btxt), Pair Classification (Pr Clf), Classification (Clf), Semantic Text Similarity (STS), Multi-label Classification (Multi. Clf), Retrieval (Rtrvl), Clustering (Clust) and Reranking (Rrnk). We introduce additional the five datasets highlighted in bold. AfriMTEB is a regional extension of MTEB designed to evaluate text embeddings for African languages across multiple task families. To address persistent gaps in language coverage and enable systematic, comparable assessment, we introduce AfriMTEB as a standardized benchmark for African languages. It consists of a Full suite covering 59 languages and 38 datasets , and a Lite suite that maintains uniform language coverage for 9 target languages over 13 representative datasets . Languages are selected to span major families and geographic regions while prioritizing practical relevance and the availability of reliable public datasets; tasks reflect core embedding applications and draw on established, well-curated resources. The Lite suite supports rapid iteration by enforcing consistent task coverage across languages through maximal task–language overlap. AfriMTEB follows the MTEB taxonomy and groups tasks into eight families. Table 1 lists the tasks, task families and the datasets included in the Full suite. Each dataset belongs to one of the eight task families: (1) Bibtext mining (2) Pair classification (3) Semantic Text Similarity (STS) (4) Clustering (5) Classification (6) Multi-label Classification (7) Retrieval (8). The first four task families performs a similarity related tasks based on two pairs of sentences, either to identity translation pairs, categorize them into a class (e.g., entailment vs. contradiction) or

Read the complete text