Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval
Kidist Amde Mekonnen, Yosef Worku Alemneh, Maarten de Rijke · 2025
Read Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval by Kidist Amde Mekonnen, Yosef Worku Alemneh, Maarten de Rijke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19356.txt.
Introduction
Kidist Amde Mekonnen, Yosef Worku Alemneh, and Maarten de Rijke build search engines that actually speak Amharic. Neural retrieval shines in big languages but stumbles on morphologically rich, low-resource ones, where training data is scarce and tokenizers chop words badly. The team fine-tuned Amharic BERT and RoBERTa models into dedicated embedders. Their RoBERTa version beat a multilingual rival five times its size by 17.6% on ranking quality.
A smaller 42 million parameter sibling stayed competitive at a thirteenth of the bulk. A late-interaction ColBERT model topped the whole table. Data, code, and models are public to seed low-resource search research.
Catalog record
| CREATOR | Kidist Amde Mekonnen, Yosef Worku Alemneh, Maarten de Rijke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval Kidist Amde Mekonnen, Yosef Worku Alemneh, Maarten de Rijke Abstract Neural retrieval methods using transformer-based pre-trained language models have advanced multilingual and cross-lingual retrieval. However, their effectiveness for low-resource, morphologically rich languages such as Amharic remains underexplored due to data scarcity and suboptimal tokenization. We address this gap by introducing Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones. Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10 and a 9.86% gain in Recall@10 over the strongest multilingual baseline, Arctic Embed 2.0 (568M parameters). More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M), remain competitive while being over 13 × \times smaller. Additionally, we train a ColBERT-based late interaction retrieval model that achieves the highest MRR@10 score (0.843) among all evaluated models. We benchmark our proposed models against both sparse and dense retrieval baselines to systematically assess retrieval effectiveness in Amharic. Our analysis highlights key challenges in low-resource settings and underscores the importance of language-specific adaptation. To foster future research in low-resource IR, we publicly release our dataset, codebase, and trained models. 1 1 1 https://github.com/kidist-amde/amharic-ir-benchmarks 1 Introduction As a foundational task in NLP ( NLP ), document retrieval plays a crucial role in applications such as open-domain question answering Chen et al. (2017) and fact-checking Thorne et al. (2018) . Traditional retrieval systems such as TF-IDF and BM25 Robertson and Walker (1997) ; Robertson and Zaragoza (2009) match queries to documents based on lexical overlap. While efficient, they struggle with vocabulary mismatch and semantic ambiguity, limiting their generalizability to synonyms and paraphrases. These challenges are particularly pronounced in morphologically rich languages, where high inflectional variability and complex morphology complicate exact-match retrieval. Suboptimal tokenization in multilingual models further exacerbates these issues, leading to over-segmentation and inefficient subword representations Rust et al. (2021) . As a result, word-based indexing methods fail to capture non-concatenative morphology, affixation, and orthographic variations, degrading retrieval effectiveness. To address these limitations, retrieval models must move beyond lexical overlap and incorporate robust semantic representations. Neural retrieval models. Recent work has introduced several families of neural retrieval methods that leverage transformer-based pre-trained language models to improve retrieval effectiveness, particularly in monolingual English settings. These methods have significantly advanced document ranking, achieving state-of-the-art performance in benchmarks such as MS MARCO Campos et al. (2016) and Natural Questions Kwiatkowski et al. (2019) . Broadly, they fall into three main categories Yates et al. (2021) : (i) learned sparse retrieval ( Formal et al., 2021a , e.g., SPLADE,) , which enhances queries and documents with context -aware term expansions; (ii) dense retrieval ( Karpukhin et al., 2020 , e.g., DPR,) , which maps text into dense vector spaces for efficient retrieval, employing a dual-encoder architecture that encodes queries and documents separately, a design that limits their effectiveness for fine-grained relevance modeling; and (iii) cross-encoders ( Nogueira and Cho, 2019 ; Nogueira et al., 2019 , e.g.,) , which address this limitation by jointly encoding query-document pairs, capturing richer contextual interactions, with a computational overhead that restricts their use to re-ranking candidate documents Humeau et al. (2020) . As an alternative, late-interaction models ( Khattab and Zaharia, 2020 , e.g., ColBERT,) , introduce token-level interactions and strike a balance between the efficiency of dense retrieval and the expressiveness of cross-encoders. A newer paradigm, GenIR ( Metzler et al., 2021 ; Tay et al., 2022 ; Chen et al., 2023 ) , uses pre-trained encoder-decoder models to consolidate indexing, retrieval, and ranking into a single generative framework. While promising, GenIR lags behind dense retrieval in handling large-scale datasets and accommodating dynamic corpora, requiring further study of its scalability and adaptability Pradeep et al. (2023) . Research gap. Despite these advances, neural retrieval remains understudied for morphologically complex, low-resource languages like Amharic. Most retrieval models are optimized for high-resource languages, and prior work has largely focused on cross-lingual transfer from these languages Zeng et al. (2023) . Despite advancements in multilingual embedding models Wang et al. (2024) ; Yu et al. (2024) , these approaches remain inadequate for morphologically rich languages due to suboptimal tokenization, poor subword segmentation, and weak cross-lingual transfer Üstün et al. (2019) . Section 2 further explores the importance of addressing this gap in information retrieval research. Our contribution. To address the gap identified above, we focus on Amharic and introduce optimized retrieval models and benchmarks, making the following key contributions: (i) Amharic text embeddings : we develop dense retrieval models for Amharic, leveraging Amharic BERT and RoBERTa as base models, improving passage ranking accuracy for morphologically complex text. (ii) The first systematic benchmark for Amharic : we evaluate both sparse and dense retrieval models on Amharic, establishing strong baselines for future research. (iii) A language-specific vs. multilingual analysis : we show that Amharic-optimized models consistently outperform multilingual embeddings, underscoring the value of language-specific adaptation. (iv) A public benchmark dataset : We repurpose the Amharic News Text Classification Dataset (AMNEWS) by treating headlines as queries and corresponding articles as passages, creating MS MARCO-style query-passage pairs with heuristic relevance labels. This enables reproducible evaluation of passage ranking models for Amharic. We refer to this processed version as the Amharic Passage Retrieval Dataset . The dataset is publicly available on Hugging Face, 2 2 2 https://huggingface.co/datasets/rasyosef/amharic-news-retrieval-dataset and all code and preprocessing scripts are released on GitHub. 3 3 3 https://github.com/kidist-amde/amharic-ir-benchmarks/tree/main/data 2 Motivation Recent studies highlight systemic shortcomings in low-resource language technologies, leading to retrieval failures, biased outputs, and exposure to harmful or policy-violating content Shen et al. (2024) ; Nigatu and Raji (2024) . For example, Nigatu and Raji (2024) find that Amharic-speaking YouTube users frequently encounter such content due to retrieval systems misinterpreting user intent behind benign queries. These errors stem from foundational limitations in IR ( IR ) systems, which are optimized for high-resource languages like English and struggle with morphologically complex languages like Amharic. The consequences extend beyond search engines: Sewunetie et al. (2024) demonstrate that retrieval failures in machine translation propagate gender bias, defaulting Amharic occupational terms to male forms even when the context is gender-neutral. Such errors reflect broader research gaps in NLP , where systems disproportionately prioritize high-resource languages, thereby exacerbating inequities faced by underrepresented linguistic communities Shen et al. (2024) . Amharic, the working language of Ethiopia’s federal government and one of the most widely spoken Semitic languages Gezmu et al. (2018) , presents unique challenges for IR . Its root-based templatic morphology allows a single root to generate numerous derived forms through affixation and vowel pattern changes. These morphological variations, combined with the Ge’ez script, an Abugida writing system with 33 base characters and over 230 syllabic forms, make Amharic structurally and morphologically distinct from Indo-European and other high-resource languages. As a result, conventional retrieval models tend to underperform without language-specific adaptation. Addressing these challenges requires Amharic-specific embedding models tailored for passage retrieval. While recent efforts Belay et al. (2021) ; Azime et al. (2024b) have advanced Amharic NLP , their primary focus has not been on optimizing retrieval performance. Our work fills this gap by developing and benchmarking retrieval methods specifically adapted to Amharic’s linguistic characteristics, laying a foundation for more equitable and semantically accurate information access in