GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training

Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, Wadii Boulila · 2025

Read GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training by Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, Wadii Boulila (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24581.txt.

Introduction

Nacar and co-authors build GATE, a text code set tuned for Arabic sense match. Sense match scores how close two texts sit in what they mean. It backs hunt, group, and sense work in word tools. Arabic work in that vein lags for lack of good sets and models. Thin stock has held back both tests and gains in the field. GATE codes use nest-style learn, where codes hold codes in size steps.

A mixed loss plan trains the codes on Arabic triple sets. Those sets come from sense-draw tasks that need fine splits. GATE tops the MTEB sense match tests in the team's runs. It beats more big models, OpenAI's in the mix. Gains of 20 to 25 percent show on those test sets. Fine traits of Arabic sense come through in the codes.

Catalog record

CREATOROmer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, Wadii Boulila
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GATE: General Arabic Text Embedding for Enhanced Semantic Textual Similarity with Matryoshka Representation Learning and Hybrid Loss Training Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, Wadii Boulila Abstract Semantic textual similarity (STS) is a critical task in natural language processing (NLP), enabling applications in retrieval, clustering, and understanding semantic relationships between texts. However, research in this area for the Arabic language remains limited due to the lack of high-quality datasets and pre-trained models. This scarcity of resources has restricted the accurate evaluation and advance of semantic similarity in Arabic text. This paper introduces General Arabic Text Embedding (GATE) models that achieve state-of-the-art performance on the Semantic Textual Similarity task within the MTEB benchmark. GATE leverages Matryoshka Representation Learning and a hybrid loss training approach with Arabic triplet datasets for Natural Language Inference, which are essential for enhancing model performance in tasks that demand fine-grained semantic understanding. GATE outperforms larger models, including OpenAI, with a 20-25% performance improvement on STS benchmarks, effectively capturing the unique semantic nuances of Arabic. 1 Introduction Text embeddings drive advances in clustering, information retrieval, and semantic similarity Reimers (2019) ; Gao et al. (2023) ; Asai et al. (2023) ; Gao et al. (2021) . These models aim to map textual information into dense, low-dimensional vector representations that preserve nuanced semantic and contextual relationships. At the heart of many highly effective embedding models lies contrastive learning, a paradigm that optimizes the quality of representation by pulling semantically similar (positive) samples closer while pushing dissimilar (negative) samples apart Gao et al. (2021) ; He et al. (2020) ; Radford et al. (2021) . Despite the versatility and success of contrastive learning, most existing text embedding pipelines rely on a two-stage pre-train-fine-tuning process: weakly supervised large-scale pre-training followed by fine-tuning on high-quality text pairs acquired through data mining or manual annotation Li et al. (2023) ; Wang et al. (2022) ; Xiao et al. (2023) . Although effective, this approach often relies on the standard InfoNCE loss with in-batch negative samples He et al. (2020) , achieving robust representations predominantly by using large batch sizes and numerous negative samples. However, InfoNCE alone is not sufficient for all downstream tasks. In particular, sentence-level tasks such as Semantic Textual Similarity (STS) have been shown to benefit less from InfoNCE-based training, indicating a limitation in capturing fine-grained similarity cues Huang et al. (2024) . Likewise, key NLP tasks such as STS and classification have yet to be thoroughly integrated into general embedding training objectives. Arabic presents specific linguistic challenges that complicate Semantic Textual Similarity (STS) tasks. Although Arabic is the fourth most used language on the Internet Li and Yang (2018) and the fifth most spoken language worldwide Bourahouat et al. (2024) , high-quality Arabic text embeddings are scarce. This scarcity exacerbates issues arising from Arabic’s rich morphological structure, characterized by a root-and-pattern system that generates a multitude of derivations, and its flexible syntax, where variable word orders can obscure semantic parallels. Additionally, the frequent omission of diacritics in written Arabic leads to significant ambiguity, as identical word forms may convey different meanings in context. These challenges collectively restrict the accurate capture of semantic nuances, making STS tasks particularly demanding for Arabic NLP applications. This paper tackles these issues by introducing GATE, a General Arabic Text Embedding model designed to excel in semantic textual similarity and other downstream tasks. Our approach integrates Matryoshka Representation Learning (MRL) Kusupati et al. (2022) with a multitask hybrid loss training method. More specifically, we exploit various loss functions tailored to different task objectives—e.g., cosine similarity-based loss for STS and classification-oriented loss for downstream classification tasks. GATE improves semantic distinction by leveraging hard negative datasets and a flexible embedding structure. It addresses the limitations of single-loss approaches like InfoNCE-only training Gutmann and Hyvärinen (2010) . Our contributions are the following: • Hybrid Loss Strategy: We propose a hybrid loss combining cosine similarity for semantic tasks and softmax-based classification, improving Arabic textual similarity beyond standard InfoNCE. • Enhanced Model Robustness: We incorporate curated Arabic NLI triplet and labeled pair datasets, capturing nuanced semantic relationships crucial for downstream tasks. • Scalable Arabic Embeddings: We adapt Matryoshka Representation Learning to Arabic, enabling efficient multi-dimensional embeddings (768, 512, 256, 128, and 64) with strong performance across tasks. Our models, and data for reproducibility are publicly available GATE Collection . The paper is organized as follows. Section 2 reviews related work. Section 3 covers the proposed GATE framework, including our annotated datasets, Matryoshka embeddings training settings, along with a hybrid loss training approach. Section 4 presents the experimental results, evaluation, benchmarking, and error analysis. 2 Related Work Work Embedding Size Primary Language Focus Hybrid Loss Multi-Dimensional Semantic-Rich Fine-Tuning OpenAI Text-Embedding v3 OpenAI (2023) 1536 / 3072 Multilingual ✗ ✓ ✗ E5-Mistral-7B-Instruct Wang et al. (2023) 4096 English-Focused ✗ ✗ ✗ Udever-Bloom-1B1 Zhang et al. (2023) 1536 Multilingual ✗ ✗ ✗ AraBERT Antoun et al. (2020) 768 Arabic-Specific ✗ ✗ ✗ MARBERT Abdul-Mageed et al. (2020) 768 Arabic-Specific ✗ ✗ ✗ LaBSE Feng et al. (2020) 768 Multilingual ✗ ✗ ✗ Multilingual E5 Wang et al. (2024) 384 Multilingual ✗ ✗ ✗ GATE Models (Proposed) 768, 512, 256, 128, 64 Semantic Arabic-Specific ✓ ✓ ✓ Table 1 : Comparison of GATE with existing models by loss type, embedding dimensions, and training. Semantic Textual Similarity (STS) Cer et al. (2017) is a fundamental task in Natural Language Processing (NLP) that measures how closely two sentences align in meaning. Unlike binary classification tasks such as textual entailment or paraphrase detection, STS provides a graded measure of semantic equivalence Zhao et al. (2024) . It serves as a cornerstone for various NLP applications, including machine translation Pathak et al. (2019) , text summarization Liu et al. (2022) , and question answering Wu et al. (2021) , making it a crucial benchmark for evaluating embedding models. Matryoshka Representation Learning (MRL) has emerged as an innovative approach to enhancing text embeddings by introducing hierarchical embedding representations, enabling models to capture multiple fidelity levels while optimizing computational efficiency Kusupati et al. (2022) . By dynamically encoding information across varying dimensions, MRL reduces storage requirements and computational overhead without compromising accuracy. Recent advancements, such as OpenAI’s text-embedding v3 OpenAI (2024) , have demonstrated the effectiveness of MRL in semantic representation learning, influencing modern embedding architectures Koenig et al. (2024) ; Lee et al. (2024) ; Infgrad (2024) . Large language models (LLMs) have significantly advanced text embeddings, leveraging massive parameter spaces for complex semantic representations. Models such as E5-Mistral-7B-Instruct Wang et al. (2023) and Udever-Bloom-1B1 Zhang et al. (2023) enhance generalization across domains but remain predominantly optimized for English. OpenAI’s third-generation embedding models OpenAI (2023) offer strong multilingual performance but are computationally expensive and lack adaptability for Arabic-specific tasks. In Arabic NLP, models like AraBERT and MARBERT have improved language understanding by transitioning from masked language models (MLMs) to sentence embeddings Reimers and Gurevych (2019) . While AraBERT focuses on formal Arabic Antoun et al. (2020) , MARBERT extends coverage to dialectal Arabic through large-scale pretraining Abdul-Mageed et al. (2020) . Multilingual models such as LaBSE, SBERT, and Multilingual E5 aim to bridge cross-lingual gaps, supporting over 100 languages. However, they struggle with fine-grained Arabic semantics,

Read the complete text