SEA-BED: Southeast Asia Embedding Benchmark

Wuttikorn Ponwitayarat 1∗, Raymond Ng 2∗, Jann Railey Montalan 2, Thura Aung 5, Jian Gang Ngui 2, Yosephine Susanto 2, William Tjhi 2, Panuthep Tasawong 1, Erik Cambria 3, Ekapol Chuangsuwanich 4, Sarana Nutanong 1, Peerat Limkonchotiwat 2∗ · 2025

Read SEA-BED: Southeast Asia Embedding Benchmark by Wuttikorn Ponwitayarat 1∗, Raymond Ng 2∗, Jann Railey Montalan 2, Thura Aung 5, Jian Gang Ngui 2, Yosephine Susanto 2, William Tjhi 2, Panuthep Tasawong 1, Erik Cambria 3, Ekapol Chuangsuwanich 4, Sarana Nutanong 1, Peerat Limkonchotiwat 2∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12243.txt.

Introduction

Ponwitayarat and co-authors introduce SEA-BED, a benchmark that tests how well embedding models handle Southeast Asian languages. Text embeddings are often assumed to capture meaning the same way for everyone. Tests here show that assumption fails in practice. The benchmark spans 10 languages of the region and many task types. It checks performance by task, by language, and by their pairing.

No single model did well across all the region's languages. A task that was easy in one language could be hard in the next. Wins on one task rarely carried to others. The map of results is strikingly uneven. The authors urge region-specific testing before models are deployed.

Catalog record

CREATORWuttikorn Ponwitayarat 1∗, Raymond Ng 2∗, Jann Railey Montalan 2, Thura Aung 5, Jian Gang Ngui 2, Yosephine Susanto 2, William Tjhi 2, Panuthep Tasawong 1, Erik Cambria 3, Ekapol Chuangsuwanich 4, Sarana Nutanong 1, Peerat Limkonchotiwat 2∗
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12243

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEA-BED: Southeast Asia Embedding Benchmark Wuttikorn Ponwitayarat 1∗, Raymond Ng 2∗, Jann Railey Montalan 2, Thura Aung 5, Jian Gang Ngui 2, Yosephine Susanto 2, William Tjhi 2, Panuthep Tasawong 1, Erik Cambria 3, Ekapol Chuangsuwanich 4, Sarana Nutanong 1, Peerat Limkonchotiwat 2∗ Abstract Sentence embeddings are essential for NLP tasks such as semantic search, re-ranking, and textual similarity. Although multilingual benchmarks like MMTEB broaden coverage, S outh e ast A sia (SEA) datasets are scarce and often machine-translated, missing native linguistic properties. With nearly 700 million speakers, the SEA region lacks a region-specific embedding benchmark. We introduce SEA-BED , the first large-scale SEA embedding benchmark with 169 datasets across 9 tasks and 10 languages, where 71% are formulated by humans, not machine generation or translation. We address three research questions: (1) which SEA languages and tasks are challenging, (2) whether SEA languages show unique performance gaps globally, and (3) how human vs. machine translations affect evaluation. We evaluate 17 embedding models across six studies, analyzing task and language challenges, cross-benchmark comparisons, and translation trade-offs. Results show sharp ranking shifts, inconsistent model performance among SEA languages, and the importance of human-curated datasets for low-resource languages like Burmese. 1 1 1 Link to SEA-BED: [HIDDEN] (we will add the link to our benchmark and codes in the published version of our paper) \jvol vv \jnum nn 2025 \affilblock 0 0 footnotetext: * Equal contribution 1 Introduction Sentence embedding plays a crucial role in Natural Language Processing (NLP) by transforming complex linguistic structures into fixed-size vector representations that capture semantic meaning. These embeddings are fundamental for various downstream tasks, including semantic textual similarity, retrieval, and re-ranking. To evaluate the robustness of embeddings, numerous studies employ benchmark suites covering these tasks Muennighoff et al. 2023 ; Enevoldsen et al. 2025 . However, most existing benchmarks, such as SentEval Conneau and Kiela 2018 and MTEB Muennighoff et al. 2023 , mainly focus on high-resource languages like English, German, Chinese, and French, leading to the underrepresentation of low-resource languages. Recently, several works have proposed multilingual sentence embedding models and benchmarks. For example, Wang et al. 2024a built a model based on Mistral-7B Jiang et al. 2023 supporting 93 languages and Jina-embedding-v3 Sturua et al. 2024a covering 89 languages spoken worldwide. To broaden benchmarking efforts, Enevoldsen et al. 2025 proposed MMTEB, an extension of MTEB that includes evaluation across 1,090 languages. Despite these advances and coverage, S outh e ast A sian (SEA) languages remain underrepresented. Existing SEA resources in MMTEB–XNLI Conneau et al. 2018 , Tatoeba community 2021 , and SIB-200 Adelani et al. 2024 –are machine-translated datasets, typically translated from English sentences to SEA languages. This lack of native-authored data limits the fluency and linguistic authenticity captured in current benchmarks. As a result, sentence embedding models may struggle to generalize effectively for SEA languages, highlighting a critical gap in current multilingual sentence embedding research. Even though the combined population of the SEA region is close to 700 million 2 2 2 United Nations, Economic and Social Commission for Asia and the Pacific (ESCAP), 2024 , and is the 3rd largest population group in the world, no SEA sentence embedding benchmarks have been established so far. Prior efforts, i.e., IndoNLG Aji et al. 2022 , SEACrowd Lovenia et al. 2024 , and SEA-VL Cahyawijaya et al. 2025 , collect more than 500 datasets and 100 million samples for SEA. Still, these resources were primarily designed for training and evaluating language generation models (i.e., decoder-based models) and are unsuitable for sentence embedding tasks. Moreover, there are many works studied on SEA languages via large language models (LLMs), i.e., SEA-LION Ng et al. 2025 and SEA-LLMs Zhang et al. 2024 . These works underscore the importance of specifically designed benchmarks to identify the challenges and gaps in collecting and processing SEA language data. Building upon these insights, our work addresses the following research questions: (RQ1) Within SEA languages, which specific tasks, scripts, or linguistic features remain problematic for current models, and why do these remain unresolved? (RQ2) How do current multilingual embedding models perform in SEA languages compared to other global languages, and do we see unique performance gaps in SEA? (RQ3) What trade-offs arise between human-annotated versus machine-generated data? To answer our research questions, we propose SEA-BED, a SEA sentence em BED ding benchmark that collects datasets created by native speakers rather than relying primarily on translation. Our benchmark contains more than 169 datasets, 9 tasks, and 10 languages. Crucially, 148 datasets ( \sim 87% of our benchmark) never appear in any sentence embedding benchmarks , and 120 datasets ( \sim 71% of our benchmark) are created by humans rather than machine translation or generation. This makes our benchmark representative of the SEA region more so than any other benchmark by evaluating sentence embedding models on human-crafted datasets, resulting in more reliable and accurate results than machine translation or generation datasets. Moreover, we propose 11 new datasets for Thai and Burmese , which allow us to evaluate the semantic textual similarity, relation understanding, and transfer learning performances. We summarize the differences between our benchmark and previous benchmarks in Table 1 . To evaluate the efficiency of sentence embeddings, we tested 17 embedding models in six distinct studies (Section 5 ) for both encoder- and decoder-based architectures that support SEA and other languages: Sections 5.1 and 5.2 address RQ1 by breaking down model performance across tasks and languages within SEA. Section 5.3 answers RQ2, comparing SEA-focused performance to the broader MMTEB setting. Section 5.4 focuses on RQ3’s design decision analysis, contrasting human-annotated data with machine-translated data in Thai and Burmese. Section 5.5 and 5.6 dive deeper into RQ1 with studies to understand the challenge of our benchmark using a tokenizer-level analysis and a SEA language similarity study. The experimental results from our studies show that the performance of existing multilingual sentence embedding significantly changes compared to MMTEB Enevoldsen et al. 2025 . For example, the ranking of the model that used to perform best over 1,090 languages dropped from second rank to eighth rank when tested on our benchmark, as shown in Figure 2 . We propose that this change in ranking is due to our benchmark including more high-quality SEA datasets than MMTEB, which makes our benchmark more challenging, and crucially more holistically representative for the SEA region at large. Moreover, we found performance and language consistency problems, where no models perform best in all cases for SEA languages. Summary of the contributions of our paper: • We propose SEA-BED, a massive collection of high-quality SEA datasets. Our benchmark consists of more than 169 datasets, 9 tasks, and 10 languages: Indonesian, Thai, Vietnamese, Burmese, Filipino, Tamil, Khmer, Malay, Lao, and Tetum. • We propose 11 new datasets for Thai and Burmese, allowing us to evaluate more tasks compared to previous benchmarks. For reproducibility, we will release the evaluation tool and datasets in the published version of our paper. • We conduct a comprehensive empirical study on SEA languages by evaluating 17 embedding models across 6 distinct studies. Our experiments reveal that performance rankings for multilingual embeddings shift significantly when tested on SEA data, and no single model excels across all tasks and languages. Benchmark # Languages

Read the complete text