SailCompass : Towards Reproducible and Robust Evaluation for Southeast Asian Languages
Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, Qian LiuNational University of Singapore; Sea AI Lab;Singapore University of Technology and Design [email protected], [email protected] {doulx, liuqian}@sea.com · 2024
Read SailCompass : Towards Reproducible and Robust Evaluation for Southeast Asian Languages by Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, Qian LiuNational University of Singapore; Sea AI Lab;Singapore University of Technology and Design [email protected], [email protected] {doulx, liuqian}@sea.com (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01186.txt.
Introduction
Jia Guo and five colleagues present a 2024 benchmark paper on judging language models in Southeast Asian languages. Tests built for English miss how models handle Malay, Indonesian, Vietnamese, and their neighbors. SailCompass gathers three main regional languages across eight task types and fourteen datasets, spanning generation, multiple choice, and sorting tasks.
The team hardened the scoring itself, trying prompt variants for multiple choice and calibration tricks for the sorting tasks so results stay reproducible. Regional specialist models still beat general ones, though the gap narrowed. Balanced language data mattered for the specialists, and careful prompting proved necessary to get honest scores. All datasets and scripts are public, giving the region a shared yardstick for model quality.
Catalog record
| CREATOR | Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, Qian LiuNational University of Singapore; Sea AI Lab;Singapore University of Technology and Design [email protected], [email protected] {doulx, liuqian}@sea.com |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01186
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SailCompass : Towards Reproducible and Robust Evaluation for Southeast Asian Languages Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, Qian LiuNational University of Singapore; Sea AI Lab;Singapore University of Technology and Design [email protected], [email protected] {doulx, liuqian}@sea.com Abstract In this paper, we introduce SailCompass, a reproducible and robust evaluation benchmark for assessing Large Language Models (LLMs) on Southeast Asian Languages (SEA). SailCompass encompasses three main SEA languages, eight primary tasks including 14 datasets covering three task types (generation, multiple-choice questions, and classification). To improve the robustness of the evaluation approach, we explore different prompt configurations for multiple-choice questions and leverage calibrations to improve the faithfulness of classification tasks. With SailCompass, we derive the following findings: (1) SEA-specialized LLMs still outperform general LLMs, although the gap has narrowed; (2) A balanced language distribution is important for developing better SEA-specialized LLMs; (3) Advanced prompting techniques (e.g., calibration, perplexity-based ranking) are necessary to better utilize LLMs. All datasets and evaluation scripts are public 1 1 1 https://github.com/sail-sg/sailcompass . 1 Introduction Recent advancements in Large Language Models (LLMs) have led to numerous successful applications in language understanding and generation. However, current LLM research are mainly focus on English, Chinese and other Western languages, often overlooking other languages especially for low-resource languages like SEA languages. Southeast Asia (SEA) is a vital region worldwide, comprising 11 countries and a population of approximately 675 million people (8.5% of the world). For NLP research, SEA region boasts remarkable linguistic diversity, which provides an ideal research ground for multilingual studies. For instance, Indonesia alone has over 700 languages spoken daily [ 3 ] , and also presents linguistic similarities, such as the shared terminology between Malay and Indonesian due to their intertwined history and culture. Recently, SEA languages have witness several outstanding open models like SeaLLM [ 27 ] , Sealion [ 2 ] and Sailor [ 14 ] , which are built from scratch or continual pre-training on English-centric models. However, most SEA benchmarks are either narrowed in task diversity or limited in examples scale, could not trustworthy measure the model performance. We believe that a reproducible and robust evaluation system for LLMs, could largely assist the researchers to assess the the system’s reliability and quantify the existing drawbacks. It just likes a Compass for the Sailors to ship in the Sea. In this paper, we present SailCompass, a reproducible and robust evaluation system for SEA languages in LLM era. Our main contributions include benchmark, evaluation approache and evaluation findings. Comprehensive Evaluation Datasets (1) For evaluation faithfulness, we encompass three main SEA languages, eight primary tasks across three task formulations types (i.e., generation, multi-choice questions and classification), that demanding for both language proficiency and cultural understanding. (2) For evaluation efficiency, we build SailCompass on the OpenCompass framework [ 28 ] , for efficient running, extensible configuration and better visualization. Robust Evaluation Approach (1) For multiple-choice question tasks, we explore all prompt variants to identify the most robust configurations. (2) For classification tasks, we leverage calibration to mitigate label bias and generate faithful outputs instead of meaningless random ones. Insightful Findings for Future Work (1) We investigate the performance gap between general LLMs and SEA-specialized LLMs to analyse the future trend for application usage. (2) We highlight the importance of balanced language distribution for developing better SEA LLMs, certified by the performance of machine translation and text summarization. (3) We recognize the significance of advanced prompting technique (e.g., calibration, PPL-based ranking) for make better use of LLMs. 2 Related Work This section focuses on SEA benchmarks. Refer to App. B for general multilingual benchmarks. SEA benchmarks for Base Model NusaCrowd [ 8 ] introduced the first large Indonesian benchmark containing 137 datasets and over 200 tasks across 19 Indonesian languages. BHASA [ 22 ] presented an evaluation suite containing linguistic and cultural evaluations for SEA languages. But BHASA only cares for zero-shot evaluation of API models like GPT-3.5, and only limited dataset size. SeaEval [ 43 ] additionally considers the cultural understanding ability of models, but the majority of the datasets are still based on high-resource languages, such as English and Chinese. SEA benchmarks for Chat Model Sea-bench [ 27 ] is a multilingual dataset with instructions across 9 SEA languages for evaluating chat model, examining five abilities like math, safety and task-solving. The linguists sourced the data by manually translating open-source English test sets, collecting real user questions from local forums and websites, collecting real math and reasoning questions, writing test instructions and questions themselves. Distinguishing from these works, SailCompass aims to examine the instruction-following and few-shot learning ability of open-source LLMs through a considerable amount of tasks and datasets, with meticulous curation of datasets collected directly from native sources. Additionally, SailCompass places particular emphasis on evaluating tasks with multiple-choice targets and providing valuable insights through analysis. In this work, SailCompass focus on open base model evaluation, covering three main SEA languages and 14 tasks, utilizing prompting and calibration to realize a robust evaluation benchmark. 3 SailCompass Benchmark This section introduces SailCompass , an integrated evaluation suite, encompassing (1) the comprehensive multilingual benchmarks; (2) the reproducible evaluation codebase. 3.1 Benchmark Construction Principle Prioritizing by the number of speakers, we consider three SEA languages: Indonesian, Vietnamese, and Thai. To build a comprehensive benchmark, we first identify four pivotal aspects for evaluating language models: (1) language proficiency; (2) reading comprehension; (3) reasoning ability; (4) cultural understanding. Based on these principles, we select a range of tasks and high-quality datasets. In addition to task diversity, we prefer datasets created by native speakers and built on native corpora rather than translated from English benchmarks. These native benchmarks involve more localized entities, better examining the models’ ability in cultural understanding and geographical knowledge. Unfortunately, such native-created datasets are limited, even after our best efforts to collect them. Thus, we also select high-quality multilingual datasets, like XNLI [ 12 ] and XQuAD [ 4 ] , to further supplement the number of datasets. Table 1: Our benchmark includes eight tasks: Question Answering (QA), Machine Translation (MT), Text Summarization(TS), Examination (Exam), Commonsense Reasoning (CR), Machine Reading Comprehension (MRC), Natural Language Inference (NLI), and Sentiment Analysis (SA). Type Datasets Thai Indonesian Vietnamese Task Domain GEN XQuAD 1,149 – 1,169 Wikipedia TyDiQA – 565 – QA Wikipedia Flores-200 1,012 1,012 1,012 MT Wikipedia ThaiSum 3,671 – – News IndoSum – 3,762 – News XLSUM – – 2,676 TS News MCQ M3Exam 2,163 371 2 2 2 Javanese 1,789 Exam School materials XCOPA 500 500 500 CR General BELEBELE 900 900 900 MRC General CLS XNLI 5,010 – 5,010 Multi-genre IndoNLI – 5,182 – NLI Multi-genre Wisesight 2,614 – – Social media Indolem – 1,002 – Social media VSMEC – – 692 SA Social media 3.2 Task and Dataset Construction We construct SailCompass on eight tasks across three task types. 3 3 3 For datasets that no publicly available test set, we employ the validation set, like TydiQA. Refer to Table 1 for statistics. (1) Generation tasks ( GEN ) : Generate the token sequence given sequence input. • Question Answering (QA) generates an answer span given the question and passage. We employ XQuAD [ 4 ] for Thai and Vietnamese and TyDiQA [ 11 ] for