LoraxBench : A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
Alham Fikri Aji Trevor Cohn, [3mm] MBZUAI The University of Melbourne Google · 2025
Read LoraxBench : A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages by Alham Fikri Aji Trevor Cohn, [3mm] MBZUAI The University of Melbourne Google (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12459.txt.
Introduction
Alham Fikri Aji and Trevor Cohn introduce a benchmark for twenty Indonesian languages in this 2025 paper. Indonesia is one of the most populous nations, with about seven hundred languages spoken. Its local languages lag behind in language technology. Their suite, LoraxBench, covers six tasks, from reading and question answering to translation. It also tests causal reasoning and questions about culture. Three languages appear in two formality registers as well.
The team ran many multilingual and region-focused AI models on the suite. Scores lagged far behind those for standard Indonesian. That gap was worst for the lowest-resource languages. Region-specific models held no clear lead over general ones. Changing the formality register also shifted scores in real ways. The authors offer the suite as a yardstick for fairer language tools.
Catalog record
| CREATOR | Alham Fikri Aji Trevor Cohn, [3mm] MBZUAI The University of Melbourne Google |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoraxBench : A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages Alham Fikri Aji Trevor Cohn, [3mm] MBZUAI The University of Melbourne Google Abstract As one of the world’s most populous countries, with 700 languages spoken, Indonesia is behind in terms of NLP progress. We introduce LoraxBench , a benchmark that focuses on low-resource languages of Indonesia and covers 6 diverse tasks: reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural QA. Our dataset covers 20 languages, with the addition of two formality registers for three languages. We evaluate a diverse set of multilingual and region-focused LLMs and found that this benchmark is challenging. We note a visible discrepancy between performance in Indonesian and other languages, especially the low-resource ones. There is no clear lead when using a region-specific model as opposed to the general multilingual model. Lastly, we show that a change in register affects model performance, especially with registers not commonly found in social media, such as high-level politeness ‘Krama’ Javanese. 1 Introduction Figure 1: Tasks covered in LoraxBench Indonesia is one of the world’s most populous nations and one of its most linguistically diverse, being home to over 700 languages. Despite this, NLP research has disproportionately focused on Bahasa Indonesian and a few dominant languages, such as Javanese and Sundanese, leaving the vast majority of languages under-resourced and under-explored Aji et al. 2022 . A major challenge for NLP in Indonesia is the lack of resources, as the shortage of data and benchmarks continues to hinder progress Hu et al. 2020 ; Joshi et al. 2020 . Moreover, even for relatively well-studied and well-resourced languages, the range of NLP tasks explored remains limited Cahyawijaya et al. 2023a . To bridge this gap, we present LoraxBench 1 1 1 LoraxBench is available at https://huggingface.co/datasets/google/LoraxBench , a benchmark of six NLP tasks across 20 Indonesian languages: reading comprehension, machine translation, cultural reasoning, natural language inference, and cultural question answering. Our data covers many low-resource languages, including some with little to no coverage according to the comprehensive region-specific catalogue NusaCrowd Cahyawijaya et al. 2023a . While focused on Indonesia, our work reflects challenges common in other linguistically diverse yet resource-scarce regions. Progress on LoraxBench can thus inform multilingual and multicultural model development globally. Beyond multilingual capabilities, our benchmark also encompasses various registers for some languages, specifically formal and casual settings. Some of Indonesian languages, in particular, are rich in registers, with highly different use of language depending on formality of the context Rahayu 2014 . Unfortunately, little research has been done in this area Farhansyah et al. 2025 . As LLMs become more integrated into daily life, such as in personal assistants, ensuring they understand and appropriately use register is vital. Our data serves as a benchmark to evaluate this capability. We built LoraxBench by adapting existing Indonesian-language datasets through expert translation. This approach mitigates common issues with translating English datasets, especially around cultural relevance, as many concepts are widely shared across Indonesian languages but absent in Western contexts. Moreover, the parallel nature of the data enables comparison across languages. We evaluate several prominent multilingual, Southeast Asian and Indonesian-focused LLMs on LoraxBench , revealing significant performance disparities across languages, particularly for lower-resource languages and the more challenging polite registers, which are less represented in online data. Finally, we explore the potential of leveraging high-quality lexicons to improve model performance on specific languages. In summary, our contributions are as follows: • We propose a new human-written benchmark for Indonesian local languages that covers 20 languages across 6 diverse tasks. • For 3 languages in our benchmark we include both casual and formal registers, facilitating analysis of robustness to register. • We benchmark various LLMs, from multilingual models to Indonesian-specific models, on this dataset. 2 Related Work Benchmarks for Indonesian Languages Several multilingual NLP benchmarks include Indonesian, such as Flores Goyal et al. 2022a , XNLI Conneau et al. 2018 , XCOPA Ponti et al. 2020 , and Massive FitzGerald et al. 2022 providing evaluation datasets for cross-lingual understanding and reasoning. However, these benchmarks typically only cover Indonesian and sometimes a small set of Indonesian local languages. Additionally, their English-centric data construction often results in content that is not contextually relevant. To address these limitations, dedicated efforts have been made to develop benchmarks with a stronger focus on Indonesian-specific content. Examples include IndoNLU Wilie et al. 2020 , IndoNLI Mahendra et al. 2021 , IndoMMLU Koto et al. 2023 , and COPAL-ID Wibowo et al. 2024 . Other benchmarks, such as NusaWrites Cahyawijaya et al. 2023b and NusaX Winata et al. 2023 , have been designed to evaluate regional languages, typically covering low-resource Indonesian languages. Our work improves in this direction by providing benchmark with Indonesian-relevant content that covers more languages and tasks. Benchmarks for Low-Resource Languages Beyond benchmarks for Indonesian languages, we also see recent progress in benchmarks for other languages, especially those that are underexplored. Efforts such as MasakhaNER Adelani et al. 2021 ; Adelani et al. 2022 and MasakhaNews Adelani et al. 2023 are enriching datasets for African languages, while initiatives for Indic languages, such as IndicNLP Suite Kakwani et al. 2020 , are driving similar advancements in the South Asian context. These efforts help to address the data gap for low-resource languages, facilitating more inclusive and robust language models across diverse communities. We also see several massively multilingual benchmarks that cover languages across the globe. MASSIVE FitzGerald et al. 2022 is an intent-classification task for 60 languages. Belebele Bandarkar et al. 2023 is a large-scale reading comprehension benchmark covering 122 languages and language variants. Flores Goyal et al. 2022b is a machine translation benchmark covering 200 languages. INCLUDE Romanou et al. 2024 and Global-MMLU Singh et al. 2024 are exam-like benchmarks for more than 40 languages. Despite their broad coverage, only a small fraction of Indonesian languages are included, typically Indonesian, Javanese, and Sundanese. Not only that, these benchmarks often were translated from English, resulting in context bias that might not fully capture cultural nuances in Indonesia Mihalcea et al. 2024 . Our proposed benchmark covers more languages that are not typically included in massively multilingual benchmarks. 3 LoraxBench 3.1 Language of Focus Language Speakers Spoken in Acehnese (ace) 3.7 M Aceh Ambonese Malay (abs) 0.2 M Ambon Balinese (ban) 4.8 M Bali Banjar (bjn) 4.0 M South Sulawesi Batak Toba (bbc) 2.5 M North Sumatra Betawi (bew) 5.6 M Jakarta Buginese (bug) 4.3 M South Sulawesi Gorontalo (gor) 1.1 M Gorontalo Iban (iba) 0.8 M West Kalimantan Jambi Malay (jax) 1.0 M Jambi Javanese (jv) 91.0 M East/Central Java Lampung Nyo (abl) 1.5 M Lampung Madurese (mad) 17.0 M East Java Makasar (mak) 1.9 M Makasar Minangkabau (min) 8.0 M West Sumatra Musi (mui) 3.1 M South Sumatra Ngaju (nij) 0.9 M Central Kalimantan Sasak (sas) 2.6 M West Nusa Tenggara Sundanese (su) 32.0 M West Java Table 1: Statistics of the languages of focus in LoraxBench , based on LinguaMeta Ritchie et al. 2024 This work focuses on Indonesian and 19 Indonesian local languages, representing a diverse range of population sizes and geographical regions, as detailed in Table 1 . Several of these languages have not previously been included in public downstream NLP tasks,