IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages
Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Fajri Koto · 2026
Read IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages by Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Fajri Koto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01260.txt.
Introduction
Hanif and co-authors build IndoBias, a bias test rooted in Indonesian life. Indonesia holds over 1,300 ethnic groups. It also holds some 700 local languages. Yet model bias there has seen little study. IndoBias tests models in Indonesian and three local tongues. These are Javanese, Sundanese, and Makasar. It runs two tracks, one deep and one broad.
The deep track uses paired contrast sentences. The broad track uses free generation, tied to social science frames. Models show strong bias toward textbook-style sentences in Indonesian. Local tongues show more bias on ideology and religion. Web crawl text adds more bias than reviewed texts. Adding local tongues to training also tends to raise bias.
Catalog record
| CREATOR | Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Fajri Koto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages Ikhlasul Akmal Hanif, Muhammad Falensi Azmi, Filbert Aurelian Tjiaranata, Eryawan Presma Yulianrifat, Fajri Koto Abstract Despite being home to more than 1300 ethnic groups and 700 indigenous languages, bias in Large Language Models has not been fully studied in Indonesia, thus leaving a critical gap in evaluating representational fairness and localized stereotypes within its uniquely vast, multilingual, and diverse sociocultural landscape. To address this, we introduce IndoBias as a culturally-grounded bias benchmark to assess LLMs bias in Indonesian and three local languages: Javanese, Sundanese, and Makasar. IndoBias features dual perspective evaluation tracks: depth-oriented (with contrastive-pairs) and breadth-oriented (with generation-based), where the latter is grounded in social science frameworks (SPI, O*NET, and WGI). Our results show that existing LLMs—particularly decoder models—exhibit strong bias towards prototypical sentences in Indonesian, while local languages suffer higher bias under Ideology and Religion category. We also find that LLMs responses exhibit a non-uniform Stereotype Polarity when prompted with various local entities. Finally, we discover that, in Indonesian, Common Crawl texts introduce more bias during pretraining, compared to human-reviewed article texts (e.g., Wikipedia, News), whereas introducing local languages to pretraining generally increases bias. This work highlights the importance of studying bias in culture-specific context. Warning: This paper contains example data that may be offensive, harmful, or biased. 1 Introduction Figure 1: Example of model bias: (a) Lower perplexity is assigned to a prototypical statement in Indonesian, whereas in Javanese, perplexity of counter-stereotypical statement is lower (both pairs representing the same underlying stereotype in different languages); (b) The model labels the Korowai as “lawless”, while the Sundanese is portrayed as “law-abiding”. Large Language Models (LLMs) perform remarkably well on many NLP tasks, yet they continue to absorb and reproduce societal stereotypes from the vast, unfiltered text corpora used for pre-training 4 ; 3 . These stereotypes often reflect widespread but inaccurate beliefs that can harm individuals and groups, even when they appear superficially positive 10 . For example, a model might complete the prompt “An ideal employee is…” with “an Asian who is hardworking,” reinforcing the “model minority” myth and implicitly narrowing the perceived abilities of an entire demographic. Because such biases can propagate to downstream applications like hiring, sentiment analysis, and content moderation, they risk disadvantaging already underrepresented communities 45 ; 63 . Figure 2: IndoBias-Pairs construction pipeline. Considerable effort has been invested in English-centric bias benchmarks, resulting in resources such as CrowS-Pairs 29 , StereoSet 28 , and WinoBias 62 . However, stereotypes are not universal; they are shaped by local cultural, linguistic, and historical contexts, making it essential to develop evaluation datasets that reflect specific societies. Following this need, culturally aware benchmarks like RuBia 16 (for Russian) adopt the CrowS-Pairs format of contrasting stereotypical and anti-stereotypical sentences. Yet this approach has a structural blind spot: it relies on widely recognized tropes to achieve reliable inter-annotator agreement, which systematically excludes smaller or less visible groups that lack “common” stereotypes. This creates an evaluation paradox where focusing on the most prominent biases perpetuates the erasure of the very minorities most vulnerable to representational harm 56 . Indonesia exemplifies both the opportunity and the challenge of culturally grounded bias research. The nation’s motto, Bhinneka Tunggal Ika (Unity in Diversity), reflects a reality of over 1,300 ethnic groups 1 1 1 https://iwgia.org/en/indonesia.html and more than 700 local languages 1 . 2 2 2 https://www.ethnologue.com/country/ID/ In this multicultural, multi-religious, multilingual landscape, bias is rarely uniform. Major ethnic groups like the Javanese or Sundanese may be subject to well-documented stereotypes, while hundreds of smaller communities from Papua, Indonesian Borneo, or East Nusa Tenggara suffer from invisibility bias . As shown in Figure 1 , Korowai 3 3 3 Korowai is a marginalized ethnic group from Papua is unfairly disadvantaged, while the opposite is observed for Sundanese 4 4 4 Sundanese is the second-largest ethnic group in Indonesia . Additionally, we find that LLMs exhibit different stereotype bias directions for different languages, highlighting the importance of involving local languages during bias evaluation. Because standard evaluation frameworks often fail to capture this intricate web of linguistic and demograpic group prejudices, we introduce IndoBias . This culturally grounded, dual-track benchmark is specifically designed to capture both the depth and breadth of bias in the Indonesian sociolinguistic landscape. (1) IndoBias-Pairs offers 544 manually curated sentence pairs in each of four languages: Indonesian, together with three local languages, Javanese, Sundanese, and Makasar, representing western and eastern Indonesia (4,352 instances in total). Every pair places a prototypical statement that reinforces a common stereotype next to a counter-stereotypical statement that challenges it. The pairs are organized into five broad bias domains central to Indonesian society and further split into 18 fine-grained subdomains. (2) IndoBias-QA is a generation-based evaluation built on the LLM Stereotype Index framework 46 . It measures stereotype polarity across 336 Indonesian demographic groups using seven task formats of increasing complexity. To allow finer evaluation, we extend the stereotype axis with indices from social science: the Social Progress Index (SPI) 37 , O*NET 31 , and the Worldwide Governance Indicators (WGI) 22 . Our contributions are threefold: 1. IndoBias (described above) is the first culturally grounded bias benchmark for the Indonesian context, covering four languages across both contrastive and generation-based evaluation tracks. 2. We evaluate LLM bias on both tracks: on IndoBias-Pairs, we benchmark a broad set of encoder and decoder models spanning general (multilingual), Southeast Asian, and Indonesian-specific architectures; on IndoBias-QA, we analyze stereotype polarity across 336 Indonesian demographic groups to explore potential bias discrepancies across 6 demographies. 3. We conduct a controlled pretraining simulation over 500,000 steps across six data compositions, examining how corpus source and multilingual data mixing shape the emergence of cultural bias throughout pretraining. Figure 3: IndoBias-QA construction pipeline. 2 Related Works Benchmarks for measuring language model bias typically follow two directions: contrastive-based evaluation or generation-based evaluation . In contrastive-based evaluation, bias is measured by comparing the model’s assigned probabilities for stereotypical versus anti-stereotypical sentences. Typically, these sentences are constructed using templates, where a specific slot is filled with the bias aspect under investigation (e.g., "Asians are [hardworking/lazy]"). Alternatively, generation-based evaluation is conducted by prompting an LLM to generate a response (can be open-ended or close-ended) to assess whether the output distribution is biased toward a certain stereotype (e.g., "Generate a story about a character from the [TRIBE_NAME] tribe"). Our research encompasses both paradigms through the IndoBias-Pairs (for contrastive-based evaluation) and the IndoBias-QA (for generation-based evaluation). Contrastive-based Evaluation The CrowS-Pairs benchmark and its adapted versions (e.g., French ( 32 ) , Hindi ( 44 ) , Dutch ( 48 ) , Filipino ( 14 ) ) evaluate LLM bias by measuring the model’s preference for stereotypical contexts when conditioned on specific demographic identifiers. Other works follow similar methodology ( 41 ; 23 ; 20 ) , while other studies propose alternative approaches, such as assessing LLM bias relative to other models ( 2 ) , measuring bias via embedding similarity ( 42 ) , or evaluating bias from information theory perspective ( 47 ; 13 ) . Recent progress has also been made in evaluating bias in multilingual contexts; for example, 60 study gender bias in masked