Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese

Hanjia Lyu, Jiebo Luo, Jian Kang, Allison Koenecke · 2025

Read Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese by Hanjia Lyu, Jiebo Luo, Jian Kang, Allison Koenecke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22645.txt.

Introduction

Hanjia Lyu, Jiebo Luo, Jian Kang, and Allison Koenecke test bias in large language models across two forms of written Chinese in this 2025 paper. Simplified Chinese is used in mainland China, while Traditional Chinese is used in Taiwan and elsewhere. The authors ask whether models answer worse, or favor one side, depending on the script.

They built two tasks drawn from real use: naming an item whose name differs by region, and picking who to hire from a list of names. Eleven leading models, both commercial and open, were tested. Most models favored Simplified answers on the naming task. On the hiring task, they oddly favored Traditional names instead. The authors link the gaps to training data mix and how the scripts are split into tokens, and they release their test data for reuse.

Catalog record

CREATORHanjia Lyu, Jiebo Luo, Jian Kang, Allison Koenecke
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE110 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22645

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese Hanjia Lyu, Jiebo Luo, Jian Kang, Allison Koenecke Abstract . While the capabilities of Large Language Models (LLMs) have been studied in both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit differential performance when prompted in these two variants of written Chinese. This understanding is critical, as disparities in the quality of LLM responses can perpetuate representational harms by ignoring the different cultural contexts underlying Simplified versus Traditional Chinese, and can exacerbate downstream harms in LLM-facilitated decision-making in domains such as education or hiring. To investigate potential LLM performance disparities, we design two benchmark tasks that reflect real-world scenarios: regional term choice (prompting the LLM to name a described item which is referred to differently in Mainland China and Taiwan), and regional name choice (prompting the LLM to choose who to hire from a list of names in both Simplified and Traditional Chinese). For both tasks, we audit the performance of 11 leading commercial LLM services and open-sourced models—spanning those primarily trained on English, Simplified Chinese, or Traditional Chinese. Our analyses indicate that biases in LLM responses are dependent on both the task and prompting language: while most LLMs disproportionately favored Simplified Chinese responses in the regional term choice task, they surprisingly favored Traditional Chinese names in the regional name choice task. We find that these disparities may arise from differences in training data representation, written character preferences, and tokenization of Simplified and Traditional Chinese. These findings highlight the need for further analysis of LLM biases; as such, we provide an open-sourced benchmark dataset to foster reproducible evaluations of future LLM behavior across Chinese language variants ( https://github.com/brucelyu17/SC-TC-Bench ). Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece The 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25), June 23–26, 2025, Athens, Greece DOI: 10.1145/3715275.3732182 ISBN: 979-8-4007-1482-5/2025/06 CCS: Computing methodologies Natural language processing CCS: Social and professional topics † † cc-license: by 1. Introduction (a) Prompted in Simplified Chinese. Prompted in Simplified Chinese. (b) Prompted in Traditional Chinese. Prompted in Traditional Chinese. Figure 1 . Examples of a prompt question (asked in Simplified Chinese in the left panel, and in Traditional Chinese in the right panel) and the corresponding response for each of three LLMs: GPT-4o , Qwen-1.5 and Taiwan-LLM (LLMs that are English, Simplified Chinese, and Traditional Chinese-oriented, respectively). LLMs do not consistently use culture-specific terms when prompted in the corresponding language variant; for example, Qwen-1.5 answers correctly when prompted in Simplified Chinese, but incorrectly when prompted in Traditional Chinese. English translations of the prompts and responses are written in blue and the script type—whether Simplified or Traditional Chinese—is indicated in bold . Language is a tool for communication and a reflection of culture and identity. In regions with different historical, political, and social paths, unique linguistic systems have developed, resulting in variations in vocabulary, syntax, and meaning. This is exemplified in the Chinese language family, which exhibits significant differences due to geopolitical developments ( Li et al., 2014 ) . Simplified Chinese is predominantly used in Mainland China, and serves as one of the official languages in both Singapore and Malaysia, while Traditional Chinese is used in regions such as Taiwan, Hong Kong, and Macau ( Liu and Hsiao, 2012 ; Yang and Wang, 2018 ) . As Large Language Models (LLMs) have become integral to various applications in daily life ( OpenAI, 2023 ; Wei et al., 2022 ; Zhao et al., 2023 ) , it is increasingly imperative to study their variance in behavior across languages and cultures ( Atari et al., 2023 ; Zhang et al., 2023 ; Yin et al., 2022 ; Acharya et al., 2021 ) , especially as these models become more multilingual—although many remain oriented towards specific languages. For instance, OpenAI’s GPT models are predominantly trained on English language corpora, while Taiwan-LLM ( Lin and Chen, 2023 ) was pre-trained primarily on Traditional Chinese corpora. It is well-studied that LLMs exhibit underperformance when tested on culture-specific commonsense knowledge ( e . g ., Shen et al. (2024) shows underperformance across Chinese, Indian, Iranian, and Kenyan knowledge), political sample simulation ( Qi et al., 2024 ) , and for low-resource languages ( Gurgurov et al., 2024 ) . While Chinese broadly is not considered a low-resource language, prior research has only focused on studying either Simplified Chinese or Traditional Chinese ( Liu et al., 2023b ; Zhang and Li, 2023 ; Xu et al., 2023 ) —but not a comparison of both (see Appendix Table 3 for a literature survey of prior Chinese LLM benchmark work). In contrast, our work aims to directly examine LLM behavior disparities in responses to prompts in either Simplified or Traditional Chinese. While one might expect LLM behavior to be relatively similar between Simplified and Traditional Chinese—especially because most written translation involves one-to-one character mappings—this does not necessarily minimize the existence of biases in the make-up of training data that reflect culturally different expressions or phrases specific to each linguistic variety. We illustrate the types of differences between Simplified and Traditional Chinese using examples of regional terms from Mainland China and Taiwan: • Same term, same word: Terms may share the same word in both Mainland China and Taiwan, although some are written identically while others appear in different scripts: – Same script: For instance, “milk tea” is often written identically in both regions. In both Simplified and Traditional Chinese, it is written as (pronunciation: “nai cha”). – Different scripts: Some terms like “brand” are written differently. In Simplified Chinese, it is written as , but in Traditional Chinese it is written as . Both are pronounced as “shang biao.” • Same term, different words: Terms may be referred to by completely different words in Mainland China and Taiwan. For instance, the term for “computer mouse” is referred to as (pronunciation: “shu biao”) in Mainland China while as (pronunciation: “hua shu”) in Taiwan. Another example is “online shopping,” which is called (pronunciation: “wang shang gou wu”) in Mainland China but (pronunciation: “wang lu gou wu”) in Taiwan. To study the differences in LLM responses to Simplified and Traditional Chinese prompts, our contributions are threefold. First, we design two tasks that reflect real-world scenarios: regional term choice (related to education) and regional name choice (related to hiring). To evaluate LLMs on these tasks, we have constructed and released a benchmark dataset, SC-TC-Bench ( S implified C hinese- T raditional C hinese Bench mark); see Section A.3 for details. Our benchmark contains both question-answer pairs for regional terms that are described above (such as the definitions and terms for “brand” and “computer mouse”), and contains matched lists of regionally-popular names across script variants, along with normalized population counts and likely gender label for each name. Second, we study 11 diverse LLM services by prompting them with questions based on SC-TC-Bench —comparing responses to questions posed in either Simplified Chinese or Traditional Chinese. Illustrative examples are shown in Figure 1 : for example, consider Qwen-1.5 (a Mainland China-based LLM which we refer to as “Simplified Chinese-oriented”). When asked about a yellow spiky tropical fruit in Simplified Chinese, Qwen-1.5 correctly

Read the complete text