Do Chinese models speak Chinese languages?
Andrea Wen-Yi Wang, Unso Eun Seo Jo, David Mimno · 2025
Read Do Chinese models speak Chinese languages? by Andrea Wen-Yi Wang, Unso Eun Seo Jo, David Mimno (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00289.txt.
Introduction
Wang, Jo, and Mimno ask in twenty twenty-five whether language models built in China actually serve the languages spoken in China. Top open Chinese models have cemented China's lead in AI. Yet global test sets reward English skill, which may skew what builders prioritise, at the cost of local language needs.
They compared Chinese and Western open models across twenty-one language variants, from Mandarin to minority and European tongues. Scores tracked closely between the two groups, except that Chinese models led in Mandarin. Chinese models did well in French and German, yet sometimes failed to even name minority languages like Kazakh and Uyghur. The authors blame shared global benchmarks and training sources for the sameness.
Catalog record
| CREATOR | Andrea Wen-Yi Wang, Unso Eun Seo Jo, David Mimno |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do Chinese models speak Chinese languages? Andrea Wen-Yi Wang, Unso Eun Seo Jo, David Mimno Abstract . The release of top-performing open-weight LLMs has cemented China’s role as a leading force in AI development. Do these models support languages spoken in China? Or do they support the same languages as models developed in the United States or in Europe? Comparing multilingual capabilities is important for two reasons. First, language ability provides insights into pre-training data curation, and thus into resource allocation and development priorities. Second, Chinese model developers need to navigate the tension between serving a linguistically diverse population domestically, and optimizing for globally visible benchmarks that are predominantly English. We investigate Chinese model developers’ priorities through a comparative study of Chinese-developed and Western-developed open-weight LLMs, on 21 language variants including Asian regional, Chinese, and European languages. Our experiments on Information Parity and reading comprehension show Chinese models’ performance across these languages correlates strongly (r=0.93) with their Western counterparts, with the sole exception being better Mandarin. Chinese-developed models are good at French and German, but they sometimes cannot identify languages spoken by Chinese minorities such as Kazakh and Uyghur. Overall, all open-weight LLMs we study have a similar multilingual performance profile, despite the diverse linguistic and cultural contexts the model developers operated within. We interpret the homogenization as consistent with the influence of global benchmarking practices and shared training resources. Rather than treating current language support as inevitable, our results highlight multilingual development as a space of prioritization and trade-offs, with implications for model developers, policymakers, and users. \conffull ( \confshort ), \confdate , \confloc Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3812333 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing Empirical studies in HCI † † cc-license: by 1. Introduction Large language models (LLMs) are prohibitively expensive to develop, pushing model developers to make decisions on resource allocations. For developers located in complex linguistic environments, this means choosing to prioritize some languages over others. In this paper, we study a salient example: China. Domestically, China is a linguistically diverse country with 1.4 billion people speaking Mandarin and hundreds of other languages and dialects, but not English ( Eberhard et al., 2024 ; Erard, 2009 ) . At the same time, for Chinese-developed LLMs to gain global attention, developers are incentivized to optimize for popular evaluation benchmarks that are predominantly English ( DeepSeek-AI, 2025 ) . Therefore, China’s model developers need to navigate the tension between serving a linguistically diverse local population or optimizing for performance on globally visible benchmarks in English. In this paper, we first propose four ways in which Chinese developers could have chosen to prioritize linguistic diversity as informed by China’s history, policy, and geopolitical context. Then, we conduct computational experiments to probe which prioritization options China’s model developers seem to have adopted. Specifically, we compare 6 Chinese- and 4 Western-developed open-weight LLMs’s multilingual performances on 21 language variants, including Mandarin Chinese, Chinese minority languages, Asian regional languages, and European languages. We evaluate multilingual performance using both task-agnostic and task-dependent experiments, measuring Information Parity, machine reading comprehension, and language identification. Given data disparity, we do not expect LLMs to do as well in minority languages, such as Kazakh, as in Mandarin. The question we ask is not whether Chinese-developed LLMs (such as Qwen ) perform similarly across languages. Instead, we ask whether Chinese-developed LLMs are comparatively better than their Western counterparts (such as Llama ) on Mandarin and minority languages such as Kazakh, given the assumption that China’s model developers would have more access to these domestic languages than their Western counterparts. Overall, we find that multilingual performance across the 21 language variants we examine is highly correlated between average Chinese-developed and average Western-developed open-weight LLMs. One notable exception is Mandarin, in which Chinese-developed models perform better than their Western counterparts. Beyond this difference, all popular open models we look at, regardless of origins, correlates highly with each other in terms of multilingual performance. Our findings suggest that China’s model developers are optimizing for popular evaluative benchmarks to gain global visibility on their models. They appear to be making similar decisions on allocating model capacity and resources across languages, resulting in highly similar multilingual performance profiles with their Western counterparts. Additionally, our findings reveal a pattern of homogenization in multilingual performance across the models we study, even when model developers operate in distinct linguistic, political, and cultural contexts. This suggests that there are certain normalizing pressures applied on LLM training via global benchmarking practices and shared resources such as training and evaluation data. Category Language Experiment 1: FLORES+ Experiment 2: Belebele Experiment 3: MC 2 Mandarin Chinese Mandarin (Simplified) ✓ ✓ - Mandarin (Traditional) ✓ ✓ - Chinese Han Dialects (Other) Yue (Cantonese) ✓ - - Chinese Minority Jingpho ✓ ✓ - Lhasa Tibetan ✓ ✓ ✓ Uyghur ✓ - ✓ Mongolian - - ✓ Kazakh - - ✓ Northeast Asian Korean ✓ ✓ - Japanese ✓ ✓ - Southeast Asian Indonesian ✓ ✓ - Lao ✓ ✓ - Burmese ✓ ✓ - Thai ✓ ✓ - Vietnamese ✓ ✓ - Standard Malay ✓ ✓ - European English \star ✓ - French ✓ ✓ - Italian ✓ ✓ - Spanish ✓ ✓ - German ✓ ✓ - Table 1. Languages evaluated in each experiment. In Experiment 1, we evaluate Information Parity (IP) on FLORES+ Machine Translation benchmark dataset ( Goyal et al., 2022 ; NLLB Team et al., 2024 ) . In Experiment 2, we evaluate Machine Reading Comprehension (MRC) on the Belebele Dataset ( Bandarkar et al., 2024 ) . In Experiment 3, we utilize Multilingual Corpus of Minority Languages in China (MC 2 ) dataset to evaluate if models’ could identify Chinese minority languages included in the dataset ( Zhang et al., 2024a ) . We categorize languages used by populations in China that the PRC determines as “minority ethnic group” as “Chinese Minority”. We evaluate the Kazakh in Kazakh Arabic script and Mongolian in Traditional Mongolian script, which are writing systems used in China. \star : English is used as the reference language ( Tsvetkov and Kipnis, 2024 ) . 2. Research Background: Language and AI Policy in China Language Policy in China For thousands of years, central governments in China have used language as a tool to manage a vast multiethnic population that speaks over a hundred languages and dialects. These policies have alternated from being pluralist to assimilationist over the eras. One of the earliest such mandates, in 221 B.C., was the First Emperor of Qin’s program to standard the Chinese script as a move to consolidate central state power after unifying the warring states. More recently, following the 1949 Communist Revolution, the Chinese Communist Party (CCP) launched extensive linguistic campaigns to unite the minority population of over 106 million people who spoke 129 languages among them to build an inclusive Chinese nation ( Mullaney, 2011 ) . 1 1 1 Also known as “Multinational state building” China adopted a more assimilationist approach in the late twentieth century with a monolingual policy (‘‘one nation, one language’’) that