Mythbuster: Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate
Simiao Ren, Xingyu Shen, Yuchen Zhou, Dennis (Tsang) Ng, Ankit Raj · 2026
Read Mythbuster: Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate by Simiao Ren, Xingyu Shen, Yuchen Zhou, Dennis (Tsang) Ng, Ankit Raj (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14210.txt.
Introduction
Ren and colleagues test the circulating claim that Chinese prompts beat English on token efficiency for vibe coding, a claim tied to savings up to 40 percent. On SWE-bench Lite, the Chinese advantage fails to appear. Token costs swing by model: MiniMax-2.7 runs about 1.28 times costlier in Chinese, while GLM-5 runs cheaper. Success rates in Chinese trail English across every model tested, so the authors warn practitioners not to expect savings or gains from switching languages.
Catalog record
| CREATOR | Simiao Ren, Xingyu Shen, Yuchen Zhou, Dennis (Tsang) Ng, Ankit Raj |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33074 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14210
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mythbuster: Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate Simiao Ren, Xingyu Shen, Yuchen Zhou, Dennis (Tsang) Ng, Ankit Raj Abstract A claim has been circulating on social media and practitioner forums that Chinese prompts are more token-efficient than English for LLM coding tasks, potentially reducing costs by up to 40%. This claim has influenced developers to consider switching to Chinese for “vibe coding” to save on API costs. In this paper, we conduct a rigorous empirical study using SWE-bench Lite, a benchmark of software engineering tasks, to evaluate whether this claim of Chinese token efficiency holds up to scrutiny. Our results reveal three key findings: First, the efficiency advantage of Chinese is not observed. Second, token cost varies by model architecture in ways that defy simple assumptions: while MiniMax-2.7 shows 1.28x higher token costs for Chinese, GLM-5 actually consumes fewer tokens with Chinese prompts. Third, and most importantly, we found that the success rate when prompting in Chinese is generally lower than in English across all models we tested. We also measure cost efficiency as expected cost per successful task—jointly accounting for token consumption and task resolution rate. These findings should be interpreted as preliminary evidence rather than a definitive conclusion, given the limited number of models evaluated and the narrow set of benchmarks tested due to resource constraints; they indicate that language effects on token cost are model-dependent, and that practitioners should not expect cost savings or performance gains just by switching their prompt language to Chinese. Keywords: large language models, multilingual reasoning, token efficiency, SWE-bench, vibe coding, code generation Figure 1: Graphical abstract showing the research workflow and key findings. 1 Introduction The deployment of large language models (LLMs) for software engineering tasks has gained significant traction, with models such as GPT-4, Claude, and open-source alternatives demonstrating remarkable capabilities in code generation, bug detection, and automated repair ( 6 ; 4 ) . AI-assisted coding has grown explosively: 84% of developers now use or plan to use AI coding tools ( 14 ) , and AI is estimated to generate approximately 29% of new U.S. software code as of late 2024 ( 15 ) . The “vibe coding” paradigm, where developers use natural language to generate code through LLMs, has become mainstream, with platforms reaching $100M ARR in as little as eight months ( 16 ) and the AI coding tools market projected to reach $37B by 2032 ( 13 ) . As token-based API pricing dominates LLM economics, with output tokens typically priced 4–5 \times higher than input tokens, even modest efficiency gains translate to substantial cost savings at scale. A claim has been circulating on social media and practitioner forums that Chinese prompts are more token-efficient than English for coding tasks, potentially reducing API costs by up to 40% ( 1 ) . This claim has influenced developers to consider switching to Chinese for “vibe coding” to save on token costs. Figure 2: The popular social media claim that Chinese prompts are 40% more token-efficient, motivating this empirical study. This YouTube short has garnered over 2.6 million views and 63k likes, with top comments endorsing the claim. Notably, the original post does not cite any empirical research to support this assertion. The social media claim typically asserts that Chinese is inherently more efficient because Chinese characters are more information-dense than English text. However, this claim conflates three distinct phenomena: (1) information density—the fact that Chinese characters can express ideas with fewer characters at the semantic level; (2) tokenizer allocation—tokenizers optimized for English training data may fragment CJK characters into multiple tokens; and (3) reasoning verbosity—some anecdotal evidence suggests Chinese chain-of-thought reasoning produces shorter output text. Without rigorous empirical validation, the popular efficiency narrative remains an unverified claim that could misguide developers. Recent academic research has begun to clarify these issues. Research on tokenization premiums demonstrates that different languages require different numbers of tokens to express equivalent content depending on tokenizer vocabulary allocation, with English often having the lowest token premium for GPT-style tokenizers while Chinese may require substantially more tokens per character ( 10 ) . Similarly, recent model technical reports describe dedicating expanded tokenizer vocabulary to non-English languages to improve compression, acknowledging that multilingual tokenization efficiency is a deliberate design choice ( 17 ; 5 ) . Despite this emerging understanding, rigorous empirical validation on domain-specific tasks like code debugging remains scarce. Most prior studies have compared English performance against machine-translated Chinese prompts without controlling for confounding factors such as model architecture and token limits. The few multilingual evaluations that exist often focus on general reasoning benchmarks rather than software engineering tasks, where precise technical understanding is critical and where token costs can accumulate substantially over long agentic interactions. In this paper, we conduct a controlled empirical study to test the popular claim that Chinese prompts are more token-efficient than English for code reasoning tasks. Our research question is: Does the choice of prompt language (English vs Chinese) significantly affect token cost and problem-solving rate in LLM-based code reasoning, and if so, which language is actually more efficient? We focus specifically on SWE-bench Lite, selecting 50 instances from its 300 real-world software engineering tasks, each requiring models to understand bug descriptions, locate relevant code, and generate patches that pass test suites. Our contributions are threefold. First, we provide a systematic comparison of English and Chinese prompts across three LLM families (MiniMax-2.7, GPT-5.4-mini, and GLM-5) on identical task instances. Second, we measure token consumption at the API level, distinguishing between prompt, completion, and reasoning tokens. Third, we analyze whether observed performance differences can be attributed to language per se or to confounding factors such as tokenization efficiency. 2 Related Work 2.1 Multilingual LLM Performance Prior work on multilingual LLM evaluation has established that model capability varies significantly across languages, with English typically serving as the best-supported language due to its dominance in training data ( 11 ; 18 ) . Studies on Chinese LLM performance have documented higher token-to-character ratios compared to English, due to the sub-word tokenization challenges inherent in logographic writing systems ( 10 ; 12 ; 7 ) . However, these studies have largely focused on general language understanding tasks rather than code-specific reasoning. Recent research on tokenization premiums demonstrates that token efficiency is tokenizer-dependent: English often has the lowest token premium for GPT-style tokenizers, while Chinese typically requires more tokens per character ( 10 ) . This work contradicts the simplistic narrative that Chinese is inherently token-efficient; instead, efficiency depends on how tokenizer vocabulary was allocated during training. Similarly, recent large language model technical reports explicitly expand tokenizer vocabulary dedicated to non-English languages to address this gap, acknowledging that multilingual compression requires deliberate design investment ( 17 ; 5 ) . 2.2 Code Generation Benchmarks SWE-bench represents a significant advancement in evaluating LLMs on real-world software engineering tasks ( 6 ) . Unlike synthetic benchmarks such as HumanEval ( 4 ) , SWE-bench provides authentic bug reports from popular open-source projects, requiring models to understand natural language descriptions, locate relevant source files, and generate precise patches. Multilingual code generation has been studied on benchmarks such as HumanEval-XL ( 9