EffiBench-X : A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
Yuhao Qing Boyu Zhu Mingzhe Du Zhijiang Guo Terry Yue Zhuo Qianru Zhang Jie M. Zhang, Heming Cui Siu-Ming Yiu Dong Huang See-Kiong Ng, Luu Anh Tuan · 2025
Read EffiBench-X : A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code by Yuhao Qing Boyu Zhu Mingzhe Du Zhijiang Guo Terry Yue Zhuo Qianru Zhang Jie M. Zhang, Heming Cui Siu-Ming Yiu Dong Huang See-Kiong Ng, Luu Anh Tuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13004.txt.
Introduction
Code made by chat models is most often judged on whether it works, not on whether it runs fast or lean. Yuhao Qing and co-authors fix that gap in 2025 with a test set that grades speed in six code tongues. Human expert fixes set the bar for each task.
Top models wrote code that worked, yet fell well short of the human bar on speed. The best model hit about three fifths of human speed on the mean. Models did better in Python, Ruby, and one web tongue than in Java or C plus plus. One strong model wrote far leaner Python than Java. The team says speed training for code models is a clear open need.
Catalog record
| CREATOR | Yuhao Qing Boyu Zhu Mingzhe Du Zhijiang Guo Terry Yue Zhuo Qianru Zhang Jie M. Zhang, Heming Cui Siu-Ming Yiu Dong Huang See-Kiong Ng, Luu Anh Tuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13004
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EffiBench-X : A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code Yuhao Qing Boyu Zhu Mingzhe Du Zhijiang Guo Terry Yue Zhuo Qianru Zhang Jie M. Zhang, Heming Cui Siu-Ming Yiu Dong Huang See-Kiong Ng, Luu Anh Tuan Abstract Existing code generation benchmarks primarily evaluate functional correctness, with limited focus on code efficiency and often restricted to a single language like Python. To address this gap, we introduce EffiBench-X , the first multi-language benchmark designed to measure the efficiency of LLM-generated code. EffiBench-X supports Python, C++, Java, JavaScript, Ruby, and Golang. It comprises competitive programming tasks with human-expert solutions as efficiency baselines. Evaluating state-of-the-art LLMs on EffiBench-X reveals that while models generate functionally correct code, they consistently underperform human experts in efficiency. Even the most efficient LLM-generated solutions (Qwen3-32B) achieve only around 62% of human efficiency on average, with significant language-specific variations. LLMs show better efficiency in Python, Ruby, and JavaScript than in Java, C++, and Golang. For instance, DeepSeek-R1’s Python code is significantly more efficient than its Java code. These results highlight the critical need for research into LLM optimization techniques to improve code efficiency across diverse languages. The dataset and evaluation infrastructure are submitted and available at https://github.com/EffiBench/EffiBench-X.git and https://huggingface.co/datasets/EffiBench/effibench-x . $*$ $*$ footnotetext: Equal Contribution. $\dagger$ $\dagger$ footnotetext: Corresponding Authors. 1 Introduction Code efficiency is becoming a critical concern for LLM-generated code as these models are more widely adopted. While many LLMs and agent frameworks successfully produce correct solutions ( 16 , 36 , 75 , 39 , 30 ) , their outputs often incur substantial resource overhead, compromising performance and feasibility in settings demanding high efficiency, such as mobile phones, embedded systems, and latency-sensitive cloud environments ( 48 , 32 , 66 , 31 , 33 ) . This underscores the need for benchmarks that measure not just correctness but also runtime efficiency ( 56 , 66 , 48 ) . Responding to this critical need, several benchmarks have recently emerged to measure the efficiency of LLM-generated code. EffiBench ( 32 ) and Mercury ( 23 ) use LeetCode problems and their solutions to evaluate Python code based on runtime and memory. Moving beyond LeetCode, EvalPerf ( 48 ) and ENAMEL ( 56 ) assess efficiency using subsets of existing benchmarks HumanEval ( 16 ) and MBPP ( 11 ) , while PIE ( 62 ) benchmarks efficiency by compiling performance-improving edits across various CodeNet challenges. Despite these valuable contributions, a closer examination reveals limitations in current benchmarks, which hinder their effectiveness in accurately evaluating efficiency. Specifically, current code efficiency benchmarks exhibit three key limitations. Firstly, they suffer from Language Diversity issues, primarily focusing on single-language measurement ( 32 , 23 , 66 , 56 , 48 ) , often Python, despite its relatively small share (25%) of the overall programming landscape † †\dagger † †\dagger \dagger https://www.tiobe.com/tiobe-index/ . This narrow focus overlooks crucial language-specific factors like compiler optimizations and memory management prevalent in languages like C++, Java, and Golang, necessitating a multi-language benchmark. Secondly, Data Contamination is a significant problem, as many benchmarks, such as EffiBench and Mercury (using LeetCode), and ENAMEL and EvalPerf (using HumanEval and MBPP), rely on dated and widely circulated problem sets. These have often been “seen” by models during training ( 12 , 40 , 71 , 22 ) , leading to performance metrics that reflect memorization rather than genuine reasoning or optimization, thereby reducing their representativeness for evaluating performance on novel challenges. Lastly, current benchmarks are hampered by Limited Complexity , featuring tasks that are often straightforward and solvable without advanced algorithms. Benchmarks like HumanEval and MBPP, used by EvalPerf and ENAMEL, include simple prompts where models have already achieved high pass rates (e.g., >95% pass@1 on HumanEval) † †\dagger † †\dagger \dagger HumanEval Leaderboard: https://paperswithcode.com/sota/code-generation-on-humaneval . . Such trivial tasks fail to reveal critical performance differentials or require substantial computational resources, making them unsuitable for measuring efficiency and highlighting the need for benchmarks incorporating more complex, computationally intensive problems to better assess an LLM’s ability to handle large inputs or implement optimized solutions for real-world applications. To address critical limitations in evaluating LLM code efficiency, we introduce EffiBench-X , the first multi-language benchmark specifically designed for this purpose. EffiBench-X evaluates efficiency across six diverse programming languages: Python, C++, Java, JavaScript, Ruby, and Golang, directly tackling the language diversity problem. It comprises recently released competitive programming tasks from various platforms, paired with canonical human expert solutions, effectively mitigating data contamination. By focusing on complex problems requiring advanced algorithms and data structures, EffiBench-X overcomes the issue of limited complexity, providing a more accurate assessment of LLMs’ efficiency in challenging scenarios. Furthermore, our comprehensive evaluation framework with high-resolution profiling ensures robust and reliable measurement of LLM-generated code efficiency within a controlled environment. Leveraging EffiBench-X and its comprehensive evaluation framework, we conducted a comprehensive study evaluating the performance of a wide range of LLMs against expert-written baselines. Our experiments revealed that while most LLMs are capable of generating functionally correct solutions, they consistently exhibit shortcomings in efficiency. Specifically, even the most efficient code produced by an LLM achieved only approximately 62% of the runtime performance demonstrated by expert-crafted code. These findings highlight a significant gap and underscore the critical need for further research into optimization techniques. Such advancements are essential to empower LLMs to generate code that is not only correct but also highly efficient across diverse programming languages. 2 Related Work Early efforts on code generation with LLMs gravitated toward assessing correctness and functionality, exemplified by HumanEval ( 16 ) and MBPP ( 11 ) , which challenge models to produce valid code snippets from natural language docstrings. Building on this foundation, subsequent works have aimed to mitigate issues of limited diversity in tasks. For instance, HumanEval-X ( 74 ) , MultiPLe ( 14 ) , and MBXP ( 10 ) expand these benchmarks to multiple programming languages, while DS-1000 ( 42 ) , ARCADE ( 72 ) , and NumpyEval ( 73 ) target data science–oriented tasks. Beyond these, efforts like APIBench ( 55 ) , BigCodeBench ( 75 ) , BiasBench ( 34 ) , and RepoBench ( 49 ) delve into broader software engineering subtasks, including library usage and repository-level code completion. While these benchmarks have been instrumental in gauging a model’s correctness, they rarely address performance, potentially overlooking inefficiencies that hinder real-world adoption. Recognizing this significant limitation, a new wave of evaluations has emerged. As shown in Table 1 ), benchmarks such as EffiBench ( 32 ) , Mercury ( 23 ) , EvalPerf ( 48 ) , ENAMEL ( 56 ) , and PIE ( 62 ) specifically target performance metrics like runtime and memory usage. While these efforts are valuable, they commonly face limitations. They are largely confined to single-language settings, predominantly Python. Furthermore, their reliance on popular tasks (e.g., HumanEval, MBPP) increases the risk of data contamination from LLM training data. Moreover, the often-simple nature of these tasks does not adequately expose nuanced performance differences critical in complex scenarios. To overcome these shortcomings, EffiBench-X offers a multi-language evaluation across Python, C++, Java, JavaScript, Ruby, and Golang . By pairing solutions with expert-optimized versions from competitive programming sites and utilizing diverse, complex tasks, EffiBench-X allows