CodeElo : Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
Shanghaoran Quan Jiaxi Yang Bowen Yu Bo Zheng Dayiheng Liu An YangXuancheng Ren Bofei Gao Yibo Miao Yunlong Feng Zekun Wang, Jian Yang Zeyu Cui Yang Fan Yichang Zhang Binyuan Hui Junyang Lin Qwen Team, Alibaba Group {quanshanghaoran,binyuan.hby,junyang.ljy}@alibaba-inc.com · 2025
Read CodeElo : Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings by Shanghaoran Quan Jiaxi Yang Bowen Yu Bo Zheng Dayiheng Liu An YangXuancheng Ren Bofei Gao Yibo Miao Yunlong Feng Zekun Wang, Jian Yang Zeyu Cui Yang Fan Yichang Zhang Binyuan Hui Junyang Lin Qwen Team, Alibaba Group {quanshanghaoran,binyuan.hby,junyang.ljy}@alibaba-inc.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-01257.txt.
Introduction
A large team from Alibaba's Qwen group ranks language models on contest programming using the same Elo scale as human competitors, in this 2025 paper. Existing code benchmarks lack private tests and faithful judging, so scores drift from real contest skill. Their benchmark, CodeElo, is built directly on the Codeforces platform.
Problems come from six months of real contests, with difficulty ratings and topic tags attached. Solutions are submitted to the platform itself, and an Elo system calibrated to human standings scores each model. Thirty three models were rated. Two reasoning models stood far ahead, while most others landed in the lowest quarter of human participants, failing even easy problems. Breakdowns by algorithm type and by language choice, C++ versus Python, point to where models must improve.
Catalog record
| CREATOR | Shanghaoran Quan Jiaxi Yang Bowen Yu Bo Zheng Dayiheng Liu An YangXuancheng Ren Bofei Gao Yibo Miao Yunlong Feng Zekun Wang, Jian Yang Zeyu Cui Yang Fan Yichang Zhang Binyuan Hui Junyang Lin Qwen Team, Alibaba Group {quanshanghaoran,binyuan.hby,junyang.ljy}@alibaba-inc.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-01257
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CodeElo : Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings Shanghaoran Quan Jiaxi Yang Bowen Yu Bo Zheng Dayiheng Liu An YangXuancheng Ren Bofei Gao Yibo Miao Yunlong Feng Zekun Wang, Jian Yang Zeyu Cui Yang Fan Yichang Zhang Binyuan Hui Junyang Lin Qwen Team, Alibaba Group {quanshanghaoran,binyuan.hby,junyang.ljy}@alibaba-inc.com Abstract With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo , a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces 1 1 1 https://codeforces.com platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo , we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies. Figure 1: The Elo rating leaderboard. The test results may be slightly lower than the actual capabilities, as we constrain the number of submissions for each problem to eight times. The green dashed lines represent the Elo ratings of human participants at the corresponding percentiles. 🖂 🖂 footnotetext: Corresponding authors 1 Introduction With the increasing capabilities of existing LLMs and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop benchmarks that effectively test their sophisticated reasoning abilities. Math and coding are two evaluation methods for this purpose, as they provide accurate and easily verifiable feedback. While math presents hard benchmarks, like AIME ( MAA 2024 ) , Omni-MATH ( Gao et al. 2024 ) , and LiveAoPSBench ( Anonymous 2024 ) , there’s still a lack of a suitable benchmark in coding to appropriately measure LLMs’ reasoning skills. We find that the CodeForces platform is suitable, and in the reports of OpenAI o1 ( OpenAI 2024b ) , o3 ( OpenAI 2024c ) and DeepSeek r1 ( DeepSeek 2024 ) , they all test hard code on CodeForces. However, we still lack a standardized CodeForces test suite, leading to repeated compiling and creating work on CodeForces and potential misaligned settings and results. These situations highlight the need for research community to standardize a CodeForces evaluation benchmark to assess the competition-level coding capabilities of LLMs and reflect their corresponding sophisticated reasoning abilities. We analyze that existing competition-level code benchmarks like LiveCodeBench ( Jain et al. 2024 ) , USACO ( Shi et al. 2024 ) , and CodeContests ( Li et al. 2022 ) cannot meet the above requirement due to certain limitations attributed to the following unique nature of competition code evaluation. (1) Firstly, unlike typical math and general coding problems, competition code problem often requires extensive, human-designed, robust test cases to validate solution correctness. Although it is easy to scrape any problems from the websites, the competition code platforms or so-called online judges often hide their test cases. Consequently, existing benchmarks can only rely on publicly available or self-generated test cases that are often small and weak, leading to high false-positive rates in judgment 2 2 2 Passing all tests in CodeContests might still result in a wrong answer (~4%) or a time limit error (~42%), according to https://alphacode.deepmind.com/ . (2) Secondly, to truly evaluate LLM capabilities and provide human-comparable Elo ratings, it’s necessary to test on all problems in contests just as human participants do. However, not all problems can be judged by directly comparing the output with the correct answer, as about 30% of problems do not have unique correct outputs, which require specific judging codes called special judges 3 3 3 Special judges are programs used to determine whether solutions are accepted for problems that do not have a unique correct answer. A detailed discussion is provided in Appendix F . to evaluate correctness. However, these special judges are often unavailable and difficult to write for some specific problems. (3) Thirdly, different from general code testing, execution time is a significantly critical factor in competitive coding since problems often have strict constraints on running times and algorithm time complexity, but offline testing faces challenges due to varied efficiency across different machines, leading to potential misaligned evaluation results. Hence, there remains a lack of comprehensive standardized benchmarks for competition-level coding. In this work, for the first time, we present the benchmark CodeElo and the corresponding evaluation method, which achieves zero false positives, supports special judges, and reaches an aligned execution environment to ensure absolutely correct judgments and provide human-comparable, standardized Elo ratings. We have compiled our test problems from CodeForces and categorized them by contest divisions, problem difficulty ratings, and problem algorithm tags for more comprehensive evaluation and analysis. To achieve absolutely correct judgments, we introduce a simple and effective method: using a bot to automatically submit model solutions to CodeForces and receive test results. Thanks to the judgment from the platform, this allows us to evaluate problems just like human participants, achieving zero false positives without needing access to the full test set, where test cases are often created adversarially and are robust. Similarly, given the favor of the platform judgments, we support special judges, which former benchmarks did not provide, and the execution environment is absolutely aligned since all use the same platform, even aligned with human participants. Based on the evaluation results and available user data on CodeForces, we also introduce an Elo rating calculation system that estimates the models’ expected Elo rating based on their performance. This system is aligned with the CodeForces platform but has a much lower variance, as detailed in Section 3.3.2 . We tested 30 existing popular open-source LLMs and 3 proprietary LLMs, with the leaderboard shown in Figure 1 . From the results, we find the OpenAI o1-mini model stands out by achieving the best performance with an Elo rating of 1578, surpassing nearly 90 percent of human participants. The o1-like reasoning model, QwQ-32B-Preview, also achieves great performance and stands out among open-source models, with an Elo rating of 1261, placing it in about the 60th percentile. This suggests that increasing the length of the chain-of-thought (CoT) is a promising way to improve the models’ reasoning ability. On the other hand, most