CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang Yong Yu · 2025
Read CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark by Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang Yong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05281.txt.
Introduction
Lingyue Fu and co-authors built a new coding test in 2025 that splits programming skill into small, clear parts. Whole-repo tasks are now common, but a single pass rate hides which kind of thinking failed. Fixed tests also grow stale as models may have seen the tasks or scores top out.
Their pipeline turns Python repos into tasks that each isolate one kind of demand within the same code. Difficulty can be raised in a controlled way. The yield of valid tasks was far higher than a known verified benchmark. Tests of leading models showed ranks shifting across kinds of demand. Skill in one area did not ensure skill in another, the team reports.
Catalog record
| CREATOR | Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang Yong Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05281
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Jun Xu, Zongyu Wang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang Yong Yu Abstract As Large Language Models (LLMs) demonstrate increasingly sophisticated code processing capabilities, evaluating their performance on engineering-level code remains challenging. Existing repository-level benchmarks primarily focus on single scenarios, such as code generation or bug fixing, without adequately capturing the diversity and complexity of real-world software or project engineering workflows. Furthermore, these benchmarks suffer from limited controllability in question positioning and reliability issues in their generated test cases. To address these limitations, we present CorePipe, a fully automated pipeline that converts repositories into comprehensive test cases, and introduce CoreCodeBench , a configurable multi-scenario repository-level benchmark. To simulate real engineering scenarios, CorePipe generates three types of atomic questions (Development, BugFix, and Test-Driven Development) specifically targeting core code segments. These atomic questions are further combined into three types of composite questions, with difficulty levels flexibly adjusted through hyperparameter tuning. CoreCodeBench provides a comprehensive and extensive repository-level benchmark to investigate the applicability of LLMs in real-world engineering projects. Experiments with 16 LLMs across diverse scenarios reveal varying capabilities and offer multi-dimensional insights into LLM performance in engineering contexts. Code of CorePipe 1 1 1 https://github.com/AGI-Eval-Official/CoreCodeBench and data of CoreCodeBench 2 2 2 https://huggingface.co/collections/tubehhh/corecodebench-68256d2faabf4b1610a08caa are available. 1 Introduction With the continuous improvement in the code processing capabilities of Large Language Models (LLMs), more researchers are starting to focus on their applications in engineering-level code. Engineering-level code often involves complex dependencies and long-context interactions, posing unique challenges for LLMs. Specialized code LLMs, such as QwenCoder [ 16 ] and DeepSeekCoder [ 13 ] , have demonstrated exceptional programming capabilities in software engineering. LLM-based products such as Copilot, Windsurf, and Cursor, significantly reduce the complexity programmers face in engineering-level projects. As the code processing capabilities of LLMs continue to evolve, there is a growing need to systematically understand their strengths and limitations across different engineering scenarios. To assess the programming capabilities of these tools at an engineering level, it is crucial to establish an effective and fair evaluation standard. Several benchmarks have been proposed, such as SWEBench [ 17 ] , REPOEXEC [ 14 ] , and BigCodeBench [ 38 ] , to evaluate the ability of LLMs in implementing engineering-level code. These benchmarks are derived and refined from real-world repositories, ensuring a high degree of alignment with real engineering code development. They focus on tasks such as natural language to code (NL2Code) translation and bug fix within the scope of engineering code development. Although existing benchmarks provide an initial reference for evaluating the programming capabilities of LLMs in engineering environments, the current evaluation framework faces two critical challenges. Figure 1: Overview of CoreCodeBench . Challenge 1: Single Scenario. Prevailing repository-level benchmarks primarily focus on the code generation task, and do not adequately encompass the diverse scenarios present in engineering development. In real-world engineering practice, developers not only need to complete function-level code completion but also engage in bug fixies for unit tests. Additionally, within modular development paradigms, engineers often need to simultaneously implement main functions alongside supporting utility functions. These scenarios require the LLMs to display not only code generation capabilities but also cross-file contextual reasoning and implementation planning abilities—skills that current evaluation systems fail to systematically assess. Challenge 2: Lack of Controllability and Reliability. Existing automated generation methods exhibit significant shortcomings in both controlling the positioning of generated questions and ensuring their reliability, directly impacting benchmark’s effectiveness. The random masking approach, while achieving positional randomness, lacks logical constraints in mask selection, which might result in overlooking critical code segments or excessively testing non-essential areas [ 37 ] . Alternative approaches such as those based on cleaning pull requests, fix testing locations to historical revision points, limiting evaluation scenario diversity [ 17 , 29 ] . These methods also suffer from low data reliability, with numerous pull requests not being self-contained and requiring substantial manual cleaning [ 28 ] . Neither method effectively ensures flexible control over test positioning while maintaining core code relevance and data quality, hindering comprehensive assessment of LLMs’ performance in engineering-level tasks. To address these limitations, we design a fully automated pipeline CorePipe that converts GitHub repositories into repository-level benchmark test cases. CorePipe generates three types of atomic questions (Development, Bugfix, Test-Driven Development) on core code segments, and further composes multiple composite question types with adjustable difficulty. Quality inspection and analysis show that the generated data are of high quality and reliability. As shown in Figure 1 , we release a meticulously Co nfigurable Re pository-level benchmark, CoreCodeBench , which effectively evaluates the actual capabilities and adaptability of LLMs in engineering-level code development. Through comprehensive evaluation of general-purpose and code-specific LLMs, we gain insights into the performance and characteristics of these models across diverse repository-level scenarios. CoreCodeBench not only enables coarse-grained differentiation of LLM code abilities, but also provides fine-grained analysis of their potential. Flexible control of CorePipe over question difficulty enables CoreCodeBench to offer a promising platform for future LLM evaluation. Our experiments further highlight several areas for improvement in LLMs’ performance on engineering-level projects, paving the way for future advancements in model capabilities. The contributions are summarized as follows: • We design CorePipe, a fully automated pipeline for generating LLM engineering code capability tests from repository source code without any human intervention. CorePipe can be adapted to any programming language and any repository. • We release the analysis and quality inspection results of the test data generated by CorePipe. The results demonstrate that CorePipe can produce high-quality and highly flexible test cases. • We provide CoreCodeBench , a repository-level benchmark that includes three atomic tasks and three composite tasks. CoreCodeBench features various question types and characteristics, offering new insights and analytical perspectives for evaluating LLM coding. • We present the evaluation results on several state-of-the-art LLMs and conduct multifaceted analyses of their performance on repository-level scenarios. 2 Background and Related Work Table 1: Comparison between existing repository-level benchmarks and CoreCodeBench . Benchmark Multi-Task Automatic Difficulty Level Flexible Position Quality Inspection Avg. Lines SWEBench [ 17 ] ✗ ✓ ✗ ✗ ✗ 38.01 DevBench [ 19 ] ✓ ✗ ✗ ✗ ✗ - ExecRepoBench [ 37 ] ✗ ✓ ✗ ✗ ✗ 2.42 Codev-Bench [ 29 ] ✗ ✓ ✗ ✗ ✗ 43.69 EvoCodeBench [ 20 ] ✗ ✗ ✗ ✗ ✗ 14.86 RepoMasterEval [ 34 ] ✗ ✓ ✗ ✗ ✗ - BigCodeBench [ 38 ] ✗ ✗ ✗ ✗ ✗ 13.55 REPOEXEC [ 14 ] ✗ ✓ ✗ ✗ ✓ 21.9 CoreCodeBench ✓ ✓ ✓ ✓ ✓ 34.14 2.1 Large Language Models for Code General-purpose LLMs have demonstrated remarkable performance not only in natural language processing but also in code-related tasks. In recent years, LLMs tailored for code generation and reasoning have consistently achieved high scores in benchmark tests. On the HumanEval benchmark [ 7 ] , the closed-source models Claude-3.5-Sonnet [ 3 ] and GPT-4o-0513 [ 24 ] have reached Pass@1 scores of 92.0% and 91.0%, respectively. Among open-source models, DeepSeek-Coder-V2-Instruct [ 9 ] and Qwen2.5-Coder-Instruct [ 16 ] have achieved Pass@1 scores of 90.2% and 88.4%. On other algorithmic problem benchmarks like MBPP [ 6