Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction

Zhe Zhang Runlin Liu Aishan Liu Xingyu Liu Xiang Gao Hailong Sun Beihang University {zhangzhe2023, liuaishan, xiang_gao, sunhl}@buaa.edu.cn · 2025

Read Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction by Zhe Zhang Runlin Liu Aishan Liu Xingyu Liu Xiang Gao Hailong Sun Beihang University {zhangzhe2023, liuaishan, xiang_gao, sunhl}@buaa.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07180.txt.

Introduction

Zhe Zhang and co-authors build a new way to test code-writing models in a 2025 paper. Static test sets get leaked into training data, and weak tests let wrong code pass. Their answer scales two things at once: fresh problems drawn from live code bases, and tough auto-built tests that must cover every branch.

The first suite holds tasks in Python and Java drawn from real projects. Testing ten top models showed they handle simple API use well but falter at writing fresh algorithms. Language ecosystem shaped scores strongly. Strict testing exposed an illusion of correctness in easier benchmarks. Readers get a harder, fairer yardstick for coding models.

Catalog record

CREATORZhe Zhang Runlin Liu Aishan Liu Xingyu Liu Xiang Gao Hailong Sun Beihang University {zhangzhe2023, liuaishan, xiang_gao, sunhl}@buaa.edu.cn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07180

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction Zhe Zhang Runlin Liu Aishan Liu Xingyu Liu Xiang Gao Hailong Sun Beihang University {zhangzhe2023, liuaishan, xiang_gao, sunhl}@buaa.edu.cn Abstract The evaluation of code-generating Large Language Models (LLMs) is fundamentally constrained by two intertwined challenges: a reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing. This paper introduces a new benchmark construction philosophy, Dual Scaling , designed to systematically address both limitations. Our approach involves continuously scaling the source of problems from dynamic, real-world code repositories and systematically scaling the rigor of tests via automated, high-coverage Property-Based Testing (PBT). We instantiate this philosophy in Code2Bench , an end-to-end framework that leverages Scope Graph analysis for principled dependency classification and a 100% branch coverage quality gate to ensure test suite integrity. Using this framework, we construct Code2Bench-2509 , a new benchmark suite with native instances in both Python and Java. Our extensive evaluation of 10 state-of-the-art LLMs on Code2Bench-2509 , powered by a novel "diagnostic fingerprint" visualization, yields three key insights: (1) models exhibit a fundamental performance gap, excelling at API application (Weakly Self-Contained tasks) but struggling with algorithmic synthesis (Self-Contained tasks); (2) a model’s performance is profoundly shaped by the target language’s ecosystem, a nuance we are the first to systematically quantify; and (3) our rigorous, scaled testing is critical in uncovering an "illusion of correctness" prevalent in simpler benchmarks. Our work presents a robust, scalable, and diagnostic paradigm for the next generation of LLM evaluation in software engineering. The code, data, and results are available at https://code2bench.github.io/ . 1 Introduction As Large Language Models (LLMs) are increasingly integrated into software development workflows Jimenez et al. (2023) ; 3 ; 1 , the need for accurate and realistic evaluation of their coding capabilities has become paramount. However, the current landscape of code benchmarks is fundamentally constrained by two intertwined challenges: a reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing . First, ❶ the static nature of canonical benchmarks like HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) leads to an inevitable obsolescence; their problems, having existed for years, are likely part of LLM training corpora, turning evaluation into an exercise in memorization rather than true generalization Carlini et al. (2021) ; Sainz et al. (2023) . While dynamic “live” benchmarks Jain et al. (2024a) ; Li et al. (2024b) have emerged, they often source problems from competitive programming, which may not reflect the complexity of real-world software engineering. Second, ❷ the superficial testing common to most benchmarks, often relying on a handful of example-based tests, creates an illusion of correctness. As highlighted by EvalPlus Liu et al. (2023a) , this insufficient test rigor fundamentally limits their ability to uncover the subtle, edge-case failures that define the gap between functional code and production-ready software. As summarized in Table 1 , existing methods fall short across key dimensions, highlighting the significant limitations remaining and the necessity to design new benchmarks. To break this cycle of obsolescence and superficiality, we argue that a paradigm shift is needed. We propose a new benchmark construction philosophy centered on two core principles: (1) Scaling the Source , by dynamically and continuously ingesting a diverse array of problems from the ever-evolving landscape of real-world code repositories; and (2) Scaling the Rigor , by systematically generating comprehensive test suites with deep, verifiable coverage through Property-Based Testing (PBT) Claessen and Hughes (2000) . We instantiate this philosophy in CODE2BENCH , a novel, end-to-end framework that automates this dual-scaling process. ❶ To Scale the Source , CODE2BENCH first addresses the challenge of classifying diverse, real-world code. Our analysis of the existing benchmark landscape reveals an implicit bifurcation in evaluation focus, which we formalize into two primary task categories: (1) Self-Contained (SC) tasks, which require pure, dependency-free logic, reflecting the focus of benchmarks like HumanEval Chen et al. (2021) on core algorithmic reasoning ; and (2) Weakly Self-Contained (WSC) tasks, which require the correct application of common libraries, capturing the focus of benchmarks like BigCodeBench Zhuo et al. (2024) on practical API application . This principled classification, enabled by our Scope Graph-based analysis, allows us to systematically generate tasks that target these distinct developer skills. ❷ To Scale the Rigor , CODE2BENCH then employs a powerful Property-Based Testing (PBT) engine and a stringent 100% branch coverage quality gate . This ensures that every problem in our benchmark is not only realistic but also a fully-explorable logical challenge, backed by a test suite capable of deep, diagnostic validation. Using this framework, we construct CODE2BENCH-2509 , a new, multi-faceted benchmark suite with native instances in Python and Java, curated from recent, real-world repositories. Our extensive evaluation of 10 state-of-the-art LLMs on this suite demonstrates the power of our approach. The synergy of a scaled source and scaled testing enables an unprecedentedly fine-grained diagnostic, revealing: (1) a performance gap between models’ ability in algorithmic synthesis (SC) and API application (WSC); (2) the profound impact of language paradigms on model failure modes, a nuance we are the first to systematically quantify; and (3) the critical role of rigorous testing in uncovering the “illusion of correctness” prevalent in simpler benchmarks. We summarize our contributions as follows: • We propose CODE2BENCH , a novel framework that introduces and operationalizes the Dual Scaling philosophy for benchmark construction, systematically scaling the source of problems with dynamic acquisition and the rigor of tests with a 100% coverage PBT quality gate. • We construct and release CODE2BENCH-2509 , a high-quality, contamination-resistant benchmark with native tasks in Python and Java, demonstrating significantly higher complexity and test rigor than prior work (Table 1 ). • We provide a deep, diagnostic analysis of state-of-the-art LLMs, introducing novel visualizations and uncovering key insights into their strengths and weaknesses in real-world coding scenarios. Table 1: Comparison of Code2Bench-2509 with existing code generation benchmarks. Benchmark Source Dynamic Deps Handled Rigorous Test Multi-lang Design HumanEval Chen et al. (2021) Manual ✗ ✗ ✗ ✗ MBPP Austin et al. (2021) Manual ✗ ✗ ✗ ✗ EvalPlus Liu et al. (2023a) Manual ✗ ✗ ✓ ✗ LiveCodeBench Jain et al. (2024a) Contests ✓ ✗ ✓ ✓ RepoBench Liu et al. (2023b) Project Codebases ✗ ✓ ✗ ✗ HumanEval-X Zheng et al. (2023b) Manual ✗ ✗ ✗ ✓ BigCodeBench Zhuo et al. (2024) Synthetic ✗ ✓ ✗ ✗ DevEval Li et al. (2024c) Project Codebases ✗ ✓ ✗ ✗ EvoCodeBench Li et al. (2024b) Project Codebases ✓ ✓ ✗ ✗ Code2Bench-2509 (Ours) Project Codebases ✓ ✓ ✓ ✓ 2 The Code2Bench Framework: A Dual Scaling Approach Figure 1: Overview of the Code2Bench Framework. In this section, we detail the architecture and technical components of the Code2Bench framework. Our methodology is built upon the core philosophy of Dual Scaling: continuously scaling the source of benchmark problems to ensure realism and novelty, and systematically scaling the rigor of our tests to enable deep, diagnostic evaluation. We organize our discussion around these two principles. 2.1 Scaling the Source: Dynamic Acquisition from Real-World Code Temporal Filtering for Contamination Resistance. The primary threat to the validity of LLM evaluation is data contamination, where benchmarks become obsolete as their contents are absorbed into training corpora.

Read the complete text