ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu · 2026
Read ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation by Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26923.txt.
Introduction
Chen and six colleagues introduce ClassEval-Pro, a benchmark of 300 class-level coding tasks across 11 domains, built by an automated three-stage pipeline from real GitHub code. Each task passes test suites with over 90 percent line coverage and validation by a language-model judge panel. The best of five frontier models scores only 45.6 percent on first-try class completion, with about 18 points separating it from the weakest. Failures split mainly into logic errors and dependency errors, marking cross-method coordination as the core bottleneck.
Catalog record
| CREATOR | Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49217 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation Yeheng Chen, Chaoxiang Xie, Yuling Shi, Wenhao Zeng, Yongpan Wang, Hongyu Zhang, Xiaodong Gu Abstract . LLMs have achieved strong results on both function-level code synthesis and repository-level code modification, yet a capability that falls between these two extremes— compositional code creation , i.e., building a complete, internally structured class from a specification—remains underserved. Current evaluations are either confined to isolated functions or rely on manually curated class-level tasks that are expensive to scale and increasingly susceptible to data contamination. We introduce ClassEval-Pro, a benchmark of 300 class-level tasks spanning 11 domains, constructed through an automated three-stage pipeline that combines complexity enhancement, cross-domain class composition, and integration of real-world GitHub code contributed after January 2025. Every task is validated by an LLM Judge Ensemble and must pass test suites with over 90% line coverage. We evaluate five frontier LLMs under five generation strategies. The best model achieves only 45.6% class-level Pass@1, with a 17.7-point gap between the strongest and weakest models, confirming the benchmark’s discriminative power. Strategy choice strongly interacts with model capability: structured approaches such as bottom-up improve weaker models by up to 9.4 percentage points, while compositional generation collapses to as low as 1.3%. Error analysis over 500 manually annotated failures reveals that logic errors (56.2%) and dependency errors (38.0%) dominate, identifying cross-method coordination as the core bottleneck. 1. Introduction LLMs now excel at two distinct modes of code generation. On function-level benchmarks, frontier models surpass 90% Pass@1 on HumanEval ( 2 ) and MBPP ( 1 ) , approaching human-level accuracy on isolated programming tasks. On repository-level benchmarks, coding agents resolve over 75% of real-world GitHub issues on SWE-bench Verified ( 18 ) . Yet real software development also demands a capability that falls between these two extremes: building a complete, internally structured class from a specification—coordinating multiple methods, managing shared state, and integrating logic across domains ( 13 ; 27 ) . Current function-level benchmarks (HumanEval, MBPP, BigCodeBench ( 51 ) ) test isolated logic synthesis, while repository-level benchmarks (SWE-bench, DevEval ( 23 ) ) test code modification within existing codebases; neither evaluates whether models can architect a coherent multi-method artifact from scratch. ClassEval ( 7 ) pioneered class-level evaluation, yet its reliance on manual curation (500 person-hours for 100 tasks across 7 domains) makes it difficult to scale, and its 2023-vintage data is increasingly exposed to training-set contamination ( 29 ; 8 ) . We refer to this underserved evaluation dimension as compositional code creation . To address this gap, we introduce ClassEval-Pro, a benchmark of 300 class-level tasks spanning 11 domains, designed to evaluate compositional code creation. Three design choices distinguish it from prior work. (1) Cross-domain composition —tasks require integrating heterogeneous domain logic (e.g., configuration management + financial computation) into a single coherent class, testing compositional reasoning that single-domain tasks cannot capture. (2) Automated, contamination-resistant construction —an LLM Judge Ensemble pipeline ( 11 ) replaces manual curation, sourcing from GitHub repositories contributed after January 2025 ( 50 ) and employing multi-model collaboration for skeleton generation, test alignment, and reference implementation. (3) Strict validation —every task is verified against automatically generated test suites that achieve over 90% line coverage, ensuring that both the specification and the reference solution are functionally correct. We evaluate five frontier LLMs—GPT-5.1, Gemini-2.5-Pro, Qwen3-480B, Qwen3-30B, and Kimi-K2—under five generation strategies (holistic, incremental, compositional, top-down, and bottom-up) and manually annotate 500 failed cases. Class-level Pass@1 ranges from 27.9% to 45.6% (17.7-point gap), far wider than on function-level benchmarks, confirming that compositional tasks discriminate frontier models effectively. Strategy choice strongly interacts with model capability: structured decomposition (bottom-up) improves weaker models by up to 9.4 percentage points, while compositional generation collapses to as low as 1.3%. Error analysis reveals that logic errors (56.2%) and dependency errors (38.0%) dominate failures, identifying cross-method coordination—not surface-level code formation—as the core bottleneck in class-level code generation. Our major contributions are as follows: • We introduce ClassEval-Pro , a benchmark of 300 class-level tasks across 11 domains for evaluating compositional code creation . It moves beyond isolated function synthesis by requiring models to generate coherent multi-method classes that integrate shared state, inter-method dependencies, interface constraints, and domain-specific logic. • We develop an automated and contamination-resistant construction pipeline for scalable class-level benchmark generation. The pipeline integrates complexity enhancement, cross-domain class composition, post-2025 GitHub code, and LLM Judge Ensembles to ensure task quality, feasibility, and reduced exposure to pretraining contamination. • We present diagnostic findings that characterise the strengths and limitations of current LLMs on class-level code generation. Through strategy–model interaction analysis and a fine-grained error taxonomy, we identify recurring bottlenecks that are not visible from pass/fail metrics alone, providing guidance for future model and agent design. 2. Related Work 2.1. Large Language Models for Code Generation Large Language Models (LLMs) have significantly advanced code generation ( 51 ; 36 ) , repair ( 34 ; 3 ) , translation ( 19 ; 15 ) , and reasoning ( 10 ; 47 ) . This progress is driven by both open-weight models (e.g., DeepSeek-Coder ( 12 ) , Qwen3-Coder ( 45 ) ) and proprietary models (e.g., GPT-5 ( 37 ) , Gemini 3 ( 5 ) ). Recent work further extends LLM capabilities to code understanding through vision–language models ( 35 ) and long-context compression for code ( 33 ) , as well as vulnerability detection ( 16 ) . Despite these advances, current benchmarks predominantly evaluate atomic function-level tasks or repository-level modifications. Despite object-oriented programming dominating real-world software ( 27 ) , evaluations for compositional class-level creation remain limited. Recent studies further extend LLM-based code generation from isolated functions to larger software units. Repository-level code generation requires models to reason over dependencies across functions, classes, and modules, and context inlining has been proposed to expose both upstream and downstream dependency information during generation ( 14 ) . Similarly, project-level code translation studies show that skeleton-guided generation can help preserve compilability and inter-module dependencies when translating complete C projects to Rust ( 38 ) . These works motivate evaluations that stress dependency management and structural consistency beyond isolated function synthesis. 2.2. Benchmarks for Code Generation Early code generation benchmarks target function-level synthesis (e.g., HumanEval ( 2 ) , MBPP ( 1 ) ). While subsequent works enhance evaluation rigor ( 25 ) , API diversity ( 51 ) , and data freshness ( 17 ) , they fundamentally fail to capture the structural complexity of real-world software ( 13 ) . To address this, recent evaluations diverge into two trajectories: repository-level modification and compositional generation. Modification benchmarks evaluate issue resolution within existing codebases (e.g., SWE-bench ( 18 ) , DevEval ( 23 ) ) or utilize repository context ( 48 ) . More recent repository- and project-level benchmarks further extend this direction by considering refactoring tasks, agentic project completion, and realistic industrial programming scenarios ( 44 ; 26 ; 46 ) . These benchmarks better reflect real software engineering workflows, increasingly supported by agentic techniques such as multi-agent debate ( 22 ) , skill-based optimization ( 41 ) , and adaptive context pruning ( 40 ) . Complementary benchmarks have also been proposed for repository-level question answering ( 28 ) .