DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu · 2026
Read DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models by Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11895.txt.
Introduction
Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, and their co-authors built a code test set from real developer data. The set, DevBench, holds 1,800 tasks in six code tongues and six task kinds. The tasks come from real telemetry, such as API use and code purpose.
Most code tests use toy tasks cut off from real work. A set drawn from real use can grade models on what devs truly do. It can also show which task kinds still trip models up. The work gives a shared scale for code models in real settings.
Catalog record
| CREATOR | Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11895
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu Abstract DevBench [ 12 ] is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry, such as API usage and code purpose understanding. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement—detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development. 1 Introduction Large Language Models (LLMs) have transformed modern software development by enabling advanced code generation, powering tools like GitHub Copilot [ 11 ] and Cursor [ 2 ] . As these systems are increasingly integrated into real-world workflows, realistic and rigorous evaluation frameworks are essential to understanding their strengths and limitations. Existing benchmarks evaluate different aspects of code generation: problem solving benchmarks for coding problems [ 5 , 3 , 13 , 14 , 29 ] , repository-based benchmarks for challenges in large projects [ 28 , 10 , 9 , 30 , 32 , 16 , 8 ] , and evolving benchmarks addressing contamination [ 18 , 15 ] . However, existing benchmarks rely on code samples scraped from open source repositories or coding challenge websites and generate target completions based on static rules for filling in line, function, or class implementations. This limits them in several ways: First, the target completions are not based on real world usage patterns for code completion tools, and therefore do not focus on common challenging completion scenarios that arise in real world usage. Second, the diagnostic value of these benchmarks is limited because they report aggregate metrics, but cannot attribute differences in performance to specific usage areas. Third, benchmarks collected from publicly available sources are prone to training data contamination, which has been observed in models overfitting to existing benchmarks [ 15 ] . To address these limitations, we introduce DevBench , a realistic and scalable benchmark grounded in observed developer behavior. DevBench focuses on common yet challenging completion scenarios, identified from internal telemetry of over one billion developer code completion interactions and synthesized into 1,800 evaluation instances spanning six languages and six task categories. Each instance is reviewed for quality and realism, ensuring that tasks reflect how developers actually use code completion tools while remaining contamination-resistant. As shown in Table 1 , DevBench advances beyond existing benchmarks in both realism [ 26 ] and scope. It offers four key advantages: (1) realism , with tasks rooted in observed developer behavior; (2) contamination resistance , through synthetic but controlled instance generation; (3) fine-grained evaluation , assessing semantic alignment and developer utility; and (4) cross-language coverage , spanning Python (Py), JavaScript (JS), TypeScript (TS), Java, C++, and C#. Together, these features provide ecological validity: DevBench reflects authentic developer challenges rather than hypothetical tasks, is validated through expert review, and captures diverse contexts across languages and developer skill levels. By enabling both overall rankings and scenario-specific diagnostics, DevBench supports informed model selection and optimization, and provides a contamination-resilient foundation for future research. We open-source the 1,800-instance benchmark and evaluation code. Table 1 : Comparison of a collection of recent code generation benchmarks across size, language coverage, focus, source, and unique features. Benchmark # Tasks Languages Focus Source Unique Feature RepoMasterEval 288 Py, TS Real-world repository completion GitHub repos (>100 stars) Mutation testing for test robustness CrossCodeEval \sim 10k Py, Java, TS, C# Cross-file dependencies GitHub repos (>3 stars) Static analysis for dependencies CoderEval 460 Py, Java Cross-file pragmatic generation GitHub repos (popular tags) Human-labeled docstrings ClassEval 100 Py Class-level generation Manually crafted Multiple interdependent methods HumanEval 164 Py Basic programming tasks Manually crafted Simple interview-style problems HumanEval+ 164 Py Enhanced testing rigor Manually crafted 80 \times more evaluation instances LiveCodeBench 511 Py Contamination-free evaluation Competition platforms Time-based contamination tracking SWE-bench 2,294 Py Repository-level bug fixing GitHub issues and PRs Real-world issues from 12 popular repos BigCodeBench 1,140 Py Diverse function calls as tools Human-LLM collaborative generation 723 function calls from 139 libraries across 7 domains DevBench (this work) 1,800 Py, JS, TS, Java, C++, C# Realistic developer-informed scenarios Synthetically generated, manually reviewed Telemetry-guided, human-validated 2 Benchmark Design We view code generation as a composite, puzzle-solving task in which models must combine distinct capabilities, such as API usage, intent understanding, and syntax control. To evaluate these skills, we define benchmark categories that isolate each capability while ensuring every instance is solvable from the provided prefix/suffix, making evaluation both realistic and fair. Although individual instances are synthesized, DevBench is telemetry-driven: its categories, task types, and scenarios are derived from analysis of over one billion real developer interactions, with synthesis used only to instantiate these empirically derived patterns in a privacy-preserving, contamination-resistant manner. 2.1 From User Telemetry to Categories The benchmark categories are derived from an internal telemetry dataset containing over one billion anonymized code completions, each recording the prefix, suffix, generated and golden completions, and user interactions (accept, reject, edit). This dataset spans diverse contexts over IDEs, geographical locations, language distribution, and developers ranging from students to senior engineers. To satisfy privacy and compliance requirements, we avoid using raw user code. Instead, we construct synthetic evaluation instances that reproduce the structural complexity and usage patterns observed in telemetry. As shown in Figure 1 , the benchmark-generation pipeline begins by sampling telemetry completions and annotating them to identify common failure modes, bottlenecks, and characteristic structures, which we use to derive the benchmark categories. We then refine these findings through iterative discussions with a research group that includes language specialists, ensuring the categories reflect both statistical prevalence and realistic, high-impact developer workflows. Finally, we verify that the reviewed samples are representative, so the resulting benchmark categories capture common scenarios with clear evaluation criteria, incorporate edge cases, and present realistic challenge levels directly grounded in developer behavior. 2.2 Benchmark categories We define six benchmark categories based on our analysis of user telemetry. Each category targets a distinct type of developer intent and is consistently evaluated across languages, with adaptations to reflect the idioms and ecosystems of each target language (see Table 2 ). The categories are described in detail below (examples in Appendix A ). Table 2 : Language-specific adaptations. Here, ML = Machine Learning, HOFs = Higher-Order Functions, RAII = Resource Acquisition Is Initialization, and HW accel. = Hardware Acceleration. Category Python C# C++ Java JavaScript TypeScript API Usage ML libs, scientific computing .NET Core, ASP.NET Systems prog., graphics, HW accel. JDK, enterprise frameworks Browser APIs, Node.js modules Same as JS w/ types Code Purpose Iterators/ generators, context mgrs LINQ/ collections, async-await Iterators/ algorithms, multithreading Streams/ collections, lambdas Closures, promises/ async Same as JS w/ type systems Code2NL Docstrings XML docs Doxygen comments Javadoc JSDoc TSDoc w/ type annot. Low Context Decorators, context mgrs Complex generics, LINQ Template metaprog.,