Evaluating LLM-Generated Code: A Benchmark and Developer Study DOI: XXXXXXX.XXXXXXX Conference: International Conference on Evaluation and Assessment in Software Engineering; June 09–12, 2026; Glasgow, Scotland ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation CCS: Human-centered computing Empirical studies in HCI
Joanna Szych, Prof. Dr. Anne Schwerk · 2026
Read Evaluating LLM-Generated Code: A Benchmark and Developer Study DOI: XXXXXXX.XXXXXXX Conference: International Conference on Evaluation and Assessment in Software Engineering; June 09–12, 2026; Glasgow, Scotland ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation CCS: Human-centered computing Empirical studies in HCI by Joanna Szych, Prof. Dr. Anne Schwerk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09059.txt.
Introduction
Joanna Szych and Anne Schwerk argue that judging AI-written code only by whether it runs misses much of what matters. Their study sets out a fuller way to grade code from large language models, one that also weighs quality and how usable the code feels to working programmers.
The method combines a correctness test built on a complex, multi-level class project with checks of code quality and a survey of developers who reviewed samples in a structured process. They apply it to three well-known models. Developer reviews, they find, surface problems about whether code is ready for real use that standard correctness tests alone would never show.
Catalog record
| CREATOR | Joanna Szych, Prof. Dr. Anne Schwerk |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09059
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating LLM-Generated Code: A Benchmark and Developer Study DOI: XXXXXXX.XXXXXXX Conference: International Conference on Evaluation and Assessment in Software Engineering; June 09–12, 2026; Glasgow, Scotland ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Empirical software validation CCS: Computing methodologies Natural language generation CCS: Human-centered computing Empirical studies in HCI Joanna Szych, Prof. Dr. Anne Schwerk Abstract . Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers’ opinions on generated code samples gathered through a structured code-review process. The proposed methodology’s usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach. 1. Introduction In the code generation task (NL2Code), the most common evaluation method is a benchmark that measures the percentage of predefined tasks solved correctly, verified by unit tests. A characteristic of benchmarks is their structure, where each task focuses on a single assignment. Thanks to this, tasks are independent of one another and can be executed in any order. However, real-world coding assignments require a very different approach to programming. In many personal and work-based use cases, it is often necessary to create a slightly larger codebase than a simple script to solve the chosen task. In such cases, when one wants to use a Large Language Model (LLM) for help, it may be necessary to ask for different functionalities in separate prompts. That means that solving some projects with LLMs, especially from the beginning, may require a longer conversation with the model. Unfortunately, most benchmarks do not assume this type of sequential prompting for the creation of whole projects, especially larger ones with many files and dependencies. Moreover, when working on such projects, it is common for an author to share the code with others or pick it back up after a few months’ break. This is why, in real-world coding projects, in addition to working correctly, the code should be easy to read, understand, and maintain. While some common good practices can be verified using static analysis, many aspects remain difficult to detect without a human eye. That is one of the reasons why, in many professional scenarios, a newly created code is verified by at least one other person, who checks whether it follows agreed coding standards. Unfortunately, there is a scarcity of research in this area that involves human experts evaluating code generated by Large Language Models. This paper aims to introduce an evaluation methodology that bridges the aforementioned research gaps by developing a reproducible approach to using LLMs for complex project creation and verifying the quality of those solutions through human evaluation. To reach this goal, the following three research questions were posed as a means of guiding and summarising further work: RQ1 :: How can we design an evaluation framework that assesses LLM-generated code in realistic, multi-component project contexts rather than isolated function-level tasks? RQ2 :: How can we combine automated correctness testing and human quality assessment to comprehensively evaluate LLM-generated project implementations? RQ3 :: To what extent do automated correctness metrics and human quality assessments align and complement each other when evaluating LLM-generated projects? To address the research questions, a custom three-fold evaluation methodology was developed that combines the complexity of the generated code with human feedback on its quality. It is based on a multi-part computer science project, which LLMs are asked to solve through a series of predefined prompts. To verify the correctness of the prepared solution, a small custom benchmark was created for this assignment. The results were combined with a code quality score from dedicated software for a comprehensive overview of the solution. In the last step, the generated code undergoes a structured code review conducted by volunteer developers using a specially designed questionnaire. Presenting this approach leads this paper to make the following contributions to the community: • A three-fold evaluation methodology framework combining automated testing with code quality verification and developer surveys to capture both correctness and usability • A project-level correctness benchmark to assess generated software implementations • A survey design dedicated to gathering code reviews from developers in a structured manner • A demonstration of the framework through the evaluation of three LLMs on a Python project The details of the listed contributions are presented in the following sections. First, an overview of related work is presented to provide background on what has already been discovered in the field of code generation evaluation. After that, the creation and setup of each element contributing to the designed evaluation methodology are described. This is followed by an overview of experimental results based on three general-purpose LLMs: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. Finally, the work’s limitations and lessons learned from the experiments are discussed, followed by a conclusion. 2. Related Work To provide an objective and meaningful literature review, which sets a solid background for future deliberations, guidelines defined in “Preferred Reporting Items for Systematic Reviews and Meta- Analyse" (PRISMA) methodology were applied. The search strategy was to select articles written in English from six reputable databases, queried using the keywords “LLM code generation”, “LLM comparison methods”, “LLM evaluation benchmarks”, and “LLM code generation benchmark”, combined with the “OR” operator. The articles were either included or excluded from the subsequent stage of the process based on whether they revolved around comparing models’ coding capabilities in standard programming languages or not. From each source, the first 15 results from the years 2018-2025 were chosen for further examination, which gives 90 results. From those, 73 were removed due to duplicated titles or being outside the scope, leaving only 17 relevant articles. A few of those selected papers are surveys, which summarise all discoveries in the area of Large Language Models for the code generation task from recent years. This type of literature contains an extensive list of resources, from which an additional 26 papers were categorised as containing extremely useful information about popular code generation benchmark and evaluation approaches. The following section synthesises the knowledge gathered in 43 selected papers and highlights the most relevant findings. 2.1. Correctness evaluation One of the basic dimensions of evaluating code generation, and what most of the currently available benchmarks aim to measure, is the correctness of the generated solution. Some of the most popular benchmarks, such as HumanEval ( 3 ) , HumanEval+ ( 9 ) , or MBPP ( 2 ) , are often based on Python code generation. However, there are also expanded versions of those datasets, such as Multilingual HumanEval