Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code? PubID: pubid: 979-8-3315-9147-2/25/$31.00 ©2025 IEEE

Alfred Santa Molison 1 , Marcia Moraes 2 , Glaucia Melo 1 , Fabio Santos 2 , Wesley K. G. Assunção 3 · 2025

Read Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code? PubID: pubid: 979-8-3315-9147-2/25/$31.00 ©2025 IEEE by Alfred Santa Molison 1 , Marcia Moraes 2 , Glaucia Melo 1 , Fabio Santos 2 , Wesley K. G. Assunção 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00700.txt.

Introduction

Molison and four co-authors compare code written by language models with code written by people in a 2025 paper. The study asks whether model code is more maintainable and reliable. Coding tasks at three difficulty levels supplied the test material.

Three model setups were tried: zero-shot, few-shot, and fine-tuned. A quality tool scored the code for bugs and fix effort. Model code had fewer bugs overall and took less effort to repair. Fine-tuning cut the share of severe issues but lowered task performance. On the hardest problems, model solutions sometimes added structural flaws absent from human code. The authors urge careful checking of model output.

Catalog record

CREATORAlfred Santa Molison 1 , Marcia Moraes 2 , Glaucia Melo 1 , Fabio Santos 2 , Wesley K. G. Assunção 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is LLM-Generated Code More Maintainable & Reliable than Human-Written Code? PubID: pubid: 979-8-3315-9147-2/25/$31.00 ©2025 IEEE Alfred Santa Molison 1 , Marcia Moraes 2 , Glaucia Melo 1 , Fabio Santos 2 , Wesley K. G. Assunção 3 Abstract Background: The rise of Large Language Models (LLMs) in software development has opened new possibilities for code generation. Despite the widespread use of this technology, it remains unclear how well LLMs generate code solutions in terms of software quality and how they compare to human-written code. Aims: This study compares the internal quality attributes of LLM-generated and human-written code. Method: Our empirical study integrates datasets of coding tasks, three LLM configurations (zero-shot, few-shot, and fine-tuning), and SonarQube to assess software quality. The dataset comprises Python code solutions across three difficulty levels: introductory, interview, and competition. We analyzed key code quality metrics, including maintainability and reliability, and the estimated effort required to resolve code issues. Results: Our analysis shows that LLM-generated code has fewer bugs and requires less effort to fix them overall. Interestingly, fine-tuned models reduced the prevalence of high-severity issues, such as blocker and critical bugs, and shifted them to lower-severity categories, but decreased the model’s performance. In competition-level problems, the LLM solutions sometimes introduce structural issues that are not present in human-written code. Conclusion: Our findings provide valuable insights into the quality of LLM-generated code; however, the introduction of critical issues in more complex scenarios highlights the need for a systematic evaluation and validation of LLM solutions. Our work deepens the understanding of the strengths and limitations of LLMs for code generation. I Introduction The rapid advancement of Large Language Models (LLMs), such as OpenAI’s ChatGPT, has led to their increasing application in various domains [ 1 ] . These models have demonstrated remarkable capabilities in understanding and generating human-like text [ 2 ] . In software development, LLMs have opened up new possibilities, with promising results for code generation [ 3 , 4 , 5 ] . As this new technology becomes more prevalent in software development workflows [ 6 , 7 , 8 ] , assessing its performance in generating high-quality code and comparing it with human-generated code is crucial, particularly in terms of code quality issues [ 9 ] . Previous studies on the capabilities of LLMs in code generation focused mainly on model performance [ 10 , 11 , 12 , 13 ] , not considering aspects of code quality. Jamil et al. [ 14 ] have benchmarked LLM code generation against human-written code using static quality metrics (e.g., Pylint, Radon) and multi-criteria ranking methods (e.g., TOPSIS). Our study advances the state of the art by incorporating task difficulty, prompt strategy, and post-generation fix effort through SonarQube, providing an actionable analysis of LLM code quality in realistic development settings. Our study centers on the following research question (RQ): Is LLM-Generated Code More Maintainable and Reliable than Human-Written Code? This RQ quantifies maintainability and reliability differences between code generated by a popular LLM and code written by humans. The robustness of our study lies in comparing code using established software quality metrics with a standardized dataset of coding questions (composed of different difficulty levels) [ 15 ] . We analyzed code generated by the GPT API using various learning strategies, namely zero-shot and few-shot prompts, and a fine-tuned model. The LLM-generated code is then compared with the code snippets in the APPS dataset [ 15 ] , and the quality metrics of all are analyzed. The results of our study show that LLM-generated code has fewer bugs and often requires less effort to fix them. Interestingly, fine-tuned models reduced the prevalence of high-severity issues, such as blocker and critical bugs, and shifted them to lower-severity categories. In competition-level problems, the LLM solutions sometimes introduce critical structural issues that are not present in human-written code. However, our results show that the fine-tuned model solutions lost performance (0.47) compared to the few-shot model (0.86) in Pass@1, while we observed the highest performance in human-generated solutions. Our contributions include a comparative analysis of the maintainability and reliability of human-written and LLM-generated code, as well as the effort time and code performance. Additionally, we make an experimentation infrastructure available that can be used in future studies, replications, or to automate code quality analysis (replication package available in the repository [ 16 ] ). Furthermore, our findings provide valuable insights into the quality of LLM-generated code. Due to the introduction of critical issues in more complex cases, our study emphasizes the need for systematic evaluation and validation of LLM solutions. Our work deepens the understanding of the strengths and limitations of LLMs in automated code generation by providing a systematic analysis focused on maintainability and reliability issues, done through the use of a novel experimental infrastructure. Furthermore, with the methodology and results of our study, we are one step closer to establishing evidence-based guidelines for creating more efficient quality assessment frameworks for academic and industry environments. II Background and Related Work The integration of Artificial Intelligence (AI) into software development has significantly advanced in recent years [ 17 ] . This section provides an overview of relevant literature and foundational concepts that support this study. II-A LLMs in Code Generation LLMs leverage advanced natural language processing capabilities to facilitate the automatic generation of source code, significantly enhancing developer productivity [ 18 ] . Tosi [ 19 ] highlights that integrating AI generative models in software development optimizes coding practices and redefines the overall software engineering landscape. However, while studies like Kwon et al.’s [ 20 ] research on automated program repair demonstrate the utility of LLMs in debugging and patching buggy scripts, they primarily focus on correctness rather than broader software quality metrics such as maintainability or reliability. Recent empirical evaluations provide further insights into these aspects. For instance, Siddiq et al. [ 21 ] and Moratis et al. [ 22 ] identified code smells and harmful patterns in LLM-generated outputs. However, their findings lack detailed numerical benchmarks to assess long-term maintainability or adherence to software quality standards. Recent work by Abbassi et al. [ 23 ] introduces a taxonomy of inefficiencies in LLM-generated code, highlighting that maintainability issues, such as unnecessary complexity and improper documentation, are prevalent, but vary significantly across models depending on their training data. These findings underscore the critical need for quantitative evaluations of maintainability and reliability in LLM-generated code to ensure its suitability for production environments. As developers increasingly rely on LLMs for code generation [ 24 ] , understanding these nuances becomes crucial for maintaining software quality and ensuring that generated code meets the required standards. Leung et al. [ 18 ] emphasize the importance of developer-LLM interaction interfaces, which facilitate seamless collaboration between human expertise and machine-generated code. This collaboration not only enhances software development efficiency but also raises questions about the quality and security of LLM-generated code. II-B Software Quality Metrics Quality metrics play an important role in assessing and ensuring the quality of software products [ 25 , 26 ] . Metrics can facilitate stakeholder communication, providing a common language to discuss software quality and project status. They are particularly relevant in Agile environments, where teams use metrics to assess project success and

Read the complete text