AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil Conference: 23rd International Conference on Mining Software Repositories; April 13–1

Anwar Ghammam, Mohamed Almukhtar · 2026

Read AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil Conference: 23rd International Conference on Mining Software Repositories; April 13–1 by Anwar Ghammam, Mohamed Almukhtar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16839.txt.

Introduction

Anwar Ghammam and Mohamed Almukhtar check the build code that AI tools write. Build files tell how source code turns into apps. They are a key part of the software life span. Yet past checks of AI code skipped build files. Their 2026 preprint fills that gap with a broad check.

The team grades bot-made build code for flaws and upkeep cost. Weak build code can break whole teams' work. It can also hide flaws in shipped apps. Their grades show where bots need a human eye. That helps firms set rules for bot use in build tasks.

Catalog record

CREATORAnwar Ghammam, Mohamed Almukhtar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793563 ISBN: 979-8-4007-2474-9/2026/04 Anwar Ghammam, Mohamed Almukhtar Abstract . The rapid adoption of AI coding agents for software development has raised important questions about the quality and maintainability of the code they produce. While prior studies have examined AI-generated source code, the impact of AI coding agents on build systems—a critical yet understudied component of the software lifecycle—remains largely unexplored. This data mining challenge focuses on AIDev, the first large-scale, openly available dataset capturing agent-authored pull requests (Agentic-PRs) from real-world GitHub repositories. Our paper leverages this dataset to investigate (RQ1) whether AI coding agents generate build code with quality issues (e.g., code smells), (RQ2) to what extent AI agents can eliminate code smells from build code, and (RQ3) to what extent Agentic-PRs are accepted by developers. We identified 364 maintainability- and security-related build smells across varying severity levels, indicating that AI-generated build code can introduce quality issues—such as lack of error handling, and hardcoded paths or URLs—while also, in some cases, removing existing smells through refactorings (e.g., Pull Up Module and Externalize Properties). Notably, more than 61% of Agentic-PRs are approved and merged with minimal human intervention. This dual impact underscores the need for future research on AI-aware build code quality assessment to systematically evaluate, guide, and govern AI-generated build systems code. † † cc-license: by 1. Introduction The recent advances in modern software systems have enabled their rapid adoption of Large language Models and AI coding agents, and they are widely integrated into software development workflows ( Grewal et al. 2024 ; Nam et al. 2024 ; Coignion et al. 2024 ; Fakhoury et al. 2024 ; Yetiştiren et al. 2023 ; Becker et al. 2023 ) . A growing body of research has examined the quality, correctness, and maintainability of AI-generated source code across multiple domains ( Vaithilingam et al. 2022 ; Pearce et al. 2025 ; Asare et al. 2023 ; Siddiq and Santos 2022 ; Liu et al. 2024 ; Siddiq et al. 2024 ; Grewal et al. 2024 ) . However, while AI agents’ assistance has been extensively studied for source code, comparatively little attention has been given to their emerging role in build systems. Modern software development relies heavily on build tools such as Maven ( Porter [n. d.] ) , Gradle ( 2023 [n. d.] ) , and Cmake ( Swidzinski 2022 ) to manage dependencies, compilation, packaging, and deployment, yet the quality of AI-generated build code remains largely unexplored. In particular, we lack empirical evidence on whether AI agents introduce or remove build-specific code quality issues and how developers respond to automated build code generation in practice. We narrow this gap by conducting the first empirical study of AI-generated build code assessment across common build systems such as Maven, Gradle, CMake, and Make. Specifically, we focus on the following research questions: RQ1: Do AI-coding agents generate build code with smells? A recent study ( Siddiq et al. 2022 ) showed that datasets used to train open-source AI coding agents contain code and security smells that leak to the output generated by these models. In this RQ, we investigate whether these agents also generate build code with quality issues. Across 387 PRs and 945 build files, we found that AI agents mainly introduce maintainability-related code smells (e.g, Deprecated Dependencies, and Lack of Error Handling) and security-related code smells (e.g., Hardcoded Credentials). RQ2: To what extent can AI coding agents remove code smells from Build code? We examined whether AI changes reduce existing code issues in build scripts. We identified 31 build files where smells were eliminated, and a manual inspection showed that many improvements stemmed from refactoring changes, such as Pull Up Module, and Remove unused dependencies, which align with build-specific maintenance patterns reported in prior research on build refactoring ( Ghammam et al. 2025 ) . RQ3: To what extent are Agentic-PRs accepted by developers? We examine whether developers accept AI-generated build-related code and analyze any modifications they make prior to merging. Our findings show that most AI-generated build PRs are merged with little to no manual adjustment, suggesting an early trust and adoption among developers. The main contributions of our paper are as follows: • The first empirical study to assess the quality of AI-generated build code. • Identification of refactoring actions performed by AI agents that help remove code smells in build scripts. • An open-source dataset of labeled pull requests, including associated code smells and introduced refactorings, along with all manual annotations and fully reproducible code, is available in our replication package ( rep 2025 ) . 2. Background 2.1. Build Systems Build automation is a critical part of modern software development ( McIntosh et al. 2012 ) . Build tools provide flexible ways to model, manage, and maintain complex software projects. They rely on configuration files to define dependencies, specify build tasks, and orchestrate build process workflows ( Muschko 2014 ; 2023 [n. d.] ; MacNeill [n. d.] ; Porter [n. d.] ; McIntosh et al. 2012 ; Clemencic and Mato 2012 ) . Within this work, we specifically focus on Maven ( Porter [n. d.] ) , Gradle ( 2023 [n. d.] ) , Cmake ( Clemencic and Mato 2012 ) and Makefile ( mak 2025 ) , widely used build systems that provide a good representation of the different approaches to build automation. 2.2. Code Smells & Refactoring A code smell is an indicator of an improper choice of system design and implementation strategy ( Fowler [n. d.] ) . These flaws can stifle software development or increase the chance of future errors ( Reis et al. 2020 ; Tamanna et al. 2025 ) . Refactoring is defined as the process of restructuring existing code without changing its external behavior to improve readability, maintainability, and extensibility ( Kruchten et al. 2013 ) . Refactoring can help reduce smells by improving code design and quality ( Suryanarayana et al. 2014 ; Ghammam et al. 2025 ; Ivers et al. 2024 ; Almukhtar et al. 2025 ) . In the context of build code, refactoring can involve simplifying build scripts, removing duplication, and improving maintainability ( Duncan 2009 ; Ghammam et al. 2025 ) . 2.3. Sniffer: A Static Analyser for build code Sniffer ( Tamanna et al. 2025 ) is a static analysis tool specifically designed to detect code smells in build systems such as Maven, Gradle, CMake, and Make. Sniffer provides automated detection of both maintainability-related and security-relevant code smells, including outdated or inconsistent dependency declarations, Hardcoded Dependencies, insecure URLs, hardcoded credentials, Suspicious Comments, and overly complex build logic . Sniffer analyzes each build file independently and reports build-specific smells along with their severity (low, medium, or high), a textual description, and the location of the issue. A comprehensive catalog of the code smells categories identified by Sniffer is provided in ( Tamanna

Read the complete text