Agentic Refactoring: An Empirical Study of AI Coding Agents DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Maintaining software CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution
Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan · 2025
Read Agentic Refactoring: An Empirical Study of AI Coding Agents DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Maintaining software CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution by Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04824.txt.
Introduction
Kosei Horikawa leads six authors studying how AI coding agents refactor real software. Tools like Codex, Claude Code, and Cursor now act as autonomous teammates on development tasks. Refactoring, improving code structure without changing behavior, is a core maintenance job.
How agents refactor in practice, and how they compare with humans, was largely unknown. The team mines agent-made refactorings across real open-source Java projects at scale. Their study weighs usage patterns and code quality effects together. The findings give engineering teams early evidence on where agent refactoring helps and where oversight stays essential.
Catalog record
| CREATOR | Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentic Refactoring: An Empirical Study of AI Coding Agents DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/18/06 CCS: Software and its engineering Maintaining software CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan Abstract . Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape. These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks. Agents have become active participants in refactoring, a cornerstone of sustainable software development aimed at improving internal code quality without altering observable behavior. Despite their increasing adoption, there is a critical lack of empirical understanding regarding how agentic refactoring is utilized in practice, how it compares to human-driven refactoring, and what impact it has on code quality. To address this empirical gap, we present a large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15 451 15\,451 refactoring instances across 12 256 12\,256 pull requests and 14 998 14\,998 commits derived from the AIDev dataset. Our empirical analysis shows that refactoring is a common and intentional activity in this development paradigm, with agents explicitly targeting refactoring in 26.1% of commits. Analysis of refactoring types reveals that agentic efforts are dominated by low-level, consistency-oriented edits, such as Change Variable Type (11.8%), Rename Parameter (10.4%), and Rename Variable (8.5%), reflecting a preference for localized improvements over the high-level design changes common in human refactoring. Additionally, the motivations behind agentic refactoring focus overwhelmingly on internal quality concerns, with maintainability (52.5%) and readability (28.1%). Furthermore, quantitative evaluation of code quality metrics shows that agentic refactoring yields small but statistically significant improvements in structural metrics, particularly for medium-level changes, reducing class size and complexity ( e.g., Class LOC median Δ \Delta = -15.25). 1. Introduction Refactoring is a cornerstone of sustainable software development that improves a software system’s internal quality without changing its observable behavior ( 35 ) . Since Martin Fowler’s catalog ( 19 ) established the field’s conceptual foundation, subsequent studies have emphasized refactoring as a core maintenance activity that supports long-term software evolvability. Prior work links refactoring to improved readability ( 26 ; 45 ) , higher developer productivity ( 33 ) , and the removal of code smells ( 52 ) , as well as to preparing a codebase for future modifications ( 39 ) . Despite its importance, refactoring demands deep domain knowledge and specialized skills. When executed improperly, refactoring can introduce bugs ( 38 ) , break existing tests ( 40 ; 23 ) , or destabilize the system ( 6 ) . To mitigate these inherent risks, researchers have proposed many sophisticated approaches over the decades. These techniques include methods to identify appropriate refactoring locations ( 56 ) , development of refactoring recommendation systems ( 32 ) , and automatic refactoring approaches ( 48 ) . While several studies reported that practitioners did not use refactoring tools ( 34 ) , the advent of large language models (LLMs) has recently accelerated these techniques and spurred practical adoption of automatic refactoring ( 50 ; 17 ; 5 ) . Many studies report that practitioners now actively leverage LLMs to generate and apply refactoring operations ( 58 ; 60 ; 3 ; 16 ) . The software engineering landscape is now undergoing another fundamental transformation with the rise of agentic coding tools ( 21 ) . Unlike traditional prompt-based LLM workflows, where developers manually guide the AI step-by-step, agentic coding tools such as OpenAI Codex, 1 1 1 https://chatgpt.com/features/codex Claude Code, 2 2 2 https://www.claude.com/product/claude-code and Cursor 3 3 3 https://cursor.com/ operate as AI teammates ( 28 ) . These agents can autonomously plan, execute, test, and iterate on complex development tasks with minimal human intervention ( 44 ) . In this new paradigm, agents act as collaborative refactoring teammates, actively participating in the refactoring process, ranging from simple cleanups to substantial design modifications. However, while these tools are being rapidly adopted, there is currently no empirical understanding of how this new class of agentic refactoring is used in practice, how it compares to human-driven refactoring, and what impact it has on code quality. To address these critical gaps, we conduct a large-scale empirical study of refactorings generated by AI agents in real-world open-source projects based on the AIDev dataset ( 28 ) . We identify 15\,451 refactoring instances across 12\,256 pull requests and 14\,998 commits in Java generated by agentic coding tools. We examine these agentic refactorings along four core dimensions (i.e., prevalence, types, purposes, and impacts) to answer the following research questions (RQs): RQ_{1} : To what extent is refactoring activity in agentic commits? To determine whether refactoring is a significant and intentional activity in agentic software development, we first quantify how often it occurs. We find that refactoring is common, appearing in 26.1% of agentic Java commits, which contain a total of 7127 refactoring instances. RQ_{2} : What are the common types of agentic refactoring? To assess the current sophistication of AI agents, it is crucial to determine whether they perform simple cosmetic cleanups or engage in more complex structural transformations. Our analysis reveals that agentic refactoring is dominated by low-level, consistency-oriented edits like renaming and type changes, showing a clear preference for localized improvements over the high-level design changes common in human refactoring. RQ_{3} : What is the purpose of agentic refactoring? Understanding the why behind these operations is essential to see if developers use agents for proactive quality improvement or for other tactical reasons. We find that agentic refactoring is overwhelmingly driven by the desire to improve internal code quality, specifically maintainability (52.5%) and readability (28.1%). RQ_{4} : To what extent does agentic refactoring affect code quality? To justify agentic refactoring adoption, it is essential to quantitatively measure whether agentic refactoring provides tangible benefits to the codebase’s structural health. The results show that agentic refactoring yields small but significant improvements in structural metrics like code size and complexity, yet it currently fails to reduce the overall count of known design and implementation smells. Our findings provide the first large-scale empirical baseline of agentic refactoring. These results have direct implications for developers, helping them set expectations for what current agents can (and cannot) reliably refactor. For researchers, our work opens new avenues for studying human-agent collaboration in software maintenance and provides a foundation for future studies. Finally, for coding agent builders, our findings highlight current limitations ( e.g., focus on low-level consistency, limited impact on design smells) and opportunities for creating agents that can perform more sophisticated, structurally-aware refactorings. Replication Package. To facilitate replication and further studies, we provide the data used in our replication package. 4 4 4 https://github.com/Mont9165/Agent_Refactoring_Analysis Paper Structure. The remainder of this paper is organized as follows. Section 2 presents the motivating examples. Section 3 details our study design, data collection, and analysis methodology. Section 4 reports the empirical findings. Section 5 summarizes the key insights and discusses the implications of our study. Section 6 reviews related work. Section 7 outlines threats to validity, and Section 8 concludes the paper. 2. Motivating Example The integration of AI agents into software development workflows has accelerated rapidly in recent years.