CodeTaste : Can LLMs Generate Human-Level Code Refactorings?

Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev · 2026

Read CodeTaste : Can LLMs Generate Human-Level Code Refactorings? by Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04177.txt.

Introduction

Thillen and others ask whether language models can generate human-level code refactorings, introducing the CodeTaste benchmark. Results show a clear gap: agents implement well-specified refactorings well but often fail to discover human refactoring choices given only a focus area. A propose-then-implement split aids match, and choosing the best-aligned proposal before implementation yields further gains.

Catalog record

CREATORAlex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90105
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04177

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CodeTaste : Can LLMs Generate Human-Level Code Refactorings? Alex Thillen, Niels Mündler, Veselin Raychev, Martin Vechev Abstract Large language model (LLM) coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. In this paper, we investigate if LLM agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. We present CodeTaste , a benchmark of refactoring tasks mined from large-scale multi-file changes in open-source repositories. To score solutions, we combine repository test suites with custom static checks that verify removal of undesired patterns and introduction of desired patterns using dataflow reasoning. Our experimental results indicate a clear gap across frontier models: agents perform well when refactorings are specified in detail, but often fail to discover the human refactoring choices when only presented with a focus area for improvement. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases. 1 Introduction Figure 1 : Overview of the CodeTaste benchmark pipeline. The workflow follows six key stages: (1) Commit Discovery from GitHub; (2) Generation of Task Descriptions; (3) Rule Generation of static analysis rules that capture refactoring intent; (4) Build Environment to run test suite and agent inference in containerized environment; (5) Inference where the agent generates a patch; and (6) Evaluation of generated patches against static analysis rules and test suite. Coding agents are being rapidly adopted across the software engineering industry ( 21 ; 29 ) , mostly fueled by their recent impressive performance on autonomous resolution of real-world software engineering tasks ( 15 ; 3 ) . However, success on issue resolution does not imply that agents can sustain code quality across iterations. SlopCodeBench ( 24 ) evaluates iterative specification updates and measures verbosity and structural erosion, showing that coding-agent-generated code becomes longer and more structurally strained as requirements evolve. A recent effort to build a browser rendering engine using autonomous coding agents ( 32 ) illustrates some practical consequences: despite making notable early progress, the agents effectively stalled, unable to continue extending a codebase they had themselves created. Similarly, an experiment tasking a team of agents to autonomously build a C compiler ( 4 ) produced a functional 100,000-line codebase capable of compiling the Linux kernel, but resulted in rigid, inefficient code that lacked human-like abstraction. These observations suggest that while current methods excel at producing functional patches, they struggle to produce code that remains comprehensible and extensible over time. Lack of challenging refactoring benchmarks Refactoring is the software engineering practice of applying behavior-preserving transformations that improve structure and maintainability. OpenAI reports that they train and evaluate their models on large-scale code refactoring tasks, without disclosing their internal evaluation frameworks ( 22 ) . The academic community has also contributed benchmarks for refactoring: RefactorBench provides few-file tasks with syntactic tests ( 11 ) , and SWE-Refactor collects 1,099 refactoring of a fixed set of method-level refactorings from 18 Java repositories ( 33 ) . These benchmarks have a key limitation: they only cover refactorings of small and limited scope, which results in them quickly becoming obsolete. For example, GPT-5 Codex already achieves 75\% accuracy on SWE-Refactor. Beyond this size limitation, these benchmarks do not measure whether the model can determine autonomously which refactoring should be performed. This distinction matters: to bootstrap a code base out of accumulated technical debt, agents must recognize when code has become unmaintainable and determine what transformations are necessary to restore clarity ( 25 ) . A recent related study found that LLMs rarely identify refactoring opportunities on their own: given Java files as input, ChatGPT identified only 28 out of 180 refactoring opportunities that human experts had selected ( 19 ) . Our work In this work, we investigate this challenge extensively and use precise code-level measurements to assess whether state-of-the-art coding agents can autonomously discover and implement relevant refactorings in real-world code bases. We introduce an alignment score that measures how many occurrences of undesired coding patterns are converted to desired versions, counting only functionally correct implementations. Our results indicate a clear gap: while frontier models score highly on implementing complex refactorings that are specified in detail, achieving up to 70\% alignment in our evaluation, they fail to mitigate undesired coding patterns correctly when no concrete refactoring is specified, achieving overall less than 8\% alignment. We introduce CodeTaste , a benchmark designed to measure this gap. We publicly release its leaderboard, our code and the dataset 1 1 1 Leaderboard, dataset, and code are available at: https://codetaste.logicstar.ai/ A scalable pipeline for discovering refactoring instances. We mine popular GitHub repositories for 100 large multi-file refactorings performed by human developers. For each task, we generate an executable environment and static analysis rules that match the changed code patterns. The latter support semantic dataflow reasoning within files, which provides more flexibility than syntactic matching, and is automatically verifiable. After the model applies a refactoring, we measure performance, ensuring that (i) the repository test suite passes and (ii) the static analysis rules confirm code pattern transformation. Two complementary tracks for evaluating agent performance. Our benchmark features two tracks: In the Instructed track the agent is tasked to correctly implement a detailed refactoring. Frontier models achieve alignment scores of up to 69.6\% , indicating both correct implementations and consequent transformation. The Open track evaluates whether agents can propose refactorings that align with human developer choices. In this track, the agent is only tasked to improve a general focus area. We discover that agents struggle significantly here, achieving only single-digit scores, even with oracle support. We observe that frontier models consistently outperform lower capability models on our benchmark, with for example GPT-5 Codex achieving 7.7\% alignment in the open track direct setting compared to 2.3\% for Qwen3 . This suggests that human refactoring preferences are already partially captured by general model training. Still, our case studies, we observe that current agents often take shortcuts and tend towards apply minimal, insufficient changes, unless explicitly instructed to plan out a refactoring. Figure 2 : Data funnel for obtaining the CodeTaste benchmark instances. In each step of the main commit pipeline, we filter for the top n entries using different scoring schemes. Finally, we extract 100 instances for which our agentic setup produced a working execution environment. 2 Methodology In Figure 1 , we provide a summary of the CodeTaste pipeline. Based on a dataset of historic code changes, we (i) mine large, multi-file refactorings performed by human developers, (ii) translate each refactoring into task descriptions for our two tracks, (iii) build a reproducible execution environment that can run each repository’s unit tests, and (iv) generate static analysis rules that capture desired and undesired patterns of the transformation. We then run agents to produce a candidate patch and evaluate it using both the repository test suite and our static rules. During evaluation, we task coding agents to reimplement the original human refactoring. These agents can interact with the task description, the codebase, and the

Read the complete text