Leveraging LLMs, IDEs, and Semantic Embeddings for Automated Move Method Refactoring
Fraol Batole *, Abhiram Bellur *, Malinda Dilhara, Mohammed Raihan Ullah, Yaroslav Zharov, Timofey Bryksin, Kai Ishikawa, Haifeng Chen, Masaharu Morimoto, Shota Motoura, Takeo Hosomi, Tien N. Nguyen, Hridesh Rajan, Nikolaos Tsantalis, Danny Dig · 2025
Read Leveraging LLMs, IDEs, and Semantic Embeddings for Automated Move Method Refactoring by Fraol Batole *, Abhiram Bellur *, Malinda Dilhara, Mohammed Raihan Ullah, Yaroslav Zharov, Timofey Bryksin, Kai Ishikawa, Haifeng Chen, Masaharu Morimoto, Shota Motoura, Takeo Hosomi, Tien N. Nguyen, Hridesh Rajan, Nikolaos Tsantalis, Danny Dig (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20934.txt.
Introduction
Abhiram Bellur and Fraol Batole lead a team of fifteen authors building the first fully model-powered assistant for moving a method to a better class. Move Method is a signature refactoring, and expert developers place methods differently from existing recommendation tools. Language models, trained on natural code, suggest expert-like moves, the authors' formative study found.
The catch is reliability: up to eighty percent of raw model suggestions were hallucinations. Their assistant filters suggestions through static analysis in the development environment and forces the model to critique and rank its own proposals. Project-level retrieval supplies the global context the task needs. The system automates the whole cycle, from recommendation to executing the move.
Catalog record
| CREATOR | Fraol Batole *, Abhiram Bellur *, Malinda Dilhara, Mohammed Raihan Ullah, Yaroslav Zharov, Timofey Bryksin, Kai Ishikawa, Haifeng Chen, Masaharu Morimoto, Shota Motoura, Takeo Hosomi, Tien N. Nguyen, Hridesh Rajan, Nikolaos Tsantalis, Danny Dig |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20934
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging LLMs, IDEs, and Semantic Embeddings for Automated Move Method Refactoring Fraol Batole *, Abhiram Bellur *, Malinda Dilhara, Mohammed Raihan Ullah, Yaroslav Zharov, Timofey Bryksin, Kai Ishikawa, Haifeng Chen, Masaharu Morimoto, Shota Motoura, Takeo Hosomi, Tien N. Nguyen, Hridesh Rajan, Nikolaos Tsantalis, Danny Dig Abstract MoveMethod is a hallmark refactoring. Despite a plethora of research tools that recommend which methods to move and where, these recommendations do not align with how expert developers perform MoveMethod . Given the extensive training of Large Language Models and their reliance upon naturalness of code , they should expertly recommend which methods are misplaced in a given class and which classes are better hosts. Our formative study of 2016 LLM recommendations revealed that LLMs give expert suggestions, yet they are unreliable: up to 80% of the suggestions are hallucinations. We introduce the first LLM fully powered assistant for MoveMethod refactoring that automates its whole end-to-end lifecycle, from recommendation to execution. We designed novel solutions that automatically filter LLM hallucinations using static analysis from IDEs and a novel workflow that requires LLMs to be self-consistent, critique, and rank refactoring suggestions. As MoveMethod refactoring requires global, project-level reasoning, we solved the limited context size of LLMs by employing refactoring-aware retrieval augment generation (RAG). Our approach, MM-assist , synergistically combines the strengths of the LLM, IDE, static analysis, and semantic relevance. In our thorough, multi-methodology empirical evaluation, we compare MM-assist with the previous state-of-the-art approaches. MM-assist significantly outperforms them: (i) on a benchmark widely used by other researchers, our Recall@1 and Recall@3 show a 1.7x improvement; (ii) on a corpus of 210 recent refactorings from Open-source software, our Recall rates improve by at least 2.4x. Lastly, we conducted a user study with 30 experienced participants who used MM-assist to refactor their own code for one week. They rated 82.8% of MM-assist recommendations positively. This shows that MM-assist is both effective and useful. I Introduction MoveMethod is a key refactoring [ 1 ] that relocates a misplaced method to a more suitable class. A method is misplaced if it interacts more with another class’ state than its own. MoveMethod improves modularity by aligning methods with relevant data, enhances cohesion, and reduces coupling. It removes code smells like FeatureEnvy [ 2 ] , GodClass [ 3 ] , DuplicatedCode [ 4 ] , and MessageChain [ 1 ] , reducing technical debt. It ranks among the top-5 most common refactorings [ 5 , 6 , 7 ] , both manually and automatically performed. The MoveMethod lifecycle has four phases: (i) identifying a misplaced method m in its host class H , (ii) finding a suitable target class T , (iii) ensuring refactoring pre- and post-conditions to preserve behavior, and (iv) executing the transformation. Each phase is challenging—identifying candidates requires understanding design principles and the codebase, while checking preconditions [ 8 , 2 ] demands complex static analysis. The mechanics involve relocating m , updating call sites, and adjusting accesses. Due to these complexities, existing solutions are incomplete; IDEs handle preconditions and mechanics, while research tools aim to identify opportunities. The research community has proposed various approaches [ 2 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] for identifying misplaced methods or recommending new target classes, typically optimizing software quality metrics like cohesion and coupling. These approaches fall into three categories: (i) static analysis-based [ 2 , 9 , 10 ] , (ii) machine learning classifiers [ 11 , 12 , 13 ] , and (iii) deep learning-based [ 14 , 15 , 16 ] . However, static analysis relies on expert-defined thresholds, and ML/DL methods require continual retraining as coding standards evolve, often diverging from real-world development practices. We hypothesize that achieving good software design that is easy to understand and resilient to future changes is a balancing act between science (e.g., metrics, design principles) and art (e.g., experience, expertise, and intuition about what constitute good abstractions). This can explain why refactorings that optimize software quality metrics are not always accepted in practice [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] . In this paper, we introduce the first approach to automate the entire MoveMethod refactoring lifecycle using Large Language Models (LLMs). We hypothesize that, due to their extensive pre-training on billions of methods and their reliance on the naturalness of code , LLMs can generate an abundance of MoveMethod recommendations. We also expect LLM recommendations to better align with expert practices. In our formative study, we found LLMs (GPT-4o, in particular) are prolific in generating suggestions, averaging 6 recommendations per class. However, two major challenges must be addressed to make this approach practical. First, LLMs produce hallucinations , i.e., recommendations that seem plausible but are flawed. In our formative study of 2016 LLM recommendations, we identified three types of hallucinations (e.g., recommendations where the target class does not exist), and found that up to 80% of LLM recommendations are hallucinations. We discovered novel ways to automatically eliminate LLM hallucinations, by complementing LLM reasoning (i.e., the creative, non-deterministic, and artistic part akin to human naturalness) with static analysis embedded in the IDE (i.e., the rigorous, deterministic, scientific part ). We utilized code-trained vector embeddings from AI models to identify misplaced methods, and used refactoring preconditions [ 8 ] in existing IDEs (IntelliJ IDEA) to effectively remove all LLM hallucinations. We present these techniques in Section III-B . Second, MoveMethod refactoring requires global, project-level reasoning to determine the best target classes where to relocate a misplaced method. However, passing an entire project in the prompt is beyond the limited window size of current LLMs [ 27 ] . Even with larger window sizes, passing the whole project as context introduces noise and redundancy, as not all classes are relevant; instead this further distracts the LLM [ 27 , 28 ] . We address the limited context size of LLMs by using retrieval augmented generation (RAG) to enhance the LLM’s input. Our two-step retrieval process combines mechanical feasibility (IDE-based static analysis) with semantic relevance (VoyageAI [ 29 ] ), enabling our approach to make informed decisions and perform global project-level reasoning. We coin this approach refactoring-aware retrieval augmented generation , which addresses LLM hallucinations and context limitations while fulfilling the specific needs of MoveMethod refactoring (see Section III-C ). We designed, implemented, and evaluated these novel solutions as an IntelliJ IDEA plugin for Java, MM-assist (Move Method Professional). It synergistically combines the strengths of the LLM, IDE, static analysis, and semantic relevance. MM-assist generates candidates, filters LLM hallucinations, validates and ranks recommendations, and then finally executes the correct refactoring based on user approval using the IDE. We designed a comprehensive, multi-methodology evaluation of MM-assist to corroborate, complement, and expand research findings: formative study, comparative study, replication of real-world refactorings, repository mining, user/case study, and questionnaire surveys. We compare MM-assist with the previous best in class approaches in their respective categories: JMove [ 10 ] – uses static analysis, feTruth [ 16 ] – uses Deep Learning, and HMove [ 30 ] – uses graph neural network to suggest moves and LLM to verify refactoring preconditions. These have been