Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs

Rohit Gheyi, Márcio Ribeiro, Jonhnanthan Oliveira · 2025

Read Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs by Rohit Gheyi, Márcio Ribeiro, Jonhnanthan Oliveira (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18454.txt.

Introduction

Rohit Gheyi, Márcio Ribeiro, and Jonhnanthan Oliveira published a paper in 2025 on whether small language models can catch bugs that code editors introduce when refactoring. Refactoring should change a program's shape, never its behavior, but popular development tools ship flawed transformations, and the classical detectors, handwritten preconditions plus heavy analysis, scale poorly. The team tested eight small models, open and proprietary, on one hundred real refactoring bugs from Java and Python editors.

Sixteen refactoring types were covered, with the models reasoning cold, no task training, on consumer hardware. The best proprietary model found about eighty-four percent of behavior-breaking bugs, and an open fourteen-billion-parameter model performed comparably. Blocked-but-valid transformations tripped everyone, the top model managing only forty percent there. Small models, the authors conclude, are credible refactoring watchdogs.

Catalog record

CREATORRohit Gheyi, Márcio Ribeiro, Jonhnanthan Oliveira
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18454

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs Rohit Gheyi, Márcio Ribeiro, Jonhnanthan Oliveira Abstract Popular Integrated Development Environments (IDEs) frequently contain bugs in their refactoring implementations. Ensuring that a transformation preserves a program’s behavior is a complex task. Traditional detection methods rely on predefined preconditions for each refactoring type, limiting their scalability and adaptability to new transformations. These methods often require extensive static and dynamic analyses, which are computationally expensive, time-consuming, and may still fail to detect certain refactoring bugs. This study evaluates the effectiveness of Small Language Models (SLMs) in detecting two types of refactoring bugs in Java and Python: (i) transformations that introduce errors or behavioral changes ( Type I ) and (ii) transformations unnecessarily blocked by IDEs despite being valid ( Type II ). We assess whether Llama 3.2 3B , Mistral 7B , Gemma 2 9B , Gemma 3 12B , DeepSeek-R1 14B , Phi-4 14B , o1-mini , and o3-mini-high can accurately detect 100 refactoring bugs reported in widely used Java and Python IDEs, such as Eclipse and NetBeans . The study covers 16 refactoring types and employs zero-shot prompting on consumer-grade hardware to evaluate the models’ ability to reason about refactoring correctness without explicit prior training. The proprietary o3-mini-high model achieved the highest detection rate, identifying 84.3% of Type I bugs. The open-source Phi-4 14B performed comparably well, demonstrating strong effectiveness across both bug types. However, o3-mini-high struggled with Type II bugs, correctly identifying and applying valid but blocked transformations in only 40% of cases. Some open models, such as Phi-4 14B , exhibited comparable performance, underscoring a broader challenge in effectively addressing these issues. The findings highlight the potential of SLMs for efficiently detecting refactoring bugs, particularly in verifying behavioral changes. Integrating SLMs into refactoring tools can enhance their reliability in a lightweight manner, reducing reliance on complex static and dynamic analyses. Additionally, SLMs offer a more adaptable solution capable of generalizing across different refactoring types and programming languages, addressing key limitations of traditional approaches. ∎ 1 Introduction Refactoring [ 1 , 2 ] is the process of restructuring a program to improve its internal quality while preserving its observable behavior. The first refactoring tool was proposed by Roberts [ 3 ] . Integrated Development Environments (IDEs), such as Eclipse and NetBeans , provide automated support for various refactorings, making the process more accessible to developers. This automation is important for maintaining code quality and improving software maintainability over time. However, ensuring that refactoring transformations preserve a program’s behavior remains a significant challenge [ 4 ] . Despite extensive verification efforts, automated refactorings in IDEs frequently contain bugs that introduce compilation or runtime errors or alter program behavior. Several approaches have been proposed to mitigate these issues, often focusing on detecting errors introduced by refactoring implementations [ 5 , 6 , 7 ] . For instance, SafeRefactor [ 8 ] automatically generates and runs test cases to detect unintended behavioral changes. Other techniques address cases where a valid refactoring is unnecessarily blocked due to overly restrictive preconditions [ 9 , 10 ] . Implementing correct refactorings is inherently complex, as it requires defining and enforcing multiple preconditions while managing diverse transformations [ 4 , 11 ] . The difficulty of ensuring correctness in refactoring tools may explain why many developers still prefer manual refactorings [ 12 ] . More recently, Large and Small Language Models (LLMs and SLMs) have emerged as a promising, cost-effective alternative for detecting software defects, offering efficient analysis on consumer-grade hardware [ 13 , 14 ] . Compared to large-scale models such as ChatGPT and Claude , SLMs require fewer computational resources, making them easier to integrate into real-world development environments, including continuous integration pipelines and on-premise setups. Additionally, their reduced size allows for lower inference latency and more efficient fine-tuning, making them accessible to organizations with limited computational infrastructure. However, the effectiveness of SLMs in accurately identifying refactoring-related bugs remains an open question [ 15 ] , to the best of our knowledge. This study addresses this gap by evaluating the extent to which Llama 3.2 3B , Mistral 7B , Gemma 2 9B , Gemma 3 12B , DeepSeek-R1 14B , Phi-4 14B , o1-mini , and o3-mini-high can detect refactoring bugs in Java and Python. We assess the performance of widely used SLMs in identifying 100 across 16 refactoring types reported in popular IDEs such as Eclipse and NetBeans . We are analyzing bugs related to code transformations rather than traditional bugs that occur within a single program. Our findings indicate that SLMs successfully detected 71.4% of the reported bugs involving incorrect transformations ( Type I bugs). Notably, the proprietary o3-mini-high model achieved the highest detection rate, while the open-source Phi-4 14B demonstrated comparable performance across both bug categories. However, identifying Type II bugs – valid refactorings incorrectly blocked by IDEs – remains a challenge, with models achieving a success rate of only 40%. A mixture of experts approach could be a highly effective strategy for enhancing refactoring bug detection, leveraging the strengths of multiple models to improve accuracy and coverage. This study enhances the understanding of how SLMs can be leveraged for software maintenance, identifying opportunities to improve automated refactoring validation tools. Unlike previous approaches that depend on predefined preconditions for specific refactoring types, our method has the potential to generalize across different refactorings and programming languages, increasing scalability and adaptability. Furthermore, integrating SLMs into IDEs using our setup, similar to Windsurf [ 16 ] and Cursor [ 17 ] AI-powered IDEs, can enhance reliability and automation, potentially reducing dependence on manual refactorings, which remain widespread despite the availability of automated tools [ 12 ] . A cost-effective approach would be to use Phi-4 14B for most cases and leverage o3-mini-high only when human reviewers are uncertain, optimizing resource allocation while maintaining high detection accuracy. This article is structured as follows. Section 2 introduces a motivating example. Sections 3 and 4 detail our methodology and present our findings, respectively. Section 5 provides an in-depth discussion of the results. Section 6 examines related work, and Section 7 summarizes our conclusions and outlines future research directions. 2 Motivating Example To ensure that a refactoring transformation preserves program correctness, it must satisfy a set of preconditions . These preconditions guarantee that the resulting program compiles, maintains or improves its quality, and preserves the original program’s behavior. For example, in the Extract Class refactoring [ 1 ] , the newly introduced class must not have a name that conflicts with an existing class to avoid compilation errors. In practice, developers can refactor code manually, which is error-prone and time-consuming, or use automated tools that support various refactorings, such as VSCode [ 18 ] , IntelliJ [ 19 ] , Eclipse [ 20 ] , and NetBeans [ 21 ] . Currently, verifying behavior preservation is commonly done by compiling the refactored program and running its test suite. However, existing tools are not entirely reliable. They may introduce defects ranging from compilation errors, which are relatively straightforward to detect, to subtle behavioral changes, which are significantly harder to identify. This issue arises because many refactoring implementations do not formally define all necessary preconditions to ensure behavior

Read the complete text