Foundation Models as Oracles for Refactoring Correctness Detection
Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca · 2026
Read Foundation Models as Oracles for Refactoring Correctness Detection by Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02096.txt.
Description
Rohit Gheyi and colleagues test whether foundation models can act as oracles for spotting bugs introduced by refactoring tools. Popular development environments can change behaviour or break compilation when they transform Java code, which erodes trust. Past detection used hand-built preconditions and static or dynamic analysis, with limited adaptability.
The study uses zero-shot prompting, with no task training, on 226 real refactoring bugs gathered over more than a decade from three major Java environments, across forty-seven refactoring types. Results varied by model. One open model reached about eighty percent accuracy on first runs, while a stronger model reached about ninety-four percent. The authors assess what that spread means for tool builders.
Catalog record
| CREATOR | Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02096