Foundation Models as Oracles for Refactoring Correctness Detection

Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca · 2026

Read Foundation Models as Oracles for Refactoring Correctness Detection by Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02096.txt.

Description

Rohit Gheyi and colleagues test whether foundation models can act as oracles for spotting bugs introduced by refactoring tools. Popular development environments can change behaviour or break compilation when they transform Java code, which erodes trust. Past detection used hand-built preconditions and static or dynamic analysis, with limited adaptability.

The study uses zero-shot prompting, with no task training, on 226 real refactoring bugs gathered over more than a decade from three major Java environments, across forty-seven refactoring types. Results varied by model. One open model reached about eighty percent accuracy on first runs, while a stronger model reached about ninety-four percent. The authors assess what that spread means for tool builders.

Catalog record

CREATORRohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, Baldoino Fonseca
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE133 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02096

Browse the full library · About the Tower of Babel