A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis

Katharina Stengg, Christian Macho, Martin Pinzger · 2025

Read A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis by Katharina Stengg, Christian Macho, Martin Pinzger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19481.txt.

Description

Katharina Stengg, Christian Macho, and Martin Pinzger test GPT-5 on a chore every developer knows: predicting what else a code change will touch. Impact analysis is usually manual and slow. The team first built a dataset that records, for each commit, the seed changes, the change pairs, and the change types, data that older sets lack.

GPT-5 and its mini sibling were tried with the seed change and parent tree alone, and again with the diff hunks added. Both models performed poorly in both setups, though the full model beat the mini. The sobering result maps how far language models still sit from dependable change reasoning.

Catalog record

CREATORKatharina Stengg, Christian Macho, Martin Pinzger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19481

Browse the full library · About the Tower of Babel