A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis
Katharina Stengg, Christian Macho, Martin Pinzger · 2025
Read A Dataset and Preliminary Study of Using GPT-5 for Code-change Impact Analysis by Katharina Stengg, Christian Macho, Martin Pinzger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19481.txt.
Description
Katharina Stengg, Christian Macho, and Martin Pinzger test GPT-5 on a chore every developer knows: predicting what else a code change will touch. Impact analysis is usually manual and slow. The team first built a dataset that records, for each commit, the seed changes, the change pairs, and the change types, data that older sets lack.
GPT-5 and its mini sibling were tried with the seed change and parent tree alone, and again with the diff hunks added. Both models performed poorly in both setups, though the full model beat the mini. The sobering result maps how far language models still sit from dependable change reasoning.
Catalog record
| CREATOR | Katharina Stengg, Christian Macho, Martin Pinzger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-19481