Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Natural language processing
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi · 2026
Read Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Natural language processing by Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10496.txt.
Description
Ali Al-Kaswan leads a team with Claudio Spiess, Prem Devanbu, Arie van Deursen, and Maliheh Izadi. They ask if code models pick a bug they have seen over a fix they have not. Models now write and debug code each day. Their code can hold bugs that came from train text. A fix may be right but new to the model.
The team test that pull. They show a model pairs of code: a known bug and its fix. They track which one the model rates high. They also track how oft the model saw each in train. The work shows if fame can beat truth in code picks.
Catalog record
| CREATOR | Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10496