Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Natural language processing

Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi · 2026

Read Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Natural language processing by Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10496.txt.

Description

Ali Al-Kaswan leads a team with Claudio Spiess, Prem Devanbu, Arie van Deursen, and Maliheh Izadi. They ask if code models pick a bug they have seen over a fix they have not. Models now write and debug code each day. Their code can hold bugs that came from train text. A fix may be right but new to the model.

The team test that pull. They show a model pairs of code: a known bug and its fix. They track which one the model rates high. They also track how oft the model saw each in train. The work shows if fame can beat truth in code picks.

Catalog record

CREATORAli Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10496

Browse the full library · About the Tower of Babel