LLMs are Bug Replicators: An Empirical Study on LLMs’ Capability in Completing Bug-prone Code
Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu · 2025
Read LLMs are Bug Replicators: An Empirical Study on LLMs’ Capability in Completing Bug-prone Code by Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11082.txt.
Description
Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie Zhang, Zheng Li, and Yong Liu wrote this paper in two thousand twenty-five. It tests whether language models, fed on past code, copy old bugs when they complete bug-prone code.
Across seven models and the Defects4J set, bug-prone completion proves far harder than plain completion. A model is about as apt to write the buggy line as the right one. A large share of its fresh bugs match the old pre-fix code word for word, a sign of bias from training on flawed code. Post fixes raise agreement but not accuracy. The authors call for models and tools built with this flaw in mind.
Catalog record
| CREATOR | Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11082