LLMs are Bug Replicators: An Empirical Study on LLMs’ Capability in Completing Bug-prone Code

Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu · 2025

Read LLMs are Bug Replicators: An Empirical Study on LLMs’ Capability in Completing Bug-prone Code by Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11082.txt.

Description

Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie Zhang, Zheng Li, and Yong Liu wrote this paper in two thousand twenty-five. It tests whether language models, fed on past code, copy old bugs when they complete bug-prone code.

Across seven models and the Defects4J set, bug-prone completion proves far harder than plain completion. A model is about as apt to write the buggy line as the right one. A large share of its fresh bugs match the old pre-fix code word for word, a sign of bias from training on flawed code. Post fixes raise agreement but not accuracy. The authors call for models and tools built with this flaw in mind.

Catalog record

CREATORLiwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M. Zhang, Zheng Li, Yong Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11082

Browse the full library · About the Tower of Babel