Can LLMs be Effective Code Contributors? A Study on Open-source Projects

Chun Jie Chong, Muyeed Ahmed, Zhihao (Zephyr) Yao, Iulian Neamtiu · 2026

Read Can LLMs be Effective Code Contributors? A Study on Open-source Projects by Chun Jie Chong, Muyeed Ahmed, Zhihao (Zephyr) Yao, Iulian Neamtiu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23340.txt.

Description

Chong and four colleagues test whether large language models can contribute effectively to production open-source code. Their verification and validation framework judges an LLM's fitness to fix bugs or add features in existing projects. Applied to 212 commits across eight projects and three models, GPT-4o, Ministral3, and Qwen3-Coder, success ranges from 0 to 60 percent by project. Failures span syntactically incorrect code, failed static verification, and failed test suites, with special weakness in new code, out-of-range contexts, and parroted training changes.

Catalog record

CREATORChun Jie Chong, Muyeed Ahmed, Zhihao (Zephyr) Yao, Iulian Neamtiu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53088
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23340

Browse the full library · About the Tower of Babel