Examining LLMs Ability to Summarize Code Through Mutation-Analysis Conference: ; ; DOI: XXXXXXX.XXXXXXX Conference: ; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06
Lara Khatib, Michael Pu, Bogdan Vasilescu, Meiyappan Nagappan · 2026
Read Examining LLMs Ability to Summarize Code Through Mutation-Analysis Conference: ; ; DOI: XXXXXXX.XXXXXXX Conference: ; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Lara Khatib, Michael Pu, Bogdan Vasilescu, Meiyappan Nagappan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17838.txt.
Description
Lara Khatib and team test whether language model code summaries truly match program behavior, using mutation analysis. Developers rely on model summaries for documentation, testing and review. Yet models often describe intended code conduct, its intent, while missing edge cases or logic changes that define actual behavior.
The method generates a summary, injects a targeted mutation, and checks whether the model updates its summary to the new behavior. Three experiments total over six hundred evaluations across sixty-two programs. On controlled synthetic programs, summary accuracy falls sharply with complexity. It drops from about three quarters for single functions to under one fifth for multi-threaded systems. Mutation type and location matter less. Tests on human-written programs confirm the same patterns, with models describing algorithmic intent over mutated behavior. GPT-4 versus GPT-5.2 shows a substantial leap, yet both still struggle to separate implementation details from standard algorithmic patterns.
Catalog record
| CREATOR | Lara Khatib, Michael Pu, Bogdan Vasilescu, Meiyappan Nagappan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-17838