On the Evaluation of Large Language Models in Multilingual Vulnerability Repair Journal: JACM 111 CCS: Software and its engineering Software testing and debugging
Dong Wang, Junji Yu, Honglin Shu, Michael Fu, Chakkrit Tantithamthavorn, Yasutaka Kamei, Junjie Chen · 2025
Read On the Evaluation of Large Language Models in Multilingual Vulnerability Repair Journal: JACM 111 CCS: Software and its engineering Software testing and debugging by Dong Wang, Junji Yu, Honglin Shu, Michael Fu, Chakkrit Tantithamthavorn, Yasutaka Kamei, Junjie Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03470.txt.
Description
Dong Wang and co-authors evaluate language models for repairing code flaws across languages in a 2025 study. Past automated repair tools worked mainly on one language family. Large language models promise repair in any language, but their real skill was unproven.
The team ran a large study across seven programming languages, comparing top models with a leading repair tool. A leading model, tuned with a few examples, matched the best tool overall. Models also fixed flaws the tool missed, and the reverse held too. The results suggest hybrid repair, where models and classic tools cover each other's gaps.
Catalog record
| CREATOR | Dong Wang, Junji Yu, Honglin Shu, Michael Fu, Chakkrit Tantithamthavorn, Yasutaka Kamei, Junjie Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03470