Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities

Honglin Shu, Michael Fu, Junji Yu, Dong Wang 🖂, Chakkrit Tantithamthavorn, Junjie Chen, Yasutaka Kamei · 2025

Read Evaluating Large Language Models for Multilingual Vulnerability Detection at Dual Granularities by Honglin Shu, Michael Fu, Junji Yu, Dong Wang 🖂, Chakkrit Tantithamthavorn, Junjie Chen, Yasutaka Kamei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07503.txt.

Description

Honglin Shu and co-authors present a 2025 study on how well language models find security flaws in code across many programming languages. Most past work stuck to one or two languages and judged whole functions only. Real flaws, though, hide in single lines and in every language a team uses.

The team built a test bed from over thirty thousand real patches that fixed flaws, spanning seven languages. Models were scored at two levels: spotting a flawed function, and pinpointing the flawed line. A leading large model, sharpened with instruction tuning and a few examples, beat all rivals, including a code-specific model. The study maps where each model type stands for multilingual security screening.

Catalog record

CREATORHonglin Shu, Michael Fu, Junji Yu, Dong Wang 🖂, Chakkrit Tantithamthavorn, Junjie Chen, Yasutaka Kamei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE137 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07503

Browse the full library · About the Tower of Babel