Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python

Muntasir Adnan, Carlos C. N. Kuhn · 2026

Read Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python by Muntasir Adnan, Carlos C. N. Kuhn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01320.txt.

Description

Adnan and Kuhn test how well language models and older scanning tools can name the exact weakness in flawed Python code. Simple pass or fail marks are too blunt, they argue, because a near miss still helps a coder more than a wild guess. Their benchmark, called ALPHA, grades answers with that difference in mind.

Seven language models and two scanning tools were compared at the level of single functions. Models did far better overall, though scanners were more precise when they did flag a flaw. Agreement between models ranged from very low to fairly high. The authors suggest using their grading in training so future tools learn finer distinctions.

Catalog record

CREATORMuntasir Adnan, Carlos C. N. Kuhn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01320

Browse the full library · About the Tower of Babel