Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python
Muntasir Adnan, Carlos C. N. Kuhn · 2026
Read Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python by Muntasir Adnan, Carlos C. N. Kuhn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01320.txt.
Description
Adnan and Kuhn test how well language models and older scanning tools can name the exact weakness in flawed Python code. Simple pass or fail marks are too blunt, they argue, because a near miss still helps a coder more than a wild guess. Their benchmark, called ALPHA, grades answers with that difference in mind.
Seven language models and two scanning tools were compared at the level of single functions. Models did far better overall, though scanners were more precise when they did flag a flaw. Agreement between models ranged from very low to fairly high. The authors suggest using their grading in training so future tools learn finer distinctions.
Catalog record
| CREATOR | Muntasir Adnan, Carlos C. N. Kuhn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01320