CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection
Richard A. Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi · 2025
Read CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection by Richard A. Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09433.txt.
Introduction
Richard Dubniczky and colleagues build a fair arena where code security tools of three species compete on the same flaws. Static analyzers, language models, and formal verifiers all hunt vulnerabilities, but they are rarely graded alike. Writing in twenty twenty-five, the team hand-crafted hundreds of small benchmark programs covering dozens of common weakness types. A new scoring rule makes the comparison even-handed. Thirteen static tools, ten language models, and two verifiers entered.
The formal verifier made the fewest false alarms but missed flaw classes outside model checking, like weak cryptography. Static analyzers cried wolf often, burdening developers. Language models excelled on small snippets, then faded and hallucinated as code grew. The authors suggest models will shine inside code completion, warning as code is written. The data set is public.
Catalog record
| CREATOR | Richard A. Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09433
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection Richard A. Dubniczky, Krisztofer Zoltan Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi Abstract Identifying vulnerabilities in source code is crucial, especially in critical software components. Existing methods such as static analysis, dynamic analysis, formal verification, and recently Large Language Models are widely used to detect security flaws. This paper introduces CASTLE (CWE Automated Security Testing and Low-Level Evaluation), a benchmarking framework for evaluating the vulnerability detection capabilities of different methods. We assess 13 static analysis tools, 10 LLMs, and 2 formal verification tools using a hand-crafted dataset of 250 micro-benchmark programs covering 25 common CWEs. We propose the CASTLE Score, a novel evaluation metric for fair comparison. Our results reveal key differences: ESBMC (a formal verification tool) minimizes false positives but struggles with vulnerabilities beyond model checking, such as weak cryptography or SQL injection. Static analyzers suffer from high false positives, increasing manual validation efforts for developers. LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets. However, their accuracy declines, and hallucinations increase as the code size grows. These results suggest that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The dataset is accessible at https://github.com/CASTLE-Benchmark . 1 Introduction Rapid advancements in artificial intelligence (AI) have sparked both excitement and concern about the future of traditional software engineering. For instance, Meta’s recent announcement that AI could soon replace many software engineering roles highlights a shifting landscape in code development [ 1 ] . While AI-driven code generation offers remarkable efficiency, a study by Tihanyi et al. found that all examined large language models (LLMs) produced vulnerable C code [ 2 ] . Similar conclusions have been reached in studies examining other programming languages, such as PHP and Python [ 3 , 4 ] . These large-scale studies consistently indicate that such vulnerabilities arise partly because LLMs lack contextual understanding during the generation process. Several studies highlight that once the code is generated, and a vulnerability is identified, LLMs excel in fixing these [ 5 , 6 ] . The question is, how do we find the vulnerabilities? Despite the growing importance of automated software verification, developers and security practitioners lack clear guidance on which tools are most reliable for detecting vulnerabilities in C code. Several interrelated issues contribute to this uncertainty: 1. Diverse vulnerability types. Security flaws in C code range from classic memory management issues (e.g., buffer overflows) to subtler logical errors. We need to understand which detection methods can reliably detect different categories. 2. Emergence of LLMs. While LLMs exhibit promise in automated code generation, bug fixing, and vulnerability detection, their reliability in different vulnerabilities and coding scenarios is unclear. 3. Lack of standardized benchmarks. Existing datasets often contain too many samples with imbalanced CWE representations, and fail to represent the breadth of CWE vulnerabilities. Tools that rely on compilable code—particularly FV methods—are especially disadvantaged without realistic, fully functional programs. To gauge each tool’s performance accurately, a benchmark must be rigorously validated, contain clearly labeled vulnerabilities, and support line-level detection granularity. Given these challenges, a robust and compilable benchmark dataset that accurately captures major CWE vulnerabilities is paramount to answer the following research questions: • RQ1 : How do state-of-the-art static analysis tools, formal verification methods, and LLM-based approaches compare to effectively detecting C code vulnerabilities? • RQ2 : Are combinations of tools more effective than using a single tool? • RQ3 : What metrics can reliably demonstrate these differences among various tools? Our study holds the following contributions: • We introduce CASTLE (CWE Automated Security Testing and Low-Level Evaluation) , a curated collection of 250 compilable, compact C programs, each containing a single CWE. This benchmark is aimed at enabling direct comparisons among current and future vulnerability scanning tools, including traditional static analyzers, FV techniques, and LLM-based approaches. • We conduct a broad comparison of the most widely used static code analyzers and popular LLMs to assess their effectiveness in detecting important vulnerabilities in the C language, using a new metric called CASTLE Score , thereby providing crucial insights into their relative strengths and weaknesses. The rest of this work is structured as follows: Section 2 reviews related literature and outlines the current state of vulnerability scanning tools and AI-based code analysis. Section 3 details the construction of the CASTLE benchmark, including the selection criteria for CWEs and the methodology for creating the curated C programs. Section 4 discusses the results, and presents the experimental setup and comparative analysis of the 11 static code analyzers and 11 LLMs. Section 5 overviews limitations. Finally, Section 6 concludes the paper and outlines potential directions for further research. 2 Related work Ensuring software correctness, safety, and security is central to software engineering. Examining related literature on the role of AI in software development, most of the existing work and benchmarking approaches focused on testing LLMs’ capabilities in producing functionally correct code. However, safety and security are just as important. 2.1 Datasets and Benchmarks Existing vulnerability datasets are frequently used for fine-tuning machine learning models, yet they exhibit several shortcomings that make them unsuitable for comprehensive benchmarking. First, many datasets offer imbalanced representations of CWE categories, failing to provide adequate test coverage of certain vulnerability types. Second, an extreme or uneven distribution of vulnerable versus non-vulnerable samples either hinders accurate false-positive evaluation (when nearly all samples are vulnerable) or fails to capture diverse false-negative scenarios (when some vulnerability types remain underrepresented). Table 1: C/C++ Datasets for Vulnerability Detection Dataset Size #Multiple Vuln./File Vuln. Snippets Compilable Granularity Labelling Source Draper [ 7 ] 1274k ✔ 5.62% ✘ function Stat mixed Big-Vul [ 8 ] 264k ✘ 100% ✘ function Patch real-world DiverseVul [ 9 ] 349k ✘ 7.02% ✘ function Patch real-world FormAI-v2 [ 2 ] 331k ✔ 62.07% ✔ file FV AI Gen. PrimeVul [ 10 ] 235k ✘ 3% ✘ function Manual real-world SARD [ 11 ] 101k ✘ 100% ✔ file B/S/M mixed Juliet (C/C++) [ 12 ] 64k ✘ 100% ✔ file BDV synthetic Devign [ 13 ] 28k ✘ 46.05% ✘ function Manual real-world REVEAL [ 14 ] 23k ✘ 9.85% ✘ function Patch real-world CVEfixes [ 15 ] 20k ✘ 100% ✘ commit Patch real-world Legend: Patch : GitHub Commits Patching a Vulnerability, Stat : Static Analyzer, BDV : By Design Vulnerable, FV : Formal Verification with ESBMC, Manual : Manual Labeling by Human Experts Furthermore, a key challenge is that many popular datasets lack compilable programs, making it impossible to meaningfully assess formal verification tools such as the Efficient SMT-based Context-Bounded Model Checker (ESBMC ). Among compilable options, SARD stands out for including the Juliet test cases and 45,437 C samples mapped to CWE categories. However, some files exceed 3,000 lines of code, creating two constraints: 1. Large token sizes impose high computational costs on LLM-based approaches and limit the use of smaller-parameter models. 2. The complexity and volume of large files can overwhelm formal verification tools, dramatically increasing runtime and impeding direct comparisons with other analyzers. Another example is FormAI, a large-scale dataset labeled using ESBMC