SecureAgentBench : Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios
Junkai Chen 1 * , Huihui Huang 1 * , Yunbo Lyu 1 , Junwen An 2 , Jieke Shi 1 , Chengran Yang 1 , Ting Zhang 3 , Haoye Tian 4 , Yikun Li 1 , Zhenhao Li 5 , Xin Zhou 1 , Xing Hu 6 , David Lo · 2025
Read SecureAgentBench : Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios by Junkai Chen 1 * , Huihui Huang 1 * , Yunbo Lyu 1 , Junwen An 2 , Jieke Shi 1 , Chengran Yang 1 , Ting Zhang 3 , Haoye Tian 4 , Yikun Li 1 , Zhenhao Li 5 , Xin Zhou 1 , Xing Hu 6 , David Lo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22097.txt.
Introduction
AI coding agents now write real software, but the code they produce can carry security holes. Junkai Chen and twelve co-authors present SecureVibeBench in this 2025 paper, a benchmark for secure code generation, accepted at ACL 2026. Earlier benchmarks miss how flaws actually enter code: they do not recreate the situations in which human developers introduced real vulnerabilities, so agents and humans cannot be compared fairly.
The benchmark holds just over one hundred secure coding tasks in C and C++, drawn from dozens of projects in a large fuzzing program. Each task demands edits across several files in a big repository. Scoring combines working-code tests with security checks, both static and dynamic. Five popular agents, backed by five large models, were evaluated. Even the best produced correct and secure solutions in under a quarter of the tasks, which the authors read as a warning about current coding agents.
Catalog record
| CREATOR | Junkai Chen 1 * , Huihui Huang 1 * , Yunbo Lyu 1 , Junwen An 2 , Jieke Shi 1 , Chengran Yang 1 , Ting Zhang 3 , Haoye Tian 4 , Yikun Li 1 , Zhenhao Li 5 , Xin Zhou 1 , Xing Hu 6 , David Lo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22097
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SecureAgentBench : Benchmarking Secure Code Generation under Realistic Vulnerability Scenarios Junkai Chen 1 * , Huihui Huang 1 * , Yunbo Lyu 1 , Junwen An 2 , Jieke Shi 1 , Chengran Yang 1 , Ting Zhang 3 , Haoye Tian 4 , Yikun Li 1 , Zhenhao Li 5 , Xin Zhou 1 , Xing Hu 6 , David Lo Abstract Large language model (LLM)-powered code agents are rapidly transforming software engineering by automating tasks such as testing, debugging, and repairing, yet the security risks of their generated code have become a critical concern. Existing benchmarks have offered valuable insights but remain insufficient: they often simplify tasks to function-level completion, overlook the genuine context in which vulnerabilities were introduced, or adopt narrow evaluation protocols that fail to capture either functional correctness or newly introduced vulnerabilities. We therefore introduce SecureAgentBench , a benchmark of 105 coding tasks designed to rigorously evaluate code agents’ capabilities in secure code generation. Each task includes (i) realistic task settings that require multi-file edits in large repositories, (ii) aligned contexts based on real-world open-source vulnerabilities with precisely identified introduction points, and (iii) comprehensive evaluation that combines functionality testing, vulnerability checking through proof-of-concept exploits, and detection of newly introduced vulnerabilities using static analysis. We evaluate three representative agents (SWE-agent, OpenHands, and Aider) with three state-of-the-art LLMs (Claude 3.7 Sonnet, GPT-4.1, and DeepSeek-V3.1). Results show that (i) current agents struggle to produce secure code, as even the best-performing one, SWE-agent supported by DeepSeek-V3.1, achieves merely 15.2% correct-and-secure solutions, (ii) some agents produce functionally correct code well but introduce vulnerabilities, even including new ones not previously recorded, and (iii) adding explicit security instructions for agents does not significantly improve secure coding, underscoring the need for further research. These findings establish SecureAgentBench as a rigorous benchmark for secure code generation and a step toward more reliable software development with LLMs. Repository : https://github.com/iCSawyer/SecureAgentBench = Contact : Junkai Chen ( [email protected] ) and Huihui Huang ( [email protected] ) 1 1 footnotetext: Equal contribution 1 Introduction Recent years have witnessed the remarkable success of large language models (LLMs) in software engineering (SE) ( Hou et al. 2024 ) , spurring the emergence of code agents. Defined as LLM-powered systems capable of autonomously generating, editing, and executing code, these agents substantially improve developer productivity in SE tasks such as software testing ( Mündler et al. 2024 ) , debugging ( Chen et al. 2024 ) , and program repair ( Jimenez et al. 2024 ) , and have become increasingly prominent in modern development workflows ( Cursor 2024 , Yang et al. 2024a , Aider 2025 , Lyu et al. 2025 ) . However, the insecurity of their generated code has emerged as a critical concern ( Asare et al. 2023 ) , e.g., Pearce et al. 2025 shows that about 40% of GitHub Copilot’s code completions were vulnerable and could be attacked and exploited. To facilitate systematic evaluation of such risks, several benchmarks for secure coding of LLMs have been proposed, such as CyberSecEval ( Bhatt et al. 2023 ) , LLMSecEval ( Tony et al. 2023 ) , CWEval ( Peng et al. 2025 ) , and recent efforts such as SecCodeBench 2025 . Figure 1 : Task illustration of SecureAgentBench . Table 1 : Comparison against prior works. “Repo”: repository-level; “C/E”: completion or editing; “Real”: real vulnerabilities; “Intro.": vulnerability introduction context; “Func.”: functional evaluation; “New”: newly introduced vulnerabilities; , , : no, partial, full support. Name Time Task Context Evaluation (yymm) Repo. C/E Real Intro. Func. New CodeLMSec ( Hajipour et al. 2024a ) 2302 C LLMSecEval ( Tony et al. 2023 ) 2303 C SecCodePLT ( Yang et al. 2024c ) 2401 C CWEval ( Peng et al. 2025 ) 2501 C CyberSecEval ( Bhatt et al. 2023 ) 2504 C SecRepoBench ( Dilgren et al. 2025 ) 2504 C SafeGenBench ( Li et al. 2025c ) 2506 C SecCodeBench ( SecCodeBench 2025 ) 2507 C SecureAgentBench (Ours) 2509 E Limitations of Existing Benchmarks. While existing benchmarks have made valuable progress, several critical limitations remain, as summarized in Table 1 , which render them inadequate and necessitate our work. Specifically, ❶ Task Form . Real-world software maintenance typically occurs at the repository level, where developers need to handle and edit multiple files and consider project-wide dependencies. In contrast, most benchmarks define their tasks at the function level, restricting the context to only a few preceding lines (e.g., import statements and a function signature), with agents then completing the remaining code. ❷ Context Alignment . Most existing benchmarks are constructed by synthesizing simplified coding scenarios from coarse-grained vulnerability descriptions (e.g., CWE 1 1 1 CWE (Common Weakness Enumeration) is a catalog of common software and hardware security weaknesses. Each CWE item summarizes similar vulnerabilities into one category. categories) instead of directly leveraging vulnerabilities from real-world code repositories. Even when using genuine data (e.g., Dilgren et al. 2025 ) from real vulnerability databases, these works typically use coding contexts (e.g., repository structure, APIs, and documentation) from the time of vulnerability fixing or discovery, rather than from the point of introduction. This produces mismatched contexts that fail to capture how vulnerabilities were originally introduced by humans and thereby undermines the realism of evaluation. ❸ Evaluation . The evaluation scope of prior benchmarks remains limited. Functional correctness, which is a prerequisite for meaningful security evaluation ( Vero et al. 2025 ) , is rarely considered in them. More importantly, most benchmarks focus only on predefined vulnerability categories, neglecting the fact that code agents may introduce entirely new security risks and therefore lack mechanisms to detect them. Our Solution. In this paper, we propose SecureAgentBench , a new benchmark for evaluating code agents’ capability in secure code generation, which addresses the limitations of prior benchmarks by providing more realistic and challenging scenarios. As illustrated in Figure 1 , the benchmark incorporates three key characteristics: ❶ Realistic Task Form . Rather than function completion within a limited context, we adopt a task form that is more challenging yet more faithful to real-world software maintenance ( Jimenez et al. 2024 ) : given a programming requirement in natural language, a code agent is expected to implement it by editing multiple files across the repository. ❷ Aligned Context . Our benchmark leverages real-world vulnerabilities documented in public databases ( OSS-Fuzz Project ) , and further employs a two-stage method to precisely identify when each vulnerability was introduced and extract the corresponding context. In this way, it constructs security-sensitive coding scenarios that are both genuine to real-world cases and faithful to the context of how vulnerabilities were originally introduced. ❸ Comprehensive Evaluation . We evaluate both the functionality and security of code generated by agents. Functionality is assessed by differential testing ( McKeeman 1998 ) , which compares the execution behavior of the generated code with the developer’s reference implementation. For security, we use proof-of-concept (PoC) 2 2 2 Proof-of-concept (PoC) is a program that can confirm the presence of a specific vulnerability. In this paper, expressions like “PoC crashes (due to this vulnerablity)” or “vulnerability is triggered ” both indicate that we use the PoC to validate that the specific vulnerability exists in the code