EVMbench : Evaluating AI Agents on Smart Contract Security
Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins, [1em] OpenAI · 2026
Read EVMbench : Evaluating AI Agents on Smart Contract Security by Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins, [1em] OpenAI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04915.txt.
Introduction
Justin Wang and colleagues introduce EVMbench, a benchmark testing whether AI agents can detect, patch, and exploit vulnerabilities in smart contracts on public blockchains. Built from 117 curated vulnerabilities across 40 repositories, it grades agents on tests and blockchain state inside a local Ethereum environment. Frontier agents proved able to find and exploit vulnerabilities in live blockchain instances from end to end. The team releases code, tasks, and tooling for continued measurement of agent security capabilities and risks.
Catalog record
| CREATOR | Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins, [1em] OpenAI |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93452 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04915
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EVMbench : Evaluating AI Agents on Smart Contract Security Justin Wang, Andreas Bigger, Xiaohai Xu, Justin W. Lin, Andy Applebaum, Tejal Patwardhan, Alpin Yukseloglu, Olivia Watkins, [1em] OpenAI Abstract Smart contracts on public blockchains now manage large amounts of value, and vulnerabilities in these systems can lead to substantial losses. As AI agents become more capable at reading, writing, and running code, it is natural to ask how well they can already navigate this landscape, both in ways that improve security and in ways that might increase risk. We introduce EVMbench , an evaluation that measures the ability of agents to detect, patch, and exploit smart contract vulnerabilities. EVMbench draws on 117 curated vulnerabilities from 40 repositories and, in the most realistic setting, uses programmatic grading based on tests and blockchain state under a local Ethereum execution environment. We evaluate a range of frontier agents and find that they are capable of discovering and exploiting vulnerabilities end-to-end against live blockchain instances. We release code, tasks, and tooling to support continued measurement of these capabilities and future work on security. 1 Introduction Figure 1 : Overview of the three evaluation modes in our benchmark. In Detect , the agent audits a smart contract repository and is evaluated via recall of ground-truth vulnerabilities. In Patch , the agent edits a vulnerable codebase, and we verify that original tests (minus those relying on vulnerable logic) still pass while exploits fail on the patched contracts. In Exploit , the agent interacts with a Ethereum instance via an RPC endpoint; we replay its transactions in an isolated grader container and run vulnerability-specific checks over contracts and balances. In the past decade, public blockchains [ 9 , 3 , 18 ] have progressed from experimental ledgers to mature financial infrastructure. Stablecoins settle trillions of dollars of transactions every month and operate at a scale comparable to the world’s largest payment networks. More than $100B of assets routinely sit in open-source smart contracts and more than $300B are held in stablecoins [ 7 ] . As AI models get better at finding bugs in software, a growing portion of the assets on crypto rails could be at risk. While the speed and irreversibility of smart contracts enable seamless financial transactions, the same properties also mean that exploits have immediate and immense consequences. Much like how malicious geopolitical actors have successfully self-financed through large-scale crypto hacks [ 14 ] , sufficiently capable AI systems could acquire substantial financial resources by exploiting smart contracts. This is closely related to the high-risk cyber scenarios described in OpenAI’s Preparedness Framework [ 10 ] , where advanced models enable running end-to-end cyber operations, including the discovery and exploitation of vulnerabilities. As capabilities advance, it becomes increasingly important to measure and monitor the development of skills relevant to smart contract security, both to inform deployment decisions and to guide defensive efforts. Additionally, smart contract security is an economically relevant and challenging domain which provides a uniquely clean evaluation surface, since blockchains support deterministic execution, programmatic verification, and isolated environments that do not sacrifice realism. However, existing work focuses on traditional cybersecurity vulnerabilities [ 20 , 15 , 22 ] or targets narrow subtasks or simplified environments [ 21 ] rather than the full cycle of discovery, mitigation, and exploitation of real vulnerabilities. To study these capabilities, we introduce a framework and task suite that evaluates the ability of agents to detect, patch, and exploit fund-draining vulnerabilities in production-grade smart contract environments. A central skill evaluated is comprehensive coverage: agents must find and address all relevant vulnerabilities in a codebase, rather than stopping after a single success. Exploit capabilities are particularly concerning given the immutable nature of blockchains, so our framework connects the agent to a local Ethereum chain and grades its behavior via on-chain events and balances. In this mode, the agent must realize an end-to-end exploit against the blockchain, causing state to update as it would under a real attack. Our contributions are as follows: • We introduce EVMbench , a harness and task suite that evaluates agents on detecting, patching, and exploiting critical smart contract vulnerabilities. In the Patch and Exploit modes, grading is fully programmatic, and in the Exploit mode, grading is based on on-chain events and balance deltas. • We develop a Rust-based re-execution framework that replays agent transactions against a local Ethereum node, enabling fast and reproducible Exploit evaluation while guarding against cheating. • We curate high-severity vulnerabilities from real audits, provide oracle patches and exploits for verification, and create evaluation setups that preserve realistic codebases, constraints, and attacker incentives. • We evaluate frontier agents in all three modes, compare their performance, and provide qualitative analysis of rollouts and failure modes. • We open source our dataset and evaluation harness to support further measurement of these capabilities and encourage future work on security. 2 Background Figure 2 : An example of a vulnerable smart contract. In this section we briefly review smart contracts and the Ethereum Virtual Machine (EVM), how smart contract exploits and audits are conducted, and why blockchains form a compelling evaluation environment. 2.1 Smart Contracts and the EVM On Ethereum and EVM-compatible chains, smart contracts are most commonly written in high-level languages such as Solidity, then compiled down to low-level bytecode and deployed at a fixed address. A user interaction looks roughly like a remote procedure call: a transaction specifies which contract address to call, which function to invoke, the arguments to that function, and (optionally) how much currency to send along. In practice, these programs underpin a wide range of applications, including automated exchanges [ 2 ] and lending markets [ 6 , 1 ] , that collectively secure billions of dollars in user funds. The Ethereum Virtual Machine [ 16 ] is the execution environment for these contracts. Conceptually, it is a virtual CPU that processes one transaction at a time in a fully deterministic way: given the same starting state and sequence of transactions, every node in the network computes the same result. The EVM maintains a global state that tracks account balances, deployed contract code, and each contract’s persistent storage, as well as per-transaction data such as intermediate call frames and emitted logs. 2.2 Smart Contract Exploits and Audits in Practice Due to the immutable and permissionless nature of blockchains, security failures can be extremely costly. Unlike many traditional cybersecurity vulnerabilities, where impact can be contained or rolled back, smart contract vulnerabilities often translate into instant, irreversible fund loss. To surface these issues before deployment, projects run competitive audit contests, in which independent auditors race to discover impactful vulnerabilities within a fixed time window. However, high-impact exploits remain common, highlighting the need for more capable preventative vulnerability detection. The execution environment is adversarial and high-speed: once a profitable attack path is discovered, it can be executed and generalized quickly. Ethereum in particular is often described as a “dark forest” where, because pending transactions are publicly visible, automated searchers and bots continuously scan for opportunities, and exploitable patterns may be front-run or copied by others almost instantly [ 13 ] . 2.3 Blockchains as an Evaluation Substrate For AI agents, this “dark forest” setting is both a challenging deployment environment and a