Prompt to Pwn: Automated Exploit Generation for Smart Contracts
Zeke Xiao, Yuekang Li, Qin Wang, Shiping Chen · 2025
Read Prompt to Pwn: Automated Exploit Generation for Smart Contracts by Zeke Xiao, Yuekang Li, Qin Wang, Shiping Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01371.txt.
Introduction
Xiao and co-authors test whether language models can write working attacks against smart contracts, in a paper published in 2025. Smart contracts guard digital money and cannot be patched once live. Past work used these models to spot flaws; generating a full exploit is a step beyond. The team's framework ties model-written attack code to a standard testing stack. Exploits are compiled, run, and validated against real behavior, not judged by eye.
They evaluated five recent models across eight common flaw types. The test set curates 38 real incident exploits. Aids like prompt restructuring, compiler feedback, and test templates lifted success. Frontier models often produced working proofs for single-contract flaws. That finding cuts both ways: attackers gain a tool, but so do auditors before deployment. The paper maps the current frontier of automated exploitation.
Catalog record
| CREATOR | Zeke Xiao, Yuekang Li, Qin Wang, Shiping Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01371
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt to Pwn: Automated Exploit Generation for Smart Contracts Zeke Xiao, Yuekang Li, Qin Wang, Shiping Chen Abstract We explore the feasibility of using LLMs for Automated Exploit Generation (AEG) against vulnerable smart contracts. We present ReX , a framework integrating LLM-based exploit synthesis with the Foundry testing suite, enabling the automated generation and validation of proof-of-concept (PoC) exploits. We evaluate five state-of-the-art LLMs (GPT-4.1, Gemini 2.5 Pro, Claude Opus 4, DeepSeek, and Qwen3 Plus) on both synthetic benchmarks and real-world smart contracts affected by known high-impact exploits. Our results show that modern LLMs can reliably generate functional PoC exploits for diverse vulnerability types, with success rates reaching up to 92%. Notably, Gemini 2.5 Pro and GPT-4.1 consistently outperform others in both synthetic and real-world scenarios. We further analyze factors influencing AEG effectiveness, including model capabilities, contract structure, and vulnerability types. We also collect the first curated dataset of real-world PoC exploits to support future research. Introduction Once smart contracts were deployed, even a small vulnerability may be permanently exploitable, resulting in substantial financial losses. We show an example. In February 2025, a smart contract exploit on Bybit’s Safe multi-signature wallet allowed attackers to upgrade the contract implementation and drain 1.5 billion US dollars ( Rommen and Sehmbi 2025 ) . To address the security issues, many vulnerability detection tools have been proposed, including Slither ( Feist, Grieco, and Groce 2019 ) , Mythril ( Nikolić et al. 2018 ) , and Oyente ( Luu et al. 2016 ) , but existing static and symbolic analysis tools suffer from low accuracy and limited scalability and perform poorly in practice ( Sendner et al. 2024 ) . Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, including code-related tasks such as generation ( Laurie et al. 2022 ) , summarization, and bug fixing ( Pearce et al. 2022 ) . Not surprisingly, LLMs also hold strong potential for assisting in the identification of smart contract vulnerabilities ( Chen et al. 2025 ; David et al. 2023 ; Xiao et al. 2025 ) . We step further to explore their effectiveness in generating verifiable exploits. We focus on a few research questions: • RQ1: How well can LLMs perform automated exploit generation (AEG) for smart contract vulnerabilities on both benchmarks and real-world attack data? • RQ2: What factors affect AEG effectiveness: contract properties (e.g., size, complexity), vulnerability types, or specific prompt design? • RQ3: What defensive practices can mitigate such LLM-driven threats? To investigate whether LLMs can move beyond detection to AEG and validate executable proof-of-concept (PoC) exploits for smart contracts, we propose ReX , a new framework that integrates LLM-based exploit synthesis with the Foundry testing environment, enabling end-to-end exploit generation, compilation, execution, and verification. We evaluate five most latest LLMs (GPT-4.1, Gemini 2.5 Pro, Claude Opus 4, DeepSeek-R1, Qwen3 Plus) on their ability to generate full exploit contracts and test scripts, targeting eight distinct real-world vulnerability types (e.g., reentrancy, integer overflows, improper access control). To further improve output automation, we design three optimization techniques, including prompt refactoring, compiler feedback loops, and templated test harness generation. These enhancements significantly improve success rates for underperforming models. Our results show that: • GPT-4.1, Gemini 2.5 Pro, and Claude Opus 4 achieve success rates 80%+ on most vulnerability categories, with GPT-4.1 peaking at 92.5% on arithmetic bugs. • DeepSeek-R1 and Qwen3 Plus produce valid exploits in 60–70% of cases but show inconsistency across more complex vulnerability types. • Over 76% of generated artifacts pass compilation and test execution end-to-end without human intervention. In addition, we curate the first dataset 1 1 1 provided upon request. of manually written PoC exploits targeting real-world contracts, consisting of 38+ audited attack cases from expert security researchers. Finally, we validate LLM-generated exploits on smart contracts that suffered from historical high-impact attacks. In 4 out of 10 cases, Gemini 2.5 Pro was able to independently rediscover valid exploits. It not only demonstrated autonomous exploit discovery capabilities, but also exhibited expert-like reasoning strategies similar to those employed by seasoned smart contract auditors. To facilitate reproducible research and open science, we provide the code, including the detailed prompts and raw experiment data as supplementary materials, and will open-source them upon paper acceptance. Technical Warmups Smart contract vulnerabilities. Smart contracts are vulnerable to bugs and exploits ( Tolmach et al. 2021 ) . Once deployed, contracts cannot be altered, leaving any flaws permanently exposed. To standardize and detect such issues, the SWC Registry ( swc 2018 ) categorizes common vulnerabilities like reentrancy (SWC-107), integer overflows (SWC-101), and access control flaws (SWC-105), forming the basis for many tools and benchmarks. As decentralized applications grow in complexity, attackers increasingly leverage sophisticated techniques such as arbitrage ( Torres, Steichen et al. 2019 ; Cernera et al. 2023 ) , transaction order manipulation ( Eskandari, Moosavi, and Clark 2019 ) , and proxy-based upgrade attacks ( Meisami and Bodell III 2023 ) . LLMs. We evaluate five latest LLMs in this work. GPT-4.1 (OpenAI) is known for strong reasoning and consistent performance across code and language tasks. Gemini 2.5 Pro (Google) handles text, code, and images well, matching GPT-4-class performance. Claude Opus 4 (Anthropic) offers long-context reasoning and is suited for safety-critical tasks. DeepSeek-V2 delivers high performance in bilingual and code tasks. Qwen3 Plus (Alibaba) provides strong code and dialogue capabilities, especially in English and Chinese. Contract testing suite. We use Foundry, a Rust-based toolchain for smart contract development and testing ( Fou 2025 ) . Foundry supports dependency management, compilation, deployment, and testing. Its built-in test runner (forge test) enables efficient, large-scale validation of Solidity exploits, outperforming tools like Hardhat and Truffle. Static analytic tools. We use two. SUMO is a static analysis tool for smart contracts that estimates code understandability through a composite complexity score ( Barboni, Morichetta, and Polini 2022 ) . It captures Solidity-specific features like control-flow depth, nesting, and branching. Slither is another widely used static analysis tool for detecting vulnerabilities and computing code metrics in Solidity contracts ( Feist, Grieco, and Groce 2019 ) . It provides accurate and fine-grained structural insights. ReX and Datasets Figure 1: Overview of the ReX framework As shown in Figure 1 , ReX is a fully automated pipeline and proceeds in five steps: Step 1: data preprocessing. We first clean the input smart contracts by removing comments and non-functional content to eliminate noise and avoid misleading the LLMs. We ensure that the LLM focuses only on the core contract logic when generating the exploit. Step 2: script generation. Gven a vulnerable smart contract, through carefully designed prompt, the LLM will generate two related Foundry scripts: the first is an exploit contract designed to exercise the vulnerable code path; while the second is a test contract that validates whether the exploit succeeds under some specific conditions. We enabled the LLMs to iteratively optimize its prompts ( Chen et al. 2025 ; Grubisic et al. 2024 ; Sepidband et al. 2025 ) . As a result, it is capable of generating exploit and test scripts with intact import paths, designed to be as compilable and executable as possible. At the same time, we enabled the model’s reasoning mode by guiding it