BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems

Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyu · 2025

Read BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems by Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15216.txt.

Description

Andy Zhang and a large team priced AI agents as hackers and guards in a 2025 paper. Their test bed, BountyBench, sets agents loose on twenty five real software systems with live code bases. Tasks span the life of a flaw: spot a fresh one, break in through a given one, or patch one shut.

Forty real bug bounties, worth from ten dollars to more than thirty thousand, anchor the tests. A new grading mark keeps scoring local and fair across flaw types. Ten agents were run, from top commercial tools to custom builds. Guides that leak more or less about the flaw tune how hard each task is. The authors tally each agent's dollar impact on attack and defense.

Catalog record

CREATORAndy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Y. Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, Sara Hong, Nardos Demilew, Shivatmica Murgai, Jason Tran, Nishka Kacheria, Ethan Ho, Denis Liu, Lauren McLane, Olivia Bruvik, Dai-Rong Han, Seungwoo Kim, Akhil Vyas, Cuiyu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE242 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15216

Browse the full library · About the Tower of Babel