Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents
María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres · 2025
Read Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents by María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24317.txt.
Description
María Sanz-Gómez and five co-authors present CAIBench, a meta benchmark for evaluating AI agents in cybersecurity. Security spans many linked domains, so existing benchmarks that test isolated skills miss integrated, job relevant performance. The authors also find that a model's stored security knowledge does not imply real attack and defense ability.
CAIBench integrates five evaluation categories with over ten thousand instances, including jeopardy style and attack and defense capture the flag events and cyber range exercises. Its modular design lets teams evaluate both language models and agents across offensive and defensive tasks. The goal is a meaningful measure of how useful these systems are for real security work.
Catalog record
| CREATOR | María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24317