Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents

María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres · 2025

Read Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents by María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24317.txt.

Description

María Sanz-Gómez and five co-authors present CAIBench, a meta benchmark for evaluating AI agents in cybersecurity. Security spans many linked domains, so existing benchmarks that test isolated skills miss integrated, job relevant performance. The authors also find that a model's stored security knowledge does not imply real attack and defense ability.

CAIBench integrates five evaluation categories with over ten thousand instances, including jeopardy style and attack and defense capture the flag events and cyber range exercises. Its modular design lets teams evaluate both language models and agents across offensive and defensive tasks. The goal is a meaningful measure of how useful these systems are for real security work.

Catalog record

CREATORMaría Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24317

Browse the full library · About the Tower of Babel