Cybersecurity AI: The World’s Top AI Agent for Security Capture-the-Flag (CTF)

Víctor Mayoral-Vilches, Luis Javier Navarrete-Lozano, Francesco Balassone, María Sanz-Gómez, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres and Vanesa Turiel · 2025

Read Cybersecurity AI: The World’s Top AI Agent for Security Capture-the-Flag (CTF) by Víctor Mayoral-Vilches, Luis Javier Navarrete-Lozano, Francesco Balassone, María Sanz-Gómez, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres and Vanesa Turiel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02654.txt.

Introduction

Víctor Mayoral-Vilches and six co-authors report a banner year for their security AI in hacking contests. Capture the Flag events test teams on real style security puzzles under time pressure. The authors ask whether such competitions are becoming obsolete as AI agents improve.

Their system, Cybersecurity AI, is described as topping multiple major events in 2025. The authors report it outranked thousands of human teams again and again. Whether or not contests fade, results like these force a rethink of training, defense, and how security skill is measured. Scores like these will push schools and firms to train staff in new ways for a new kind of foe.

Catalog record

CREATORVíctor Mayoral-Vilches, Luis Javier Navarrete-Lozano, Francesco Balassone, María Sanz-Gómez, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres and Vanesa Turiel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-02654

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cybersecurity AI: The World’s Top AI Agent for Security Capture-the-Flag (CTF) Víctor Mayoral-Vilches, Luis Javier Navarrete-Lozano, Francesco Balassone, María Sanz-Gómez, Cristóbal R. J. Veas Chavez, Maite del Mundo de Torres and Vanesa Turiel Abstract Are Capture-the-Flag competitions obsolete? In 2025, Cybersecurity AI (CAI) systematically conquered some of the world’s most prestigious hacking competitions, achieving Rank #1 at multiple events and consistently outperforming thousands of human teams. Across five major circuits—HTB’s AI vs Humans , Cyber Apocalypse (8,129 teams), Dragos OT CTF, UWSP Pointer Overflow, and the Neurogrid CTF showdown—CAI demonstrated that Jeopardy-style CTFs have become a solved game for well-engineered AI agents. At Neurogrid, CAI captured 41/45 flags to claim the $50,000 top prize; at Dragos OT, it sprinted 37% faster to 10K points than elite human teams; even when deliberately paused mid-competition, it maintained top-tier rankings. Critically, CAI achieved this dominance through our specialized alias1 model architecture, which delivers enterprise-scale AI security operations at unprecedented cost efficiency and with augmented autonomy—reducing 1B token inference costs from $5,940 to just $119, making continuous security agent operation financially viable for the first time. These results force an uncomfortable reckoning: if autonomous agents now dominate competitions designed to identify top security talent at negligible cost, what are CTFs actually measuring? This paper presents comprehensive evidence of AI capability across the 2025 CTF circuit and argues that the security community must urgently transition from Jeopardy-style contests to Attack & Defense formats that genuinely test adaptive reasoning and resilience—capabilities that remain uniquely human, for now. Event Area CTF Style Field size Peak / Final rank Flags / Points Active window AI vs Humans CTF IT Jeopardy 163 teams #6 (3h) / #1 AI (#21) 19/20 flags; 15.9k pts 3 h Cyber Apocalypse CTF 2025 IT Jeopardy 8,129 teams #22 (3h) / #859 30/77 flags; 19,275 pts 3 h Dragos OT CTF 2025 OT Jeopardy >1,200 teams #1 (7–8h) / #6 32/34; 18,900 pts 24 h UWSP Pointer Overflow CTF 2025 IT Jeopardy 635 teams #14 (24h) / #21 58 solves; 11,500 pts 24 h Neurogrid CTF IT Jeopardy 155 teams #1 (6h) / #1 41/45 flags; $50k prize 48 h Table 1 : Cross-event snapshot. Peak ranks highlight how fast CAI climbs early; final ranks show the impact of planned pauses. 1 Introduction In 2025, the cybersecurity landscape witnessed a paradigm shift: autonomous AI agents began systematically defeating elite human teams in Capture-the-Flag (CTF) competitions. The DARPA AI Cyber Challenge allocated $29.5 million in prizes for AI-powered vulnerability detection [ 4 ] , while specialized competitions like “AI vs Human CTF” saw AI teams achieve 95% solve rates compared to 71% for top human teams [ 11 ] . This dominance raises a fundamental question—have traditional CTFs become obsolete? This paper presents the results obtained with Cybersecurity AI (CAI) [ 1 ] , a popular AI Security framework to build autonomous agents that achieved unprecedented success across five major international CTF competitions in 2025, including winning the prestigious $50,000 Neurogrid CTF prize. Our contributions are threefold: 1. Empirical Evidence : We document CAI’s systematic dominance across diverse CTF formats—Rank #1 at Dragos OT and Neurogrid, #1 at HTB “AI vs Humans” in the AI category, and consistently solving challenges 37% faster than elite human teams. 2. CTF Format Analysis : We analyze how current Jeopardy-style CTFs have become computational exercises rather than genuine security skill assessments, revealing fundamental gaps between competition formats and real-world cybersecurity challenges. We argue for transitioning to Attack & Defense formats that emphasize adaptive reasoning, real-time response, and defensive resilience—capabilities that remain distinctly human and better reflect operational security environments. 3. Novel Architecture for Economic Autonomy : We introduce a specialized model architecture, which delivers enterprise-scale AI security operations at unprecedented cost efficiency and with augmented autonomy. Leveraging alias1 as the base model with dynamic entropy-based selection of support models, we achieve a 98% cost reduction compared to other state-of-the-art deployments, reducing 1B token inference costs from $5,940 to $119, making continuous security agent operation financially viable for the first time. Critically, the breadth of solved challenge categories validates our minimal-intervention architecture: rather than requiring extensive human guidance or specialized modules for each challenge type, the core alias1 model with selective support generalizes effectively across the cybersecurity domain, enabling unprecedented operational autonomy in CAI’s security operations. 1.1 Evolution from Augmentation to Autonomy The journey from AI-assisted to AI-dominated security began with tools like PentestGPT [ 5 , 13 ] , which augmented human pentesters but required constant supervision. The landscape shifted dramatically with autonomous agents capable of independent operation. CAI [ 11 ] derives from PentestGPT [ 5 ] and emerged from PhD research [ 13 ] with a vision: create a framework to democratize the access to AI Security. The open-source framework distinguishes between mere automation and true autonomy [ 12 ] . This distinction proved critical—while other tools excel at specific tasks, CAI’s planning and reasoning capabilities enable it to navigate entire CTF competitions in an automated manner, with humans teleoperating via Human-In-The-Loop (HITL) capabilities. The shift from augmentation to replacement accelerated in 2024-2025. DARPA’s AI Cyber Challenge demonstrated that agents could autonomously discover and patch vulnerabilities at scale [ 4 ] . Hack The Box’s inaugural “AI vs Human” competition saw AI teams achieving 95% solve rates [ 8 ] . By 2025, the question wasn’t whether AI could compete with humans, but how long humans could remain competitive. This evolution exposes a fundamental problem with current evaluation methods. Jeopardy CTFs—designed to test human ingenuity through discrete challenges—have become speed tests that favor parallel processing and 24/7 operation. However, excelling at CTF challenges does not equate to achieving cybersecurity superintelligence—the hypothetical capability of AI systems exceeding the best human security experts across all domains. CTF dominance represents a narrow optimization: systems trained to capture flags may excel at pattern matching and exploitation of known vulnerability classes, yet remain brittle when confronted with novel attack surfaces or adversarial defenders. Recent work by Balassone et al. [ 3 ] argues convincingly that Attack & Defense formats better capture real-world security dynamics. Our results provide the empirical evidence: when AI agents consistently dominate traditional CTFs, the format itself becomes the limitation, and the metric becomes meaningless as a proxy for genuine security capability. 1.2 Report Structure The remainder of this report is organized as follows: Section 2 presents comprehensive results across five major CTF competitions, demonstrating systematic AI dominance with specific competition analyses: HTB “AI vs Humans” (Section 2.1 ), Cyber Apocalypse CTF 2025 (Section 2.2 ), Dragos OT CTF (Section 2.3 ), UWSP Pointer Overflow (Section 2.4 ), and Neurogrid AI Showdown (Section 2.5 ). Section 3 provides in-depth discussion including system architecture and configuration, the meaningfulness of Jeopardy CTFs, implications for OT security, limitations, ethical considerations, and relationship to CAIBench. Section 4 delivers the verdict on Jeopardy CTF obsolescence as they are, and proposes Attack & Defense formats as the future of competitive security assessment in an AI-dominated era. 2 Results: Five CTF Dominations 80 85 90 95 100 AI vs Humans Cyber Apocalypse Dragos OT Pointer Overflow Neurogrid Elite 1% Top 5% Top 10% #21 of 163 #859 of 8129 #6 of 1200+ #21 of 635 #1

Read the complete text