Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model
Duc Huy Le, Rolf Stadler · 2025
Read Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model by Duc Huy Le, Rolf Stadler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06539.txt.
Introduction
Defending a computer network against attacks, in a benchmark game called CAGE-2, is modeled formally by Duc Huy Le and one co-author in a 2025 cybersecurity paper. Many learning methods have been tried on this benchmark, but without a formal model it is hard to know what the best possible defense looks like.
The authors rebuild CAGE-2 as a partially observable Markov decision process, a standard framework for acting under uncertainty about the true state. That model lets them define an optimal defender strategy and a practical way to learn it. Their method, BF-PPO, extends a known learning algorithm and uses a particle filter to tame the huge state space. In the CybORG test environment it beats CARDIFF, the top-ranked earlier method, on both defense quality and training time.
Catalog record
| CREATOR | Duc Huy Le, Rolf Stadler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06539
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model Duc Huy Le, Rolf Stadler Abstract CAGE-2 is an accepted benchmark for learning and evaluating defender strategies against cyberattacks. It reflects a scenario where a defender agent protects an IT infrastructure against various attacks. Many defender methods for CAGE-2 have been proposed in the literature. In this paper, we construct a formal model for CAGE-2 using the framework of Partially Observable Markov Decision Process (POMDP). Based on this model, we define an optimal defender strategy for CAGE-2 and introduce a method to efficiently learn this strategy. Our method, called BF-PPO, is based on PPO, and it uses particle filter to mitigate the computational complexity due to the large state space of the CAGE-2 model. We evaluate our method in the CAGE-2 CybORG environment and compare its performance with that of CARDIFF, the highest ranked method on the CAGE-2 leaderboard. We find that our method outperforms CARDIFF regarding the learned defender strategy and the required training time. I Introduction 0 0 footnotetext: DISTRIBUTION STATEMENT A: Approved for public release; dissemination unlimited. This research is supported by the Defense Advanced Research Project Agency (DARPA) through the CASTLE program under Contract No.W912CG23C0029. The views, opinions, and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Traditionally, systems for intrusion detection and response have relied on rule sets that trigger alarms (e.g., [ 1 , 2 ] ). The rule sets have been defined and maintained by human experts. The increasing complexity and rapid changes of digital services and infrastructures have made the maintenance of these rule sets a challenging and a time-consuming task. As a response, research efforts into automated cyberdefence have started based on the idea that defender strategies can be dynamically learned and then executed by defender agents with minimal human intervention. One can say that the rules are no longer defined by humans, but automatically constructed from observing systems under attack. Over the last decade, various learning approaches have been proposed for automated cyberdefence, including those based on reinforcement learning [ 3 , 4 , 5 ] , stochastic modelling [ 6 , 7 , 8 ] , game theory [ 9 , 10 , 11 ] , and most, recently, causal inference [ 12 , 13 , 14 ] . Currently, the most popular benchmark environment for learning and evaluating defender strategies is the Cyber Autonomy Gym for Experimentation Challenge 2 (CAGE-2) [ 15 ] . It is based on a scenario where a defender agent protects an IT infrastructure against different types of attacks (Fig. 1 ). CAGE-2 includes a simulation environment, named CybORG [ 16 ] , in which defender agents can be trained and evaluated. Fig. 1 : The network topology of CAGE-2 scenario A large number of defender methods for CAGE-2 have been proposed and published. A CAGE-2 leaderboard ranks the solutions according to a score that measures the effectiveness of the defenders against the attacks. The solutions on the leaderboard are based on heuristic approaches, and none of them is built on a formal model. As a consequence, it is not clear to which extent these solutions are optimal or close to optimal. (There is a recent work that uses a formal model, which we discuss in § II ). In this paper, we formalise the CAGE-2 scenario and develop a formal model using the framework of Partially Observable Markov Decision Process (POMDP) [ 17 ] . The model is obtained from studying the documentation and the source code of CAGE-2, as well as from conducting experiments in the CybORG environment. The model allows us to define an optimal defender strategy for CAGE-2. To learn this strategy, we propose a learning method, which we call Belief Filter Policy Proximal Optimisation (BF-PPO). It is based on the state-of-the-art reinforcement learning algorithm Policy Proximal Optimisation (PPO) [ 18 ] , and it uses the concept of particle filter [ 19 ] to address the computational complexity arising from the large state space of the CAGE-2 model. We evaluate our method and compare its performance with CARDIFF [ 20 ] , the method with the highest rank on the CAGE-2 leaderboard. We find that our method outperforms CARDIFF in terms of the learned defender strategy. Also, our method converges faster, requiring fewer training episodes. The contributions of this paper are: • We present a formal model of the CAGE-2 scenario using the POMDP framework (presented in § IV-B ) and formally define an optimal defender strategy for CAGE-2 (presented in § IV-C ). • We present BF-PPO, a solution method that iteratively approximates the optimal strategy based on PPO and particle filter (presented in § IV-D ) • We evaluate BF-PPO in the CybORG environment and show that it outperforms the state-of-the art method CARDIFF, the top performer on the CAGE-2 leaderboard (presented in § V ). II Related Work The CAGE-2 challenge has attracted much attention in the cybersecurity community. More than 35 solution methods have been developed to learn defender strategies for the CAGE-2 scenario, for example [ 15 , 21 , 22 , 3 , 23 , 24 , 12 , 25 ] . All of these methods, except for [ 12 ] , rely on heuristic techniques and are not based on a formal model. As a consequence, it cannot be shown that any of these approaches lead to an optimal defender strategy. In contrast, we present in this paper a formal model of the CAGE-2 scenario, based on which we define an optimal strategy for the defender, and develop a solution to approximate this strategy. The only published method that is based on a formal model of the CAGE-2 scenario is included in [ 12 ] . The study presents a structural causal model and proposes a defender strategy using Monte Carlo Planning. The method, however, is an online solution and is therefore of a different type than the above cited works, which all use offline training. This paper also uses offline training, which allows us to directly compare its performance with those works. The POMDP framework has been recently used by many researchers to study intrusion detection and response [ 26 , 6 , 7 , 8 , 27 , 28 ] . These works consider scenarios that are much simpler than CAGE-2 and the corresponding state spaces are much smaller. In order to address the computational complexity that results from larger state spaces, this paper uses the concept of particle filter for state estimation. III Intrusion Response Use Case: CAGE-2 The CAGE-2 scenario presents a cybersecurity use case in which a defender defends an IT infrastructure against an attacker with a fixed strategy. The network contains 11 hosts, divided into 3 subnets (see Fig. 1 ). The attacker aims to access the target host located in the operational subnet and to disrupt the services it provides. After compromising one of the clients in the client subnet , the attacker tries to break into one of the hosts in the enterprise subnet , before finally attacking the target host. To achieve this