NASimJax: GPU-Accelerated Policy Learning Framework for Penetration Testing

Raphael Simon, José Carrasquel, Wim Mees, Pieter Libin · 2026

Read NASimJax: GPU-Accelerated Policy Learning Framework for Penetration Testing by Raphael Simon, José Carrasquel, Wim Mees, Pieter Libin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19864.txt.

Introduction

Simon and co-authors present NASimJax, a fast simulator for training agents to test network security. Penetration testing is a step-by-step task with partial views and many possible actions. Older simulators run on CPUs and cover narrow scenarios. NASimJax is built in JAX and makes diverse, solvable network tasks. It runs up to 80 times faster than earlier tools. Baselines include PPO and PQN agents. Training on sparser networks gave a natural curriculum and better transfer to denser ones.

Catalog record

CREATORRaphael Simon, José Carrasquel, Wim Mees, Pieter Libin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56463
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NASimJax: GPU-Accelerated Policy Learning Framework for Penetration Testing Raphael Simon, José Carrasquel, Wim Mees, Pieter Libin Abstract Penetration testing—the practice of simulating cyberattacks to identify vulnerabilities—is a complex sequential decision-making task that is inherently partially observable and features large action spaces. Training reinforcement learning (RL) policies for this domain faces a fundamental bottleneck: existing simulators are too slow to train on realistic network scenarios at scale, resulting in policies that fail to generalize. We present NASimJax, a complete JAX-based reimplementation of the Network Attack Simulator (NASim), achieving up to 100 × \times higher environment throughput than the original simulator. By running the entire training pipeline on hardware accelerators, NASimJax enables experimentation on larger networks under fixed compute budgets that were previously infeasible. We formulate automated penetration testing as a Contextual POMDP and introduce a network generation pipeline that produces structurally diverse and guaranteed-solvable scenarios. Together, these provide a principled basis for studying zero-shot policy generalization. We use the framework to investigate action-space scaling and generalization across networks of up to 40 hosts. We find that Prioritized Level Replay better handles dense training distributions than Domain Randomization, particularly at larger scales, and that training on sparser topologies yields an implicit curriculum that improves out-of-distribution generalization, even on topologies denser than those seen during training. To handle linearly growing action spaces, we propose a two-stage action decomposition (2SAS) that substantially outperforms flat action masking at scale. Finally, we identify a failure mode arising from the interaction between Prioritized Level Replay’s episode-reset behaviour and 2SAS’s credit assignment structure. NASimJax thus provides a fast, flexible, and realistic platform for advancing RL-based penetration testing. 1 Introduction Cybersecurity has become one of the most critical domains in modern society, underpinning healthcare, banking, and energy infrastructure. Among the tools used to proactively defend these systems is penetration testing — the authorised practice of simulating attacks to identify vulnerabilities and misconfigurations in IT networks ( 25 ) . While effective, penetration testing is laborious and expensive, requiring seasoned security experts to reason over long decision horizons, weigh competing attack vectors, and adapt to the specifics of each target network. Automating this process via reinforcement learning (RL) has only recently gained traction ( 21 ) , motivated by the success of RL in other complex sequential decision-making domains such as games ( 24 ; 28 ; 44 ; 45 ) robotics ( 27 ) , disease modelling ( 4 ) and industrial control ( 43 ) . The penetration testing problem maps naturally onto a Partially Observable Markov Decision Process (POMDP) ( 34 ; 33 ) : an agent cannot directly observe the full network topology or its vulnerabilities, and must actively acquire information before acting. Beyond this inherent partial observability, training effective RL policies for penetration testing faces challenges common to real-world RL applications ( 8 ) — most notably the sim-to-real gap and generalization to unseen scenarios ( 21 ) . Real-world penetration testing involves long decision horizons, sparse and delayed rewards, combinatorial action spaces, and strong dependence on network-specific structure. Policies that overfit to narrow training scenarios will fail to generalize, and closing this gap requires simulators that expose agents to structural diversity and decision-relevant complexity rather than merely increasing surface-level realism. Existing penetration testing simulators ( 35 ; 38 ; 9 ) fall short on two fronts. First, they are primarily fixed to single or narrowly parameterized scenarios, providing insufficient diversity to train generalizable policies. Second, they rely on CPU-bound Python implementations, which creates a fundamental throughput bottleneck: RL requires millions of environment interactions due to sample inefficiency, yet these simulators cannot generate experience fast enough to fully utilize modern accelerators. This bottleneck is especially costly in penetration testing, where RL algorithms are not yet well understood and thorough hyperparameter searches — crucial for reliable results in any new domain, given the known sensitivity of RL to hyperparameter choices ( 29 ; 1 ) — are simply intractable at the speeds current tools offer. To address these challenges, we present NASimJax, a full JAX-based reimplementation and extension of the Python-native Network Attack Simulator (NASim) ( 35 ) . NASimJax preserves the core task abstraction of NASim — modelling penetration testing as a partially observable sequential decision process — but substantially reworks the environment design to support distributional training, curriculum learning, and context-dependent generalization. By running the entire training pipeline on hardware accelerators, NASimJax enables vectorized training across thousands of parallel environment instances, eliminating the CPU-GPU communication bottleneck and achieving up to 100 \times speed-up over the original (cf. Figure 2 ). This throughput unlocks experimentation on larger and more complex networks under fixed compute budgets, and makes broader hyperparameter search tractable under realistic compute constraints. Beyond computational improvements, NASimJax introduces several conceptual advances. We formulate automated penetration testing as a Contextual POMDP ( 10 ) , where each episode is conditioned on a context describing the underlying network instance. This provides a principled framework for training policies across a distribution of environments and directly facilitates zero-shot policy generalization ( 18 ) to previously unseen networks. Realizing this formulation required a fundamental redesign of the network generation process: NASimJax introduces a new generation function that produces structurally diverse, realistic, and guaranteed-solvable scenarios, enabling fine-grained control over topology and vulnerability density. To address the linearly growing action space of larger networks, we introduce a two-stage action selection (2SAS) mechanism that decomposes each decision into host selection followed by per-host action selection. Building on this setup, we evaluate Domain Randomization (DR) ( 41 ) and PLR ( 16 ) for zero-shot policy transfer, finding that low-density training yields an implicit curriculum that improves out-of-distribution generalization. This even holds on topologies denser than those seen during training. We further identify a failure mode arising from the interaction between PLR’s episode-reset behaviour and 2SAS’s credit assignment structure. 2 Preliminaries 2.1 Contextual Reinforcement Learning Setting We formalize our setting as a contextual reinforcement learning problem. We begin with the standard MDP definition and progressively extend it to partially observable and contextual settings. Markov Decision Process An MDP ( 31 ) is a tuple \langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\rho_{0},\gamma\rangle where \mathcal{S} is the state space, \mathcal{A} is the action space, \mathcal{P}(s^{\prime}|s,a) , is the transition probability distribution over next states, conditioned on the current state and action; \mathcal{R}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R} , a reward function; \rho_{0}\in\Delta(\mathcal{S}) , the initial state distribution, from which an initial state can be sampled: s_{0}\sim\rho_{0} ; and \gamma\in[0,1] a discount factor, that determines the relative weighting of future rewards. The objective is to learn a policy \pi(a|s) , a probability distribution over actions conditioned on the current state, that maximizes the expected cumulative discounted reward \mathbb{E}_{\pi}[G_{0}] where G_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k} is called the return. r_{t} is a random variable that represents the reward obtained at time step t . In finite-horizon tasks the sum is clipped to T , the maximum number of allowed steps. Partially Observable Markov Decision Process A POMDP ( 37 ) extends the MDP framework to a tuple \langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\Omega,\mathcal{O},\rho_{0},\gamma\rangle , where \Omega represents the observation space and \mathcal{O}(o|s^{\prime},a) defines the observation function. The agent does not directly observe the underlying state. Only parts of it can be observed at a given time through \mathcal{O} . The optimal policy now depends on the

Read the complete text