Strategic Cyber Defense via Reinforcement Learning-Guided Combinatorial Auctions

Mai Pham, Vikrant Vaze, Peter Chin · 2025

Read Strategic Cyber Defense via Reinforcement Learning-Guided Combinatorial Auctions by Mai Pham, Vikrant Vaze, Peter Chin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10983.txt.

Introduction

Mai Pham and 2 co-authors plan cyber defense with auctions guided by reinforcement learning in this 2025 paper. Defenders must assign scarce actions across many hosts under uncertainty, thinking far ahead. The authors price bundles of defensive actions with values taken from a learning agent's long-term utility estimates, then let a transformer-based auction mechanism divide the bundles.

The learned auctions stay roughly honest even when participants misreport their values. Against oracle and rule-based allocations, the method earned competitive returns and resisted manipulation. Its choices also lined up with where attacks and defenses were actually active. The authors present the work as a proof that auction-based planning, fed by learned values, can organize strategic defense.

Catalog record

CREATORMai Pham, Vikrant Vaze, Peter Chin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10983

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Strategic Cyber Defense via Reinforcement Learning-Guided Combinatorial Auctions Mai Pham, Vikrant Vaze, Peter Chin Abstract Cyber defense operations increasingly require long-term strategic planning under uncertainty and resource constraints. We propose a new use of combinatorial auctions for allocating defensive action bundles in a realistic cyber environment, using host-specific valuations derived from reinforcement learning (RL) Q-values. These Q-values encode long-term expected utility, allowing upstream planning. We train CAFormer, a differentiable Transformer-based auction mechanism, to produce allocations that are approximately incentive-compatible under misreporting. Rather than benchmarking against existing agents, we explore the qualitative and strategic properties of the learned mechanisms. Compared to oracle and heuristic allocations, our method achieves competitive revenue while offering robustness to misreporting. In addition, we find that allocation patterns correlate with adversarial and defensive activity, suggesting implicit alignment with operational priorities. Our results demonstrate the viability of auction-based planning in cyber defense and highlight the interpretability benefits of RL-derived value structures. I Introduction Modern cybersecurity operations can no longer rely solely on reactive measures, such as intrusion detection systems or manual incident response. As advanced persistent threats (APTs) and rapidly evolving malware campaigns continue to increase in sophistication, defenders must engage in strategic planning, taking the initiative in advance, to allocate limited defensive resources across an enterprise network [ 1 , 2 ] . These decisions must be made under adversarial pressure, where attackers exploit system complexity, resource misallocation, and latency in defense coordination. Enterprise systems encompass a diverse set of assets: user endpoints, data servers, authentication services, IoT/OT devices (e.g., industrial controllers), and mission-critical applications such as SCADA or ERP platforms. These networks are modeled in high-fidelity simulations such as DARPA’s CAGE Challenge 2 (CC2) [ 3 ] or observed in real-world datasets such as LANL’s enterprise event logs [ 4 ] . Protecting these heterogeneous assets requires selecting defensive actions—such as Analyze, Remove, or Restore—for a subset of hosts, constrained by human, operational, or computational limitations. This notion of an “operational budget” is reflected in many abstract defense models. Stackelberg security games, for example, constrain defender strategies via probabilistic action budgets [ 1 ] , while robust Markov Decision Processes (MDPs) limit feasible action subsets to reflect bounded resources [ 2 ] . Classical heuristics often address this by selecting the top k nodes based on static metrics such as risk scores or attack graph centrality [ 5 , 6 ] . However, to our knowledge, no prior work explicitly formulates an upstream resource-constrained planning layer for the DARPA CC2 environment. This paper introduces such a mechanism: a constrained auction-based planner that allocates defensive actions across hosts prior to episode execution. Guided by valuation signals derived from reinforcement learning agents, our approach reframes cyber planning as an economic coordination problem. Auctions are particularly well-suited to this task because they provide a principled, interpretable framework for allocating limited resources among self-interested, heterogeneous agents. Although underexplored in traditional cybersecurity settings, auctions have been applied effectively to related domains such as resource allocation and pricing in cloud environments [ 7 ] and mitigation of denial-of-service (DoS) attacks [ 7 ] , suggesting a broader applicability. Our abstraction enables modular, scalable, and strategic cyber planning, especially important when resource decisions must be made in advance, before threats fully materialize. This upstream perspective differs from most existing work, which emphasizes reactive or tactical decision making. Game-theoretic approaches [ 1 , 8 ] model attacker-defender interactions with strong assumptions about observability and centralized control. Attack graph-based techniques [ 9 , 5 ] use structural analysis to identify vulnerable nodes but struggle to scale or adapt dynamically. MDPs and their extensions, including robust MDPs [ 2 ] , Partially Observable MDPs (POMDPs) [ 10 ] , and reinforcement learning [ 11 , 12 ] , support sequential planning, but generally operate at the execution level and assume centralized planning. We propose a fundamentally different approach: modeling strategic cyber planning as a decentralized combinatorial auction. Each host is treated as a self-interested agent with private valuations over bundles of defensive actions. These valuations are extracted empirically from the Q-values generated by reinforcement learning agents operating in the CAGE Challenge 2 environment [ 3 ] . This allows our planner to incorporate host heterogeneity, action synergies, and limited resource budgets into a scalable, learning-based framework. Figure 1 illustrates the end-to-end pipeline of our strategic cyber planning framework, from simulation-based valuation learning to auction-driven resource allocation. Fig. 1: Overview of our cyber planning framework. Hosts in the DARPA CAGE Challenge 2 environment (e.g., user machines, servers) are modeled individually. Reinforcement learning agents generate Q-values for each possible defensive action—such as Analyze, Remove, or Restore—based on host behavior and environment context. These values are transformed into private valuations over action bundles. A trained auction mechanism then allocates limited defensive resources by computing probabilistic assignments and payments. The resulting strategic plan informs upstream resource allocation before any episode execution. Decentralization reflects the practical reality that hosts differ in roles, exposure, and impact and often operate under limited or local information. Combinatorial allocation is used because the utility of defensive actions is often non-additive: for instance, a host may benefit from Analyze only if followed by Restore , and some combinations may be redundant or conflicting. Allowing agents to bid on bundles captures these interdependencies more accurately than single-action models. We also aim for incentive compatibility, ensuring that agents (or their learned proxies) have no benefit from misreporting valuations. This improves the robustness of the allocation in learned systems, where preferences may be noisy or adaptive. Rather than maximizing total utility (as in VCG), we optimize for revenue, defined as the sum of reported valuations for assigned bundles, which aligns with truthful mechanism design under budget constraints and is tractable to learn with modern neural networks. To solve the auction, we build on recent advances in differentiable mechanism design [ 13 ] . Specifically, we adopt CAFormer, a permutation-equivariant neural auction model that learns incentive-aligned resource allocations under feasibility constraints using adversarial regret minimization. In this work, we apply CAFormer to a new domain: upstream cyber defense planning under operational budgets. We instantiate this approach in the DARPA CAGE Challenge 2 simulation environment, where private valuations for defensive actions are derived from reinforcement learning Q-values across heterogeneous hosts. We use CAFormer to allocate limited resources, such as Analyze or Restore actions, before execution begins, with the goal of improving strategic preparedness under threat. Our experimental results evaluate both the auction performance in terms of revenue and regret, and its downstream impact on cyber defense effectiveness. The remainder of the paper is organized as follows. Section II reviews related work on cyber planning and mechanism design. Section III formulates the auction-based planning problem. Section IV describes our valuation modeling and environment setup. Section V details the CAFormer architecture and the training procedure. Section VI presents the experimental results. In Section VII, we conclude with a discussion of limitations and future directions. II Related Work II-A Cybersecurity Planning: Centralized and Sequential Approaches Strategic planning in cyber defense has traditionally been approached through game-theoretic, graph-based, and decision-theoretic models. Stackelberg security games [ 1 , 8 ] model defender-attacker interactions in a leader-follower framework. These methods capture adversarial dynamics and resource constraints through

Read the complete text