ANL-26/18 Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation Strategic Security Sciences Division About Argonne National Laboratory Argonne is a U.S. Department of Energy laboratory managed by UChicago Argonne, LLC under contract DE-AC02-06CH11357. The Laboratory’s main facility is outside Chicago, at 9700 South Cass Avenue, Argonne, Illinois 60439. For information about Argonne and its pioneering science and technology programs, see www.anl.gov. DOCUMENT AVAILABILITY Online Access: U.S. Department of Energy (DOE) reports produced after 1991 and a growing number of pre-1991 documents are available free at OSTI.GOV (www.osti.gov), a service of the US Dept. of Energy’s Office of Scientific and Technical Information. Reports not in digital format may be purchased by the public from the National Technical Information Service (NTIS): U.S. Department of Commerce National Technical Information Service 5301 Shawnee Rd Alexandria, VA 22312 www.ntis.gov Phone: (800) 553-NTIS (6847) or (703) 605-6000 Fax: (703) 605-6900 Email: orders@ntis.gov Reports not in digital format are available to DOE and DOE contractors from the Office of Scientific and Technical Information (OSTI): U.S. Department of Energy Office of Scientific and Technical Information P.O. Box 62 Oak Ridge, TN 37831-0062 www.osti.gov Phone: (865) 576-8401 Fax: (865) 576-5728 Email: reports@osti.gov Disclaimer This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor UChicago Argonne, LLC, nor any of their employees or officers, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of document authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, Argonne National Laboratory, or UChicago Argonne, LLC. ANL-26/18 Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation prepared by Benjamin Blakely (lead), Akshay Ram Ramchandra, Matthew Swindall, Stefano Fenu, Shree Ram Abayankar Balaji Strategic Security Sciences Division, Argonne National Laboratory Project Leadership: Kumarsinh Jhala (PI), Siby Plathottam (PI), Prakash Ranganathan (Co-PI, UND) School of Electrical Engineering and Computer Science, University of North Dakota For Cybersecurity, Energy Security, and Emergency Response (CESER) U.S. Department of Energy May 2026 Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation BENJAMIN BLAKELY, Argonne National Laboratory, USA AKSHAY RAM RAMCHANDRA, University of North Dakota, USA MATTHEW SWINDALL, Argonne National Laboratory, USA STEFANO FENU, Argonne National Laboratory, USA SHREE RAM ABAYANKAR BALAJI, University of North Dakota, USA This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and per- defense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward This work was sponsored by the U.S. Department of Energy Office of Cybersecurity, Energy Security, and Emergency Response (CESER). Argonne National Laboratory’s work was supported under DOE contract number DE-AC02-06CH11357. The submitted manuscript has been created by UChicago Argonne, LLC, operator of Argonne National Laboratory. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. 3 4 Blakely et al. reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option. AdCyDER: Online RL with Stackelberg-Optimized Defenses 5 Contents Abstract 3 Contents 5 List of Tables 7 List of Figures 7 Executive Summary 8 1 Introduction 11 1.1 Research Context 11 1.2 Hypothesis Tested 11 1.3 Scope of This Report 12 2 Background and Related Work 12 2.1 DER Attack Surface and Industrial Control Protocol Exposure 12 2.2 Machine-Learning Detection in OT Environments 13 2.3 Cyber-Physical Testbeds for DER Security Research 14 2.4 Stackelberg Security Games for Defender Decision-Making 14 2.5 Reinforcement Learning with Delayed Reward 16 2.6 Reward Misspecification and Proxy-Objective Divergence 16 2.7 Neural-Network Calibration 16 2.8 Production ML Systems: Technical Debt and Continual Learning 17 3 System Architecture 17 3.1 Pipeline Overview 18 3.2 Synthetic SCADA Generator 18 3.3 Ingestion and Feature Aggregation 22 3.4 Reinforcement Learning Attack Classifier 24 3.5 Stackelberg Game-Theoretic Defense Selector 27 3.6 Actuator Framework and Closed-Loop Coupling 30 3.7 Cross-Cutting Design Choices 31 3.8 Deployment and Infrastructure 32 4 Experimental Methodology 33 4.1 Evaluation Configurations 34 4.2 Experimental Campaign 34 4.3 Data Extraction and Analysis Pipeline 35 4.4 Same-Architecture Supervised LSTM Ceiling Test 37 4.5 Statistical Methods 38 4.6 Reproducibility 39 5 Findings 39 5.1 Q1 — Upstream Physical-Layer Signal (ADS) 39 5.2 Q2 — Upstream Network-Layer Signal (IDS) 41 5.3 Q3 — Per-Class P(True) Elevation 41 5.4 Q4 — Output Distribution Per True Class 42 5.5 Q5 — Classifier Learning Trajectory 44 5.6 Q6 — Reward–Correctness Causal Link 44 6 Blakely et al. 5.7 Q7 — Grid Outcome Trend 46 5.8 Distributional Accuracy Summary 47 5.9 Supervised Ceilings 48 5.10 Summary of the Findings 49 6 Discussion 51 6.1 What the Evidence Supports 51 6.2 What the Evidence Does Not Support 52 6.3 The Reward Signal: Weakly Aligned, Underpowered 53 6.4 Methodology Generalization: Confidence Convergence Is Not Learning 54 6.5 Limitations 55 6.6 Implications for the Original Research Hypothesis 56 7 Conclusions and Future Work 56 7.1 Conclusions 56 7.2 Aspirational Future Work 57 7.3 Closing 59 Acknowledgements 59 References 60 Appendices 63 A Adjacent Project Contributions 64 A.1 adcyder-attack-simulator (Public Release) 64 A.2 test-outstation (Internal) 65 A.3 Workflow-Orchestrated Simulator Rewrite (Prototype) 65 AdCyDER: Online RL with Stackelberg-Optimized Defenses 7 List of Tables 1 Attack taxonomy used by the SCADA generator. 21 2 Principal MongoDB collections written by the pipeline. 24 3 Defense action set available to the GT engine. 27 4 SME-encoded effectiveness matrix Eff(𝑑, 𝑡). 29 5 The three baseline modes and the sub-question each isolates. 34 6 Distributional accuracy by mode. Mann-Whitney U tests P(correct) per record against the random baseline 1/12. 47 7 Supervised ceilings on online-, frozen-, and gt only-mode per-window features. All rows: chronological 70/30 split, predictions pooled across facilities before computing macro F1. The “× deployed” column expresses the ratio against the deployed RL classifier in the same mode. 48 List of Figures 1 End-to-end pipeline data flow. 18 2 Q1 — ADS attack-vs-normal distributions per attack class, by mode. 40 3 Q2 — IDS alert count attack-vs-normal distributions per attack class. 41 4 Q3 — mean 𝑃 (true class) per (attack class, mode). 42 5 Q4 — mean output probability per (true → predicted) class pair, online mode. 43 6 Q5 — P(correct) over training-elapsed time within an iteration, by mode. Horizontal reference: 1/12 ≈ 0.083. 44 7 Q6 — multi-lens reward-correctness coupling, online mode. 45 8 Q7 — grid-health-delta across training iterations, by mode. 47 9 Same-architecture supervised LSTM ceiling — per-class F1 by mode (predictions pooled across EIDs within each mode). 49 10 Same-architecture supervised LSTM ceiling — macro F1 per mode. 50 8 Blakely et al. EXECUTIVE SUMMARY Scope of this report. The AdCyDER project encompasses a broad research effort in DER cybersecurity. This report covers one strand of that effort: the integrated online- learning pipeline that couples a Reinforcement Learning (RL) attack classifier with a Stackelberg game-theoretic (GT) defense selector in a closed feedback loop, the end-to- end pipeline assembled around it from components developed across the project, and the experimental evaluation conducted against that integrated system. It does not aim to summarize AdCyDER as a whole. Project objective tested here. The RL/GT loop is not an attack-classification system in the conventional sense. The RL classifier produces a per-window probability distribution over attack types rather than a hard label, reflecting the operational reality that attacks are diverse and may not cleanly fit into a single predefined category; a defender selecting a mitigation action gains more from a distribution that hedges across plausible attack types than from an overconfident single classification. The Stackelberg layer consumes that distribution alongside subject-matter-expert (SME) priors over (defense, attack) effectiveness pairings and per-defense cost, and selects the defense action that minimizes expected grid impact. The loop’s objective is therefore to learn to produce probability distributions that, fed through the SME-priored Stackelberg layer, drive selection of grid-health-preserving defenses — not to maximize per-window classification accuracy in isolation. The hypothesis tested was that an online RL classifier trained from delayed grid-health-delta feedback derived from physically-grounded inverter telemetry could learn to produce such defense- driving distributions per facility without requiring labeled attack data in production. The scientific bet was that SME-encoded defense structure plus operationally-grounded reward could substitute for the labeled training data that is infeasible to collect at scale in operational DER environments. What was built. A complete containerized end-to-end pipeline. The system was originally architected for AWS deployment with LocalStack-based local replication, but was pivoted early in the project to run entirely on LocalStack when AWS access proved infeasible due to organizational approval and cost constraints; the AWS-style service architecture was preserved. Components: synthetic SCADA telemetry with embedded Fronius inverter physics and trained anomaly-detection autoencoder; coordinated network attack scenarios with Suricata-format alerting; Kafka and OpenSearch ingest; MongoDB time-windowed aggregation; a per-facility LSTM classifier trained online from grid-health-delta reward; a Stackelberg defense optimizer issuing OpenC2 commands over MQTT; and the actuator framework that closes the loop back to the simulated environment. The integrated system passes 335+ automated tests across the main repository and two service submodules. What was tested. A three-baseline-mode experimental campaign was conducted in May 2026, totaling approximately 224 hours of evaluation runtime across 28 eight-hour iterations: 16 iterations of the online RL/GT loop (full configuration), 8 iterations of a frozen-classifier baseline (RL classifier trained for three hours, then weights frozen while the GT defense selector continued operating), and 4 iterations of a Stackelberg-only baseline (uniform attack-probability prior, no RL classifier). All twelve attack classes were exercised AdCyDER: Online RL with Stackelberg-Optimized Defenses 9 across five simulated facilities in every iteration. Aggregated, the campaign produced 6,031 classification records suitable for analysis (3,429 online, 1,700 frozen, 902 gt only). Diagnostic figures were ordered along the data pipeline (physical-layer signal → network- layer signal → classifier output structure → classifier learning trajectory → reward linkage → grid outcome) so that each figure tested a distinct stage of the RL/GT loop. What was found. Both upstream signal sources — the autoencoder-based anomaly detector and the Suricata IDS — were verified functional; the autoencoder produces partial attack-vs-normal separation at the per-classification-window granularity (Cohen’s 𝑑 ≈ +0.46), and the IDS produces strong separation (𝑑 ≈ +1.38). The online RL/GT loop did not produce defense-driving distributions distinguishable from chance. The classifier’s mean P(true class) in the online mode was 0.087 vs the random baseline 0.083 (𝑝 = 0.30, not significant); argmax accuracy 0.089 (1.06× random); and grid-health over training iterations was flat across all modes. The frozen baseline performed indistinguishably from random and slightly below the continuously-trained online mode (argmax 0.077, 0.92× random, 𝑝 = 0.40), so continued online training neither rescues nor materially degrades the early- bootstrap classifier — both regimes sit at chance. The reward signal driving online updates was evaluated with a multi-lens robustness test (threshold split, hurdle decomposition, trimmed-Pearson sensitivity) chosen to avoid the over-interpretation hazard that single- statistic correlation summaries pose on heavy-tailed zero-inflated reward distributions of this kind. The convergent reading across all three lenses is that the reward signal carries weak directional coupling with classifier correctness in the methodology-expected direction: top-decile P(true) records produce observable grid-state changes more often than bottom- decile records (composite 69% → 89%, Fisher exact 𝑝 < 10−10 ) and carry slightly larger negative deltas (composite mean diff −0.046, Mann-Whitney 𝑝 = 0.029, top-vs-bot Cohen’s 𝑑 ≈ −0.19). Effect magnitudes are small (Pearson 𝑟2 < 0.5% on all three reward channels) and per-class breakdowns do not survive multi-lens convergence at the per-class sample sizes available; the coupling reading is a pooled-record property, not a per-class one. The reward channel is informative but underpowered: the gradient information available to the classifier is in the right direction but at magnitudes that the optimization procedure cannot extract over 28 iterations. The decisive diagnostic was a pair of supervised ceilings on the same per-window feature vectors. A same-architecture supervised LSTM trained offline with cross-entropy on the same per-facility structure reached macro F1 ≈ 0.34 versus 0.07 for the actual online RL classifier (a 4.7× gap with the only difference being the training procedure). A multinomial logistic-regression classifier under matched per-EID + chronological handicaps reached macro F1 ≈ 0.70 (10× deployed) and ≈ 0.79 pooled across facilities (11× deployed), showing the features themselves support substantially more signal than even the same-architecture offline-trained variant extracts at this data scale. The training-procedure deficit is bracketed between 4.7× (lower bound, architecture held constant) and 10–11× (upper bound, linear classifier on the same features). The features carry signal that could yield more informative distributions for the GT engine; the online training procedure does not extract it, and the reward signal that drives that training is misaligned with the metric the loop is meant to improve. 10 Blakely et al. What this means. The original research hypothesis is not supported by the May 2026 data on this synthetic pipeline. The primary contributions of this work are: (1) A diagnostic methodology for online RL/GT defense pipelines that learn from delayed environmental reward, comprising the pipeline-ordered Q1–Q7 figure suite and paired supervised ceilings (a same-architecture lower bound and a linear feature- signal upper bound) that bracket where a downstream failure sits between training procedure and architecture choice. The protocol distinguishes data-quality failures from training-procedure failures and catches what would otherwise be a confidently- reported false positive based on confidence trajectory alone. (2) Open, reproducible experimental infrastructure for DER cybersecurity re- search: a baseline-mode switchable RL/GT loop, a multi-iteration evaluation harness, a time-aligned synthetic generator with embedded inverter physics and IDS alerting, and a complete diagnostic analysis pipeline. (3) A characterization of the synthetic-data ceiling on this configuration: the same- architecture supervised LSTM ceiling at macro F1 ≈ 0.34 (pooled ≈ 0.41) bounds the training-procedure deficit from below at 4.7×, and a linear feature-signal ceiling at macro F1 ≈ 0.70–0.79 bounds it from above at 10–11×. The synthetic features carry attack-discriminative signal at a level adequate for methodology development; a hardware-in-the-loop testbed with higher-fidelity inverter telemetry would be the appropriate next-step environment for re-testing the underlying learning hypothesis. Aspirational next steps. The project does not currently have funding for follow-on work; the directions below are recorded for the record so that future investigators can pick up the thread. In priority order based on the bottleneck localization: (a) reward reformulation to raise the magnitude of the directionally-aligned but underpowered reward signal identified in Q6 (highest priority — the most actionable single mechanism); (b) HIL testbed evaluation with real inverter telemetry to test whether richer feature signal raises the supervised ceiling; (c) external validation of the diagnostic methodology by applying it to other online RL/GT defense pipelines. Lower-priority directions and supporting follow-on work are itemized in Section 7.2. The remainder of this report develops each of these points in detail. Section 1 introduces the research context, scope, and hypothesis tested by the RL/GT loop. Section 2 reviews relevant prior work in DER cybersecurity, online RL with delayed environmental reward, and Stackelberg security games. Section 3 documents the integrated pipeline architecture in full. Section 4 describes the experimental methodology including the three-baseline- mode evaluation campaign, the Q1–Q7 diagnostic protocol, and the same-architecture supervised LSTM ceiling test. Section 5 presents the per-stage findings with figures. Section 6 interprets the results and discusses limitations. Section 7 consolidates the contributions and recommended next steps. AdCyDER: Online RL with Stackelberg-Optimized Defenses 11 1 INTRODUCTION 1.1 Research Context Distributed energy resources — grid-connected inverters serving rooftop solar, behind- the-meter storage, electric-vehicle chargers, and small-scale wind — are now a material fraction of bulk-power generation in many regions, and their cybersecurity properties are correspondingly material to grid reliability. Unlike traditional bulk-generation assets, DER inverters are deployed at scale across customer premises, are reachable through utility-, aggregator-, and vendor-managed communication paths, and expose telemetry and control interfaces over industrial protocols (Modbus, DNP3, IEEE 2030.5) that were not designed with adversarial environments in mind. The resulting attack surface is well-documented in the DER cybersecurity literature (Section 2.1), with FDI, MITM, DoS, replay, and firmware exploitation all reported as feasible against grid-connected inverters and their control channels. The defensive picture is shaped by an operational constraint that does not affect conven- tional enterprise security ML: it is infeasible to collect labeled attack data at the scale and on the diversity that supervised classifiers typically need, in production DER environments. Real attacks are rare, ground-truth labels are expensive to obtain, and attack types observed in one facility may never recur elsewhere. The training-data scarcity problem motivates approaches that learn from operationally available signals — post-decision grid-health ob- servations, anomaly scores, network-side alert telemetry — rather than from human-labeled attack examples. 1.2 Hypothesis Tested This report presents the design and evaluation of one such approach: an integrated online RL / Stackelberg game-theoretic defense pipeline for DER cybersecurity, developed within the AdCyDER project (DOE CESER funded). The pipeline couples a per-facility online RL classifier — which produces an attack-type probability distribution from per-window features — with a Stackelberg defense selector that consumes those distributions alongside subject-matter-expert priors over (defense, attack) effectiveness pairings and per-defense costs, and selects the OpenC2 defense action that minimizes expected grid impact. The classifier is trained online from delayed grid-health-delta reward derived from the post- defense composite of an autoencoder ADS and a rolling-baseline IDS density, rather than from labeled attack data. The hypothesis tested is that, in this configuration, the loop will learn to produce per- facility attack-type distributions that drive effective Stackelberg defense selection from operational outcomes alone — without requiring the labeled training data the operational environment cannot supply. The scientific bet is that SME-encoded defense structure plus operationally-grounded reward can substitute for the labels that conventional supervised approaches require. 12 Blakely et al. 1.3 Scope of This Report AdCyDER as a project is broader than the work reported here. This report is scoped specifically to the integrated RL/GT loop, its evaluation campaign, and the diagnostic protocol applied to that evaluation. We document the integrated pipeline architecture in full (Section 3), describe the experimental methodology including a three-baseline-mode campaign and a same-architecture supervised LSTM ceiling test (Section 4), present the per- stage diagnostic findings (Section 5), interpret the results (Section 6), and consolidate the contributions and aspirational next steps (Section 7). Adjacent project software contributions that do not sit on the data path of the evaluated loop — the publicly-released attack simulator, an internal DNP3 outstation test environment, and a workflow-orchestrated simulator rewrite prototype — are documented in Appendix A. The headline result is a falsified configuration of the hypothesis above, with the failure cleanly localized by the diagnostic protocol to two specific mechanisms; the protocol itself, the open infrastructure that supports it, and the bounded characterization of the loop’s signal ceiling are the contributions we expect to outlast the negative result on the specific configuration tested. 2 BACKGROUND AND RELATED WORK This section reviews the prior work that the AdCyDER design draws on. The discussion is organized along the data path of the integrated loop — the DER attack surface and the industrial protocols that expose it; the machine-learning detection techniques used on operational-technology (OT) traffic; the testbed methodology used to evaluate cyber- physical defenses under realistic conditions; the Stackelberg game-theoretic framework for defender decision-making; the reinforcement-learning challenges introduced by sparse and delayed reward signals; the calibration properties of modern neural-network classifiers; and the production-ML engineering literature that motivates the diagnostic protocol used in Section 4. 2.1 DER Attack Surface and Industrial Control Protocol Exposure Cybersecurity research on distributed energy resources consistently identifies the same set of attack surfaces: inverter telemetry interfaces, control channels (Modbus, DNP3, IEEE 2030.5), and remote operational endpoints used by aggregators, vendors, and utilities to monitor and command grid-connected inverters [1–4]. The threat catalogue documented across these surveys includes false-data injection (FDI), man-in-the-middle (MITM), denial- of-service (DoS), replay, and firmware exploitation, with smart-inverter communication and remote-access interfaces highlighted as a primary entry point [1, 5]. Work focused on grid-connected PV inverters specifically demonstrates FDI, DoS, and replay attacks against sampled signals and issued commands at the converter interface [6], and Chen et al. study cyber-attack detection and countermeasures for distributed power-electronic converters in microgrid contexts [7]. The protocols that carry these control channels were not designed for adversarial en- vironments. Modbus was specified without security measures and uses register addresses AdCyDER: Online RL with Stackelberg-Optimized Defenses 13 for control; DNP3 was not originally designed with built-in authentication, integrity, or encryption [3, 8]. The IEEE 1815 (DNP3) standard subsequently added a Secure Authenti- cation mechanism as an extension and explicitly leaves transport-layer security (IPSec/TLS) out of scope, implicitly confirming that the base specification lacked these protections [9]. Zografopoulos et al. note the same property and document specific message interception, modification, and fabrication attacks that the lack of base-protocol security enables [2]. AdCyDER’s choice to instrument Modbus and DNP3 traffic at facilities (Section 3.2) is motivated by this exposure pattern. Standardized cyber-defense action and transport. Translating a defender’s chosen response into a machine-actionable command on a distributed fleet of DER endpoints requires both a standardized action vocabulary (so that the same instruction can be issued by any analytic engine and consumed by any responder agent without bespoke integration per pairing) and a standardized message-transport substrate (so that command delivery scales across many facilities without per-facility connection management). For the action vocabulary, AdCyDER adopts Open Command and Control (OpenC2), the OASIS standard for machine- readable cyber-defense action commands. OpenC2 specifies an action-target-args grammar (e.g., {action: deny, target: {ipv4 net: 10.0.0.0/24}}) intended to be producer- and consumer-agnostic; defense logic that emits OpenC2 can be paired with any actuator framework that consumes OpenC2, decoupling decision-making from enforcement. For the transport substrate, AdCyDER uses MQTT (Message Queuing Telemetry Transport), an OASIS-standard publish-subscribe protocol designed for many-to-many messaging over high- latency, intermittently-connected networks — characteristics that match the operational reality of geographically distributed DER endpoints behind aggregator gateways. MQTT’s broker-mediated topic structure also lets each facility subscribe only to commands targeted at it, providing a natural per-facility addressing scheme without a custom routing layer. AdCyDER’s specific use of OpenC2-over-MQTT is described in Section 3.6. 2.2 Machine-Learning Detection in OT Environments Two ML detection paradigms are relevant to AdCyDER’s anomaly-detection subsystem (ADS) and supplementary IDS layer: reconstruction-based unsupervised anomaly detection and signature- or rule-based intrusion detection. Autoencoder-based anomaly detection trains a model on normal-operation data and flags test inputs whose reconstruction error departs from the learned distribution. Oyedotun et al. apply this directly in SCADA and DCS environments, training autoencoders on normal multimodal telemetry and flagging anomalous inputs by reconstruction error [10]. Alharbi reports an analogous setup using a variational autoencoder trained on normal renewable-energy-grid behavior, with significant reconstruction error used as the anomaly signal [11]; broader smart-grid surveys document the same pattern of reconstruction- error thresholding for abnormal behavior [12]. AdCyDER’s embedded autoencoder ADS (Section 3.2), developed by Plathottam et al. [13], follows this established design. Signature-based network intrusion detection uses predefined rule sets and pattern matches to emit categorized alerts. Mogilicharla et al. describe such a rule-based NIDS in a DER 14 Blakely et al. microgrid intrusion-detection deployment [5]; this is the most direct prior reference in our corpus to the design pattern that Suricata occupies in our pipeline. We are not aware of a published evaluation of Suricata specifically on DER traffic, and we treat the rule-based NIDS layer as an established design pattern rather than a contribution. The choice of LSTM lookback window for sequential security-event classification involves a tradeoff between temporal context and per-entity data volume. Luay et al. report empirical evaluation across LSTM sequence lengths from 10 to 100 in network intrusion detection, finding that shorter windows can outperform longer ones because longer sequences dilute localized temporal patterns [14]. We do not treat this as decisive guidance for our 3-window (15-minute) lookback, but it indicates that the choice is a real design parameter rather than a free one. 2.3 Cyber-Physical Testbeds for DER Security Research Hardware-in-the-loop (HIL) testbeds combining real protective relays, inverters, smart meters, RTUs, or PMUs with simulated power-system dynamics are used to evaluate cyber- physical defense mechanisms under operationally realistic conditions. Nguyen et al. describe a real-time cyber-physical testbed integrating protective relays, PMUs, RTUs, and an RTAC with simulated NYS power-grid dynamics for cybersecurity assessment of large-scale power systems [15]; Mogilicharla et al. report a laboratory-scale testbed integrating a real plant-level inverter, PV emulator, battery, smart meter, and Modbus communications to validate a hybrid intrusion-detection method [5]; and Liu et al. note the use of HIL testbeds in DER cybersecurity research more generally [4]. OPAL-RT real-time simulators are a recurring component of these testbeds. Sangoleye et al. describe an SNL microgrid cyber-physical testbed using an OPAL-RT OP5600 simulator to run real-time simulated generators and networking devices alongside live cyber-attacks (brute-force, DoS, port-scan) [16], and Nguyen et al. similarly use OPAL-RT (alongside RTDS and EXata) as a real-time simulator hosting the control center, PDC, and detailed EMT models including offshore wind farms within a cyber-physical testbed [15]. AdCyDER’s HIL trajectory described in Section 3 fits within this established methodology. A complementary stream of work uses purely synthetic data generators in place of HIL hardware. Farsi et al. report a simulation-based smart-grid dataset providing reproducibility and labeled ground truth for cyber-attack scenarios [17]. The general tradeoff — repro- ducibility and ground truth at the cost of fidelity to operational telemetry — is accepted in this literature, and AdCyDER’s containerized synthetic generator (Section 3.2) sits at this end of the spectrum during the campaign reported here. 2.4 Stackelberg Security Games for Defender Decision-Making The Stackelberg framework for security games models defender–attacker interactions as sequential leader–follower games in which the defender commits first to a strategy and the attacker best-responds, with Bayesian extensions accommodating uncertainty over attacker types. The framework is broadly used across the cyber-physical security and IIoT literatures: Zhang et al. review attack–defense game models for IIoT and explicitly charac- terize Stackelberg games as sequential leader-commits/followers-respond interactions [18]; AdCyDER: Online RL with Stackelberg-Optimized Defenses 15 Clempner formulates Bayesian–Markov Stackelberg security games for deceptive defense and attack tactics, with defender-commits-first / attacker-best-responds structure and Bayesian handling of attacker-type uncertainty [19]; Ait Temghart et al. apply Stackelberg security games to allocate limited defensive resources in cloud-computing security [20]; Sengupta and Kambhampati formulate Bayesian Stackelberg Markov games for adaptive moving-target defense, with explicit defender-leader / attacker-follower commitment under attacker-type uncertainty [21]; Zhang and Malacaria formulate cyber-security defense as a Stackelberg game with a defender committing to a security portfolio and use Bayesian Stackelberg games for uncertainty over attack types [22]; Li and Zheng describe Stack- elberg security games as defender-leader / attacker-follower commitments and reference Stackelberg/Bayesian-Stackelberg models addressing attack-type uncertainty [23]; and Deng et al. apply Stackelberg Markov games to sequential deception defense against advanced persistent threats [24]. Cox provides early framing of attacker–defender interactions as leader–follower (Stackelberg) games with the defender committing to a defensive resource allocation observed by the attacker [25], and Lian et al. note in passing that Stackelberg games leverage the defender’s informational advantage [26]. The defender’s commitment is to a strategy, and AdCyDER’s GT defense selector (Section 3.5) instantiates exactly this structure: the defender minimizes an expected-loss functional over a discrete action set, with the loss parameterized by per-action effectiveness against each attack type, per-attack baseline impact, and per-action cost. The classifier’s posterior over attack types plays the role of the defender’s belief over attacker types in the Bayesian extension. On the source of game parameters: SME priors versus learned parameters. A live debate in the Stackelberg-security-games literature concerns where the game’s payoff parame- ters — per-action effectiveness, per-attack impact, per-action cost — should come from. Sengupta and Kambhampati argue against relying on security-expert-supplied priors for these parameters, on the grounds that experts may not provide accurate values for impact, defense performance, or switching cost, and propose learning them from interaction in- stead [21]. AdCyDER takes the opposite design choice: the GT layer’s effectiveness matrix, baseline-impact vector, and cost vector are encoded by SME judgement and held fixed during operation, while the RL classifier learns only the attack-type posterior that is fed into the otherwise-static Stackelberg loss. This is a deliberate trade-off given the project’s operating constraints — ground-truth attack labels are infeasible to collect at the scale that learning the full game parameter set would require, and SME calibration of the ef- fectiveness/impact/cost vectors is the available alternative within scope. The trade-off is itself a candidate failure mode for the loop, and we return to it in the discussion of next steps (Section 7.2): if the hypothesis tested in this report is to be re-tested under different parameterizations, jointly learning the GT parameters alongside the classifier posterior — in the spirit of Sengupta and Kambhampati — is a natural alternative to the SME-prior-only configuration evaluated here. 16 Blakely et al. 2.5 Reinforcement Learning with Delayed Reward Reinforcement learning faces a credit-assignment problem because the consequences of an action can manifest many time steps after it is taken, leaving the agent to determine retrospectively which of its earlier decisions a given reward should be attributed to [27]. AdCyDER’s online RL classifier inherits this difficulty in a particularly demanding form: the reward signal is the change in grid health following a defense action, which is itself the consequence of (i) the classifier’s posterior over attack types, (ii) the GT selector’s chosen defense, and (iii) the actuator’s physical effect on the simulated power system. The chain from classification decision to grid-health observation is several pipeline stages long, and the agent receives no per-stage feedback — only the final, lagged grid-health delta. Reinforcement-learning approaches to cybersecurity decision-making have been proposed where labeled attack data is infeasible to collect at scale: Babar et al., for example, motivate hierarchical RL for autonomous cyber-incident detection in industrial cyber-physical systems by the impracticality of manual or rule-based maintenance of detection systems [28]. The broader RL-for-IDS literature is large but, in our reading, focuses primarily on stationary supervised or batch settings rather than the closed-loop online setting AdCyDER targets. 2.6 Reward Misspecification and Proxy-Objective Divergence A failure mode well-documented in the AI-safety and RL literatures is reward hacking or specification gaming: an RL agent achieves high proxy reward under a misspecified reward function while degrading the true objective [29, 30]. Pan et al. demonstrate empirically that more capable agents exploit misspecified proxy rewards more aggressively, exhibiting phase transitions in which the true reward sharply drops as proxy reward rises [31]. Skalse et al. provide a formal definition of reward hacking as optimization of a proxy reward leading to decreased true reward, and explicitly reference the Pan et al. phase-transition observations [32]. Amodei et al. position reward hacking among the foundational accident- risk problems for RL deployed in real-world systems [30]. This literature is directly relevant to the findings reported in Section 5: the absent- or-anti-correlated relationship we observe between AdCyDER’s classifier correctness and the grid-health-delta reward (Q6) is the diagnostic signature these works describe of a misspecified or hacked proxy reward. In the framework of Amodei et al., a related methodological recommendation we adopt is the use of a baseline trained only on labeled episodes to isolate the effect of label scarcity in semi-supervised RL — the same role our supervised-LSTM ceiling test plays for online versus offline training [30]. 2.7 Neural-Network Calibration Modern deep neural-network classifiers are commonly miscalibrated: they produce over- confident softmax probabilities, and calibration tends to deteriorate during training even as classification accuracy improves [33]. Guo et al. quantify this with expected calibration error (ECE) and reliability diagrams, showing that confidence and accuracy are conceptu- ally distinct quantities — a model can be accurate while remaining poorly calibrated, or AdCyDER: Online RL with Stackelberg-Optimized Defenses 17 confident while remaining inaccurate [33]. Their proposed remedy — temperature scaling, a single-parameter variant of Platt scaling — is shown to be effective on most datasets. This distinction is central to the diagnostic methodology used in Section 4. Earlier project status reports for AdCyDER tracked classifier confidence trajectory as the headline learning-progress metric; the distributional accuracy and per-class P(correct) diagnostics that supersede that framing in this report are motivated directly by the Guo et al. finding that confidence and correctness are independent quantities. 2.8 Production ML Systems: Technical Debt and Continual Learning Two strands of production-ML literature inform the diagnostic protocol applied in Section 4. The first is the body of work on hidden technical debt and engineering practice for ML systems. Sculley et al. introduce the concept of hidden technical debt in ML systems, documenting that data dependencies, glue code, and configuration debt frequently dominate the model code itself, and arguing for treating these as first-class engineering concerns [34]. Breck et al. formalize this into the “ML test score,” a rubric for production ML readiness that requires dedicated tests and monitoring of data, features, infrastructure, and model behavior beyond conventional unit tests of the model code, and explicitly notes that many production failures originate in upstream data rather than the model itself [35]. Amershi et al. document analogous engineering challenges in software engineering for ML, including data pipelines, feedback loops, and component entanglement [36]. The pipeline-stage diagnostic protocol used in Section 4 (Q1–Q7 along the data path) is a direct application of this principle: the protocol forces each stage’s input data, intermediate output, and downstream behavior to be inspected independently before model-level conclusions are drawn. The second is the continual-learning literature on training neural networks on non- stationary data streams. Parisi et al. and Delange et al. both establish that neural networks trained continually on non-stationary streams suffer from catastrophic forgetting, and that standard evaluation contrasts continual-learning methods against joint (offline) training as an upper reference and naive fine-tuning as a lower-bound baseline [37, 38]. Kirkpatrick et al. analyze catastrophic forgetting in the sequential-learning setting and contrast against joint multitask training [39]. AdCyDER’s three baseline modes (online, frozen, gt only) and the supervised-LSTM ceiling test (Section 4) instantiate the same family of comparisons — online versus frozen approximates the continually-trained-versus-static-snapshot contrast, and the supervised offline LSTM provides the joint-training upper-reference that the continual-learning literature uses to bound achievable performance. 3 SYSTEM ARCHITECTURE This section documents the integrated pipeline that the AdCyDER online learning loop runs on. Each subsection covers one functional stage of the end-to-end data path. Source code is organized as the primary repository adcyder-infrastructure plus two service submodules; paths of the form src/stack/dockerfiles// refer to the corresponding service implementations. 18 Blakely et al. 3.1 Pipeline Overview Figure 1 shows the closed-loop data flow. The synthetic SCADA generator emits four parallel data streams (inverter telemetry, network flow records, IDS alerts, and attack ground truth) into a Kafka bus. A Fluent Bit consumer indexes telemetry into OpenSearch; an aggregator service queries OpenSearch on a fixed cadence to produce per-window feature vectors stored in MongoDB. The Reinforcement Learning (RL) attack classifier reads features from MongoDB, produces per-window attack-probability distributions, and forwards them to the Stackelberg game-theoretic (GT) defense selector. The selector chooses a defense action and emits an Open Command and Control (OpenC2) command — the OASIS standard for machine-readable cyber-defense action commands — over MQTT; an actuator framework receives the command and reflects the resulting mitigation state back to the SCADA generator, which thereafter modulates traffic and physics consistent with the deployed defense. Grid health under the new defense — a per-facility composite of the autoencoder ADS score and a rolling-baseline IDS alert density (Section 3.4.2) — is then observable from the next post-defense feature window, closing the loop. mitigation state SCADA Generator Fronius physics, ADS, Fluent Bit / OpenSearch Kafka IDS, netflow, GT Kafka Consumer MongoDB OpenSearch features, Aggregator rl/gt logs, 5-min windows rewards GT Mitigator RL Classifier Actuator Stackelberg MQTT per-facility LSTM OpenC2 𝑑* = arg min 𝐿(𝑑) Fig. 1. End-to-end pipeline. Solid black arrows: forward data flow. Dashed red arrow: feedback loop — the actuator’s deployed-mitigation state propagates back to the generator, allowing the post-defense grid-health composite (Section 3.4.2) to be sampled from the next feature window. The remainder of this section walks the loop in data-flow order: generator (Section 3.2), ingestion and aggregation (Section 3.3), RL classifier (Section 3.4), GT defense selector (Section 3.5), and actuator coupling (Section 3.6). Section 3.7 discusses cross-cutting design choices — window cadence, fail-loud error handling, and the four-phase AI-auditor correctness remediation that preceded the experimental campaign. 3.2 Synthetic SCADA Generator Origin and motivation. The SCADA generator was developed as a controlled, fully- instrumented evaluation environment in response to testbed-integration constraints encountered during the project. The original program plan called for the integrated ADS/IDS and mitigator components developed at ANL and UND to run against a hardware-in-the-loop (HIL) testbed maintained by a second university partner, with the partner team responsible for deploying the attack suite on that testbed and coupling it AdCyDER: Online RL with Stackelberg-Optimized Defenses 19 to the upstream detection and downstream actuator components. The framework was in fact deployed to the partner testbed, but the planned attack-suite deployment and ADS/IDS/mitigator coupling on the partner side did not materialize by the time of this report. The testbed’s actuator configuration also supported only firewall-level traffic blocking — a single action within the broader defense action space (rate limit, quarantine, redirect, and others) that the GT engine and actuator framework were designed around — and that action was not implemented during the project. The other mitigations developed by UND and ANL were not feasible to instantiate on the testbed as configured. To enable end-to-end evaluation of the integrated RL/GT loop, the team built the synthetic SCADA generator described below, in which all twelve attack types and the full defense action space can be exercised together against physically-grounded inverter telemetry. This substitution is a meaningful limitation of the work reported here: the synthetic generator captures attack-type variety, time-alignment across data streams, and an autoencoder-based physics anomaly signal, but it does not substitute for the operational realism that a HIL testbed would have provided. We return to this point in Section 7 as the highest-priority next-step environment for re-testing the underlying hypothesis. Components. The SCADA generator produces time-aligned synthetic data representing five DER facilities, each running the DNP3 protocol typical of inverter-tied distributed generation; its source resides under src/stack/dockerfiles/adcyder-scada-generator/. The five simulated facilities use a single Fronius Primo inverter profile (same nameplate capacity, same calibration parameters, same anomaly-detector model and thresholds) and differ from one another only by their IP subnet (172.21.0.0/24 through 172.25.0.0/24) and EID label. Each facility draws independently from a shared campaign-selection dis- tribution (CAMPAIGN MIX in settings.py), so the realized attack streams across facilities are independent random samples from the same expected distribution; a single shared CampaignSequencer additionally enforces history-aware constraints (no immediate repeti- tion, reconnaissance-to-exploitation chaining) across all five facilities globally. The result is that cross-facility variation in observed data comes from independent attack scheduling rather than from facility-intrinsic differences. Operational facility diversity that would mat- ter in production — vendor, firmware, nameplate capacity, baseline traffic load, aggregator interfaces, distribution-feeder topology — is intentionally unmodeled in this campaign. The per-facility classifier architecture is designed for the operational case where facilities additionally differ in physical and network characteristics; our simulation therefore tests only the data-volume aspect of per-facility specialization (each per-facility classifier sees one-fifth the data the pooled cross-facility variant sees) rather than the physical-diversity aspect. We discuss the implication for the supervised LSTM ceiling result in Section 5.9 (the ≈ 1.2× pooled-vs-per-EID gap is a data-volume effect alone, not a physical-diversity effect; in production with genuine facility diversity the gap could be larger or smaller depending on whether per-facility specialization gains outweigh the data-volume penalty), and as a limitation in Section 6.5. The generator orchestrates the twelve attack classes enumerated in Table 1 (eleven attack types plus NORMAL background traffic) and emits four parallel streams per facility: 20 Blakely et al. (1) Inverter physics telemetry from a single-diode Fronius Primo solar inverter model, calibrated against published Fronius datasheet values. Four primary channels are produced per timestep: AC phase current (AphA), DC current (DCA), DC voltage (DCV), and AC phase voltage (PhVphA). Attack patterns perturb these channels through per-attack-type effect functions that model the physical consequences of the attack on inverter operation. (2) Network flow records (netflow) consistent with the attack pattern — packet counts, byte counts, source/destination addresses, port distributions, and protocol fields. When the actuator framework reports an active mitigation against a source IP, the generator’s traffic shaper applies the corresponding effect: DENY collapses the flow to a single 64-byte residual; RATE LIMIT scales packets and bytes by the configured ratio. (3) Suricata-format IDS alerts synthesized by an alert generator that maps attack patterns to representative Suricata signatures (ET SCAN, ET TROJAN, etc.) at realistic rates of 47–200 alerts per attack flow. Alerts include timestamp, source/destination, alert category, and severity, matching the field structure that the OpenSearch aggregator queries downstream. (4) Attack ground truth records containing the attack class, target facility, source IP, and time window for every attack injection. These records are stored in a dedicated MongoDB collection and used only for offline analysis — the RL classifier does not see them at training or inference time. AdCyDER: Online RL with Stackelberg-Optimized Defenses 21 Table 1. Attack taxonomy used by the SCADA generator. Signal layer : Net = expected on network-flow / IDS layer; Phys = expected on inverter-physics / ADS layer; Both = expected on both. Attack class Group Signal layer Description NORMAL — — Benign background traffic and inverter op- eration. DATA EXFILTRATION Core Net Unauthorized data theft over established channels. DOS DDOS Core Both Volumetric denial-of-service against inverter or facility ingress. CONTROL OVERRIDE Core Both Manipulation of inverter setpoints or control commands. PROBE SCAN Extended Net Network reconnaissance and port scanning. BRUTE FORCE Extended Net Repeated authentication attempts against control interfaces. MALWARE C2 Extended Both Malware command-and-control beaconing from compromised host. LATERAL MOVEMENT Extended Net Internal network movement between facility hosts. PRIVILEGE ESCALATION Extended Both Local rights elevation on a compromised host. REPLAY ATTACK Protocol-aware Net Replay of previously captured DNP3/Modbus control commands. UNAUTHORIZED CONTROL Protocol-aware Both Unauthenticated or unencrypted control at- tempts exploiting protocol-base authentica- tion absence. PROTOCOL VIOLATION Protocol-aware Both Malformed or non-compliant DNP3/Modbus messages. The Signal layer column in Table 1 reflects the expected observability of each attack class given the generator’s two upstream signal sources (Suricata IDS at the network layer, autoencoder ADS at the inverter-physics layer). The Q1 per-class detection-rate readings in Section 5.1 are interpreted against this expected layer assignment. 3.2.1 Embedded Autoencoder Anomaly Detector (ADS). A trained Keras autoencoder devel- oped by the ANL CEEESA team [13] (a related preliminary perturbation-based detection framework by the same authors appears in [40]) runs inside the generator container and scores each four-channel inverter telemetry sample for reconstruction error. The autoencoder is a fixed-architecture sequence-to-sequence model that takes a short window of past teleme- try and reconstructs the next sample; reconstruction error is exposed as the per-sample anomaly score. Attack-induced perturbations to the inverter channels produce reconstruc- tion error spikes when the perturbation pattern departs from the training distribution. The autoencoder is loaded from a pre-trained model artifact at container startup and is not retrained during the closed-loop campaign — this isolates ADS performance from any drift in the upstream generator. The output of the embedded ADS is consumed by the downstream feature aggregator as a continuous anomaly score. A per-feature reconstruction-error vector covering the two reconstruction-target channels (AphA, DCA) is also produced by the autoencoder and 22 Blakely et al. emitted on the ADS event payload as instrumentation for future per-channel diagnostic analysis. This per-feature data was not propagated through the OpenSearch aggregator into the MongoDB feature store and was not analyzed within the scope of this report; the raw ADS events remain in the OpenSearch index and could be re-extracted to support such an analysis as future work. 3.2.2 Mitigation State Propagation. The closing arrow in Figure 1 carries the actuator’s deployed-mitigation state back into the generator’s mitigator state module. This is the mechanism by which the simulated environment reflects actions taken by the defense pipeline. For example, once the GT engine has issued a DENY command against a source IP, the generator’s subsequent flow synthesis observes the suppression, the inverter physics observes the (now reduced) attack intensity, and the ADS aggregates over the window therefore reflect a partially-defended environment. Without this propagation the loop would be open — the generator would produce attack traffic identical regardless of any defense action, defeating the purpose of measuring grid-health-delta as a learning signal. 3.3 Ingestion and Feature Aggregation The ingestion path moves raw streams from the SCADA generator into queryable storage suitable for the RL/GT loop’s read patterns. Three infrastructure choices determine the path’s behavior, and the rationale for each is worth recording so that future deployments evaluating substitutions can reason about the trade-offs. Kafka as broker. The four data streams (inverter telemetry, netflow, IDS alerts, ground truth) are produced at independent cadences by the generator and consumed at different cadences by downstream services (Fluent Bit at ingest cadence, the aggregator on a 5-minute window cadence, the RL classifier on training cadence). A pub-sub broker decouples the producer from the consumers and lets each consumer apply its own backpressure and retry policy without the producer needing to know about them. Kafka was chosen over alternatives (Redis Streams, RabbitMQ, NATS) for its at-least-once delivery semantics, durable replay (so a re-instantiated consumer can re-process from a chosen offset rather than losing messages), and operational familiarity within the project team. RabbitMQ is also present in the deployment but serves a different role (RPC-style request/reply for actuator command acknowledgements, not bulk telemetry transport). OpenSearch as searchable telemetry store. The downstream feature aggregator does not consume the raw event stream linearly; it queries time-windowed slices of telemetry per facility and per source IP. OpenSearch (an Elastic-compatible distributed search index) provides the time-range and field-level filtering this query pattern needs at the data scale we generate (104 –106 events per hour across the five-facility deployment), without per- aggregator-call full table scans of the underlying telemetry. A Fluent Bit indexing pipeline (replacing an earlier Logstash configuration retired for resource-overhead reasons) writes Kafka-resident events to OpenSearch indices on a per-stream basis. MongoDB as feature store and operational state. The aggregator’s per-window feature vectors, the RL classifier’s per-window output (predicted class, confidence, attack-probability distribution, feature vector), the GT engine’s per-decision logs, the reward records with AdCyDER: Online RL with Stackelberg-Optimized Defenses 23 grid-health decomposition, and the ground-truth attack records are all persisted to Mon- goDB. A document store rather than a relational database fits the schema variety here (each collection has a different field set, several fields are nested objects), and the op- erational queries are point lookups by composite key ((eid, src ip, window id)) and time-range scans rather than complex joins. MongoDB’s secondary-index support on (eid, timestamp) and patterned id provides the per-facility and per-window scoping the analy- sis pipeline needs without a query-planner-driven schema. The aggregator’s source resides under src/stack/dockerfiles/opensearch aggregator/. 3.3.1 Per-Window Feature Vectors. The aggregator runs on a fixed 5-minute cadence (RL WINDOW DURATION MS = 300000) and produces one feature record per (Equipment ID, source IP, window-start) tuple. For each window the aggregator queries OpenSearch for all netflow, IDS alert, and ADS samples within the window and joins them into a single record stored in the MongoDB features collection. The RL classifier subsequently aggregates each per-window record into a 22-dimensional numeric feature vector with the following layout. All values are scalar counts, fractions, or scalar statistics — the LSTM input contains no categorical or list-valued fields: ∙ Netflow (5): total packets, total bytes, flow count, mean packet size, and unique dst port count (a scalar count of distinct ports, not the port values them- selves). ∙ Temporal (3): hour sin, hour cos (a circular encoding of window-start hour-of- day), and duration actual (window time-span in seconds). ∙ Destination IP hash (4): a 4-bin hashing-trick encoding of the most-frequent destination IP within the window. Captures which host(s) within the facility were targeted, without exposing raw IP values to the classifier. ∙ IDS (2): per-alert-category sums over the window, sorted by category key for deterministic ordering. The number of IDS feature dimensions is determined at startup from the IDS schema observed in the data; for the configuration evaluated here, two categories were active. ∙ ADS (3): mean anomaly (window mean reconstruction error, including zero-scored samples), max anomaly (window peak), anomaly flag rate (fraction of samples whose reconstruction error exceeded the per-feature anomaly threshold). Three statistics are necessary because the autoencoder’s output is bimodal — most samples score near zero with occasional large spikes, so a single mean masks attack signal. ∙ SCADA (5): two protocol-fraction features (protocol modbus frac and protocol dnp3 frac) and three operation-rate features (write rate, control rate, and error rate). The window-id format is {eid}:{src ip}:{window start ms}, which provides natural per-attacker, per-facility, per-window scoping for downstream join operations. 3.3.2 MongoDB Collections. Table 2 summarizes the principal MongoDB collections written by the pipeline. All collections are indexed on (eid, timestamp) and additionally on patterned id for cross-collection joins. 24 Blakely et al. Collection Contents features Per-(eid, src ip, 5-min window) joined feature vectors. Source of all classifier input. rl classifications Per-window classifier output: predicted class, confidence, attack-probability distri- bution, feature vector, baseline mode tag. rl rewards Per-window reward records: grid delta, scenario tag, baseline mode, original reward (if shuffled). attack ground truth Per-attack ground truth: attack id, attack type, target eid, source IP, time window. gt decisions Per-decision GT engine state: chosen defense, expected loss, top alternatives, Stackelberg analysis metadata. pending outcomes Pending grid-health-delta lookups awaiting the post-defense aggregation window. Persisted (not in-memory) for crash safety. Table 2. Principal MongoDB collections written by the pipeline. 3.4 Reinforcement Learning Attack Classifier The RL classifier (src/stack/dockerfiles/attack classifier/src/rl engine.py) is a per-facility LSTM that consumes per-window feature vectors and produces per-attack-type probability distributions rather than hard-label predictions. The probability-distribution output (rather than argmax classification) is a deliberate design choice: real-world attacks are diverse and frequently combine elements of more than one of the twelve catalog classes (e.g., a reconnaissance scan that blends PROBE SCAN and BRUTE FORCE traffic patterns, or a malware-driven control deviation that overlaps MALWARE C2 and CONTROL OVERRIDE), and a defender selecting a mitigation gains more from a hedged distribution over plausible attack types than from an overconfident single-label commitment that may mis-target the response. Each facility is identified by its aggregator-device Equipment ID (EID), and a separate model instance is instantiated and trained per EID; per-facility traffic patterns drive per-facility classification weights without inter-facility cross-contamination. 3.4.1 Architecture. Each per-facility classifier is a Keras Sequential model. We use Keras for the classifier because the ANL CEEESA autoencoder ADS (Section 3.2) was published as a Keras artifact, and adopting the same framework throughout the pipeline avoids cross-framework serialization, version-pinning, and runtime-image-size friction. The layer stack, identical across all facilities and reproduced in the supervised LSTM ceiling test described in Section 4: (1) Input of shape (RL SEQUENCE LENGTH=3, feature count=22): a sequence of three consecutive 5-minute window feature vectors — a 15-minute lookback. The sequence length is configurable via environment variable but was left at the default for the May 2026 evaluation campaign. (2) LSTM(128, return sequences=True) → Dropout(0.2). (3) LSTM(64) → Dropout(0.2). (4) Dense(32, activation='relu') → Dropout(0.2). (5) Dense(12, activation='softmax'): the per-attack-type probability distribution. The model is compiled with the Adam optimizer (default learning rate 1e-3) and sparse categorical cross-entropy loss. Inference at runtime requires only the last three windows AdCyDER: Online RL with Stackelberg-Optimized Defenses 25 for the relevant (EID, src ip) tuple, which the classifier maintains in a per-facility sliding sequence buffer of length 2×RL SEQUENCE LENGTH. 3.4.2 Grid-Health Composite Score. The grid-health score that drives the reward signal is a per-facility composite of the autoencoder-based ADS anomaly score and a rolling-baseline IDS alert density, computed in the GT engine (gt engine.py:: get current grid health) as grid health(𝑒𝑖𝑑) = 𝑤ADS · 𝑠ADS (𝑒𝑖𝑑) + 𝑤IDS · 𝑠IDS (𝑒𝑖𝑑), 𝑤ADS = 𝑤IDS = 0.5 where both sub-scores are in [0, 1] with 0 indicating normal operation and 1 indicating maximum anomaly. The composite is computed at decision time from MongoDB state, not pre-aggregated. The 50/50 weighting reflects the modeling decision that physical- layer attacks (visible to the autoencoder) and network-layer attacks (visible to Suricata) are a priori equally important to the loop’s notion of facility health; both weights are configurable via the GRID HEALTH ADS WEIGHT and GRID HEALTH IDS WEIGHT environment variables. The two sub-scores are not symmetric in expected firing rate: of the twelve attack classes (Table 1), only three (CONTROL OVERRIDE, UNAUTHORIZED CONTROL, severe DOS DDOS) have mechanisms that perturb inverter physics in a way the autoencoder is positioned to detect, while all twelve generate network-observable traffic and are therefore reachable by Suricata. The composite reward is consequently expected to be IDS-dominated in aggregate even when ADS calibration is correct; this is a property of the attack-catalog composition, not an asymmetry to be tuned away. Section 5.1 returns to this point with quantitative per-class detection rates. The ADS sub-score 𝑠ADS (𝑒𝑖𝑑) is the most recent autoencoder reconstruction-error-derived anomaly score for the facility, taken from the latest MongoDB features document with an ads.anomaly score field. If no ADS data is available, or if the most recent document is a sentinel marking ADS unavailability at ingestion time (Section 3.3), the GT engine raises GridHealthUnavailableError and the reward computation for that pending outcome is skipped — the loop fails loudly rather than silently substituting a default value. The IDS sub-score 𝑠IDS (𝑒𝑖𝑑) is computed by normalizing the facility’s recent IDS alert count against a per-facility rolling baseline. The raw count is the sum of src ip alerts and dst ip alerts across MongoDB features documents from the last 5 minutes (IDS LOOKBACK MINUTES = 5). A per-facility exponentially-weighted moving-average baseline 𝑏(𝑒𝑖𝑑) is updated on every call as 𝑏 ← 𝛼 · raw count + (1 − 𝛼) · 𝑏 with 𝛼 = 0.1 (IDS BASELINE ALPHA); the baseline is initialized to the first observed raw count when the facility is first seen. The score is the clipped relative excess over baseline, (︁ (︁ raw count − 𝑏(𝑒𝑖𝑑) )︁)︁ 𝑠IDS (𝑒𝑖𝑑) = min 1.0, max 0, , 𝑏(𝑒𝑖𝑑) + 1 yielding 𝑠IDS ≈ 0 when the facility’s alert count is at or below its rolling baseline and 𝑠IDS approaching 1 when the alert count is many multiples above baseline. The per-facility EMA naturally accommodates facilities with different background alert rates without manual per- facility tuning. IDS unavailability (e.g., the MongoDB query fails or returns no documents) is not treated as fatal; 𝑠IDS falls back to 0 and the composite reduces to ADS-only for that 26 Blakely et al. observation, on the rationale that a quiet IDS is the expected normal state and absence of alerts should not block reward computation when ADS data is present. The signed grid-health delta used as the training signal is then Δ = grid health after − grid health before: negative delta means the composite anomaly score decreased after the deployed defense (grid health improved), and positive delta means it increased (grid health degraded). The same composite definition is used for both the pre-defense and post-defense observations so that the delta is meaningful as a difference. 3.4.3 Online Training from Grid-Health Feedback. The classifier is trained online from delayed grid-health reward rather than from labeled attack data. The training cycle is: (1) For each completed classification window, the classifier records the (EID, src ip, window start) identifier in a pending grid outcomes dict (also persisted to the MongoDB pending outcomes collection for crash recovery), along with the pre- defense grid-health composite (Section 3.4.2) sampled at the time of the GT decision. (2) After the configured grid-health observation delay (defense-dependent, in the 5– 60 minute range), the GT engine recomputes the grid-health composite for the same EID and computes grid delta = grid health after − grid health before using the composite definition of Section 3.4.2. A negative delta indicates the composite anomaly score decreased (defense worked); positive delta indicates degradation. (3) The shaped reward is computed as 𝑟shaped = 𝑐 · tanh(𝑟raw /𝑐) where the clamp scale 𝑐 is set by the constant REWARD SOFT CLAMP SCALE = 50 and 𝑟raw is the unclamped raw reward, composed as (base reward + ids bonus) · severity multiplier · magnitude multiplier. The base reward depends on the defense scenario — a correctly-defended attack scores +20, a missed attack −15, a correct ALLOW on benign traffic a small positive value, a wrong defense on benign traffic between −2 and +2, and so on. The tanh soft-clamp serves two purposes. First, it bounds reward magnitude to ±50, preventing any single feedback event with a pathological raw reward (e.g., a transient grid-health-delta spike unrelated to the deployed defense, or compounding multiplier interactions) from dominating an online weight update and destabilizing training. Second, tanh saturates smoothly rather than hard-clipping: rewards in the small-to-moderate range (where the signal is most informative) pass through approximately linearly and preserve their relative ordering, while extreme rewards saturate gracefully toward the clamp scale instead of being truncated. The smooth saturation keeps the gradient well-behaved across the full input range, in contrast to a hard clip that would zero the gradient outside the clip region. (4) Reward records are persisted to rl rewards with a baseline mode tag for down- stream comparative analysis. (5) The accumulated experience buffer is used to retrain the per-facility classifier in batches. The main loop wakes every RL SLEEP INTERVAL seconds (default 300, i.e. matching the window cadence) and triggers a training pass whenever at least 10 new reward records are available, using minibatches of 10–100 records per pass. Under this configuration training is essentially continuous over the lifetime of the deployed classifier — not gated to any external cadence. AdCyDER: Online RL with Stackelberg-Optimized Defenses 27 3.4.4 Baseline Modes. The classifier supports three configurations selected at startup by the RL BASELINE MODE environment variable: ∙ online (full configuration): LSTM trained continuously from grid-health-delta reward as described above. ∙ frozen: trains normally for the first RL FREEZE AFTER HOURS (default 3 hours), then permanently freezes weights. Inference and feedback consumption continue, but no further weight updates are made. The frozen-snapshot reward records are still logged to MongoDB for comparative analysis. ∙ gt only: bypasses LSTM inference entirely. Each window’s attack-probability dis- tribution is set to a uniform 1/12, classifier confidence is reported as 1/12, and no training occurs. The GT engine still selects defenses based on the uniform prior and grid-health rewards are still logged. This is the no-classifier control: the Stackelberg layer operates on a non-informative prior. 3.5 Stackelberg Game-Theoretic Defense Selector The GT engine (src/stack/dockerfiles/mitigator/src/gt engine.py) consumes the per-window attack-probability distribution from the RL classifier and selects a defense action by minimizing expected loss under SME-encoded effectiveness, impact, and cost parameters. Defenses are drawn from the discrete DefenseAction set enumerated in Table 3. Table 3. Defense action set available to the GT engine. Cost: SME-encoded Cost(𝑑) in the loss functional of Section 3.5. Real-time: Y = deployable in the per-window decision cycle; N = Eff(𝑑, 𝑡) = 0 across all 𝑡 because the action is not deployable at the 5-minute cadence. Defense action Description Cost Real-time? ALLOW Pass traffic with no active intervention. 0 Y DENY Block/drop matching traffic at the network 20 Y edge. RESET CONNECTION Force TCP RST on the offending session. 25 Y RATE LIMIT Throttle matching traffic to a configured ratio. 30 Y DEEP INSPECT Enable enhanced packet inspection on matching 40 Y traffic. REDIRECT Redirect traffic to a sinkhole or honeypot. 60 Y SEGMENT NETWORK Apply network segmentation/isolation rules. 80 Y INVESTIGATE Trigger forensic analysis (information-gathering 80 Y* only). SCAN Run an active vulnerability scan against the 20 Y* source. QUARANTINE Isolate the affected host from the network. 100 Y CONTAIN Apply micro-segmentation to limit blast radius. 120 Y DEPLOY HONEYPOTS Stand up new honeypot infrastructure. 5000 N PATCH CVES Patch a known vulnerability (requires down- 10000 N time). * INVESTIGATE and SCAN are real-time-deployable but their effectiveness is set to 0 (INVESTIGATE ) or a uniform 0.3 (SCAN ) across all attack types in the current SME-encoded matrix; they are retained in the enumeration but rarely chosen by the loss minimizer. 28 Blakely et al. Of the thirteen actions in the enumeration, four (PATCH CVES, DEPLOY HONEYPOTS, ALLOW, and INVESTIGATE) have Eff(𝑑, 𝑡) = 0 across all 𝑡 in the current SME-encoded matrix — either because they cannot be deployed within the per-window decision cadence (PATCH CVES, DEPLOY HONEYPOTS), because they represent the no-defense option (ALLOW), or because they gather information rather than mitigate (INVESTIGATE). The eight active defenses with non-uniform effectiveness across attack types are shown in the effectiveness matrix of Table 4 below. 3.5.1 Loss Formulation. For each candidate defense action 𝑑, the engine computes the expected loss ∑︁ (︀ )︀ 𝐿(𝑑) = 𝑝(𝑡) · 1 − Eff(𝑑, 𝑡) · 𝐵(𝑡) · 𝑠(𝑡) + Cost(𝑑), 𝑡∈𝑇 where 𝑇 is the set of attack types, 𝑝(𝑡) is the classifier’s probability that attack type 𝑡 is active, Eff(𝑑, 𝑡) ∈ [0, 1] is the SME-encoded effectiveness of defense 𝑑 against attack type 𝑡, 𝐵(𝑡) is the SME-encoded base impact of attack 𝑡, 𝑠(𝑡) ∈ [0, 1] is the per-attack-type severity estimate from the classifier, and Cost(𝑑) is the SME-encoded cost of deploying defense 𝑑. The defender chooses 𝑑* = arg min 𝐿(𝑑). 𝑑∈𝐷 Decision metadata is persisted to the gt decisions collection for offline analysis: the chosen defense, its expected loss, the top three alternatives by loss, and a stackelberg analysis record containing the per-attack-type contributions to the residual loss term and the engine’s reasoning string. 3.5.2 SME-Encoded Parameters. The effectiveness matrix Eff(𝑑, 𝑡) has been calibrated to produce seven distinct optimal defenses across the twelve attack types, ensuring meaningful action diversity rather than collapse to a single dominant defense. The active portion of the matrix is shown in Table 4. Base impact values 𝐵(𝑡) are scaled relative to a notional baseline grid impact, and defense costs Cost(𝑑) are scaled to make low-impact defenses (e.g., RATE LIMIT) preferred when uncertainty is high. A probability temperature parameter (𝑇 = 2.0) is applied to the input distribution 𝑝(𝑡) to prevent collapse of the GT loss landscape under high classifier confidence. AdCyDER: Online RL with Stackelberg-Optimized Defenses 29 Table 4. SME-encoded effectiveness matrix Eff(𝑑, 𝑡) ∈ [0, 1] for the eight active heterogeneous defenses. Bold = per-attack-type maximum before cost. Columns: SEG SEGMENT NETWORK, DEN DENY, RST RESET CONNECTION, QUA QUARANTINE, CON CONTAIN, RAT RATE LIMIT, RED REDIRECT, DPI DEEP INSPECT. Attack class SEG DEN RST QUA CON RAT RED DPI DATA EXFILTRATION 0.4 0.8 0.6 0.9 0.7 0.3 0.5 0.4 DOS DDOS 0.7 0.9 0.3 0.6 0.5 0.7 0.4 0.4 CONTROL OVERRIDE 0.7 0.4 0.5 0.8 0.6 0.2 0.3 0.4 PROBE SCAN 0.5 0.3 0.4 0.5 0.4 0.7 0.6 0.4 BRUTE FORCE 0.5 0.9 0.7 0.7 0.6 0.8 0.4 0.4 MALWARE C2 0.6 0.3 0.3 0.95 0.45 0.3 0.4 0.15 LATERAL MOVEMENT 0.7 0.2 0.5 0.8 0.9 0.4 0.3 0.4 PRIVILEGE ESCALATION 0.6 0.3 0.4 0.9 0.7 0.2 0.2 0.4 REPLAY ATTACK 0.6 0.6 0.8 0.85 0.75 0.4 0.5 0.4 UNAUTHORIZED CONTROL 0.7 0.9 0.8 0.9 0.8 0.5 0.6 0.4 PROTOCOL VIOLATION 0.5 0.3 0.7 0.7 0.6 0.6 0.5 0.4 Per-defense reading (columns of Table 4). Individual defenses specialize to particular attack-type clusters: QUARANTINE dominates host-compromise attacks (MALWARE C2, PRIVILEGE ESCALATION, DATA EXFILTRATION); DENY dominates volumetric and au- thentication attacks (DOS DDOS, BRUTE FORCE, UNAUTHORIZED CONTROL); CONTAIN is the lateral-movement specialist via micro-segmentation; RATE LIMIT is the network- reconnaissance specialist for PROBE SCAN; RESET CONNECTION is the session-attack specialist for PROTOCOL VIOLATION and replay-style traffic. Per-attack reading (rows of Table 4). For each attack class, the SME-judged effective defenses (effectiveness ≥ 0.6) are: ∙ DATA EXFILTRATION: QUARANTINE (0.9, host isolation), DENY (0.8, network block), CONTAIN (0.7, micro-segmentation). ∙ DOS DDOS: DENY (0.9, edge block), SEGMENT NETWORK (0.7), RATE LIMIT (0.7, traffic shaping). ∙ CONTROL OVERRIDE: QUARANTINE (0.8, isolate controller), SEGMENT NETWORK (0.7), CONTAIN (0.6). ∙ PROBE SCAN: RATE LIMIT (0.7, throttle reconnaissance), REDIRECT (0.6, sinkhole). ∙ BRUTE FORCE: DENY (0.9), RATE LIMIT (0.8, throttle attempts), QUARANTINE (0.7), RESET CONNECTION (0.7), CONTAIN (0.6). ∙ MALWARE C2: QUARANTINE (0.95, full host isolation), SEGMENT NETWORK (0.6). DENY is judged only weakly effective (0.3) because malware persists locally and uses alternate channels. ∙ LATERAL MOVEMENT: CONTAIN (0.9, micro-segmentation is the textbook countermea- sure), QUARANTINE (0.8), SEGMENT NETWORK (0.7). ∙ PRIVILEGE ESCALATION: QUARANTINE (0.9), CONTAIN (0.7), SEGMENT NETWORK (0.6). Network-level defenses (DENY, RATE LIMIT) are judged weak because the exploit is local. 30 Blakely et al. ∙ REPLAY ATTACK: QUARANTINE (0.85), RESET CONNECTION (0.8, terminates the replay session), CONTAIN (0.75), DENY (0.6), SEGMENT NETWORK (0.6). ∙ UNAUTHORIZED CONTROL: DENY (0.9, blocks unauthorized commands), QUARANTINE (0.9), CONTAIN (0.8), RESET CONNECTION (0.8), SEGMENT NETWORK (0.7). ∙ PROTOCOL VIOLATION: RESET CONNECTION (0.7, terminates malformed session), QUARANTINE (0.7), RATE LIMIT (0.6), CONTAIN (0.6). After the cost penalty Cost(𝑑) is applied to the loss functional, the cost-adjusted optimal defense for some attack types shifts: SEGMENT NETWORK (cost 80) is sometimes preferred over the higher-effectiveness QUARANTINE (cost 100), and RATE LIMIT (cost 30) is preferred over DENY (cost 20) for attack types where their effectiveness is comparable. The matrix is SME-encoded and not empirically calibrated; sensitivity to perturbations of these values is a stated limitation (Section 6.5). 3.5.3 Boltzmann Exploration. To prevent the GT engine from locking onto an early-learned defense for any given attack-probability shape, the engine samples defenses from a Boltzmann distribution over the loss landscape with a decaying temperature schedule (initial 𝑇0 = 50, decay 0.9999 per decision, minimum 𝑇min = 1.0). Early in training the engine selects defenses approximately stochastically; at long-run convergence it approaches the deterministic argmin. When the configured temperature is set to zero or below, the engine selects the deterministic argmin directly — this mode is used by the unit test suite. Trial-duration interaction. The decay schedule is wall-clock-time based, not decision-count based: 𝑇 (𝑡) = 𝑇0 · exp(−𝜆𝑡) where 𝑡 is wall-clock seconds since the GT engine’s exploration start. The decay rate 𝜆 is sized via the GT EXPLORATION DECAY TARGET HOURS configuration parameter (default 8 hours) so that the temperature decays from 𝑇0 = 50 to 𝑇min = 1 over exactly one trial duration. By design, then, each 8-hour evaluation iteration begins in the exploration regime (𝑇 = 50) and arrives at the near-deterministic regime (𝑇 ≈ 1) by trial end. Container restart at the start of each iteration resets exploration start time so each iteration gets a full exploration arc rather than persisting a decayed temperature across iterations. In persistent production deployment (no per-iteration restart) the temper- ature would continue decaying past 𝑇min at which point the engine would operate purely deterministically; the same configuration parameter governs the decay rate in both regimes. 3.6 Actuator Framework and Closed-Loop Coupling The actuator framework (adcyder-actuator/ submodule) translates the GT engine’s chosen defense into an OpenC2 action command (Section 2.1) and delivers it over MQTT to the responder enclave for the targeted facility. Each command carries the action verb (DENY, ALLOW, RATE LIMIT, etc.), the target specifier (source IP, port, protocol), and metadata including the originating attack-type prediction and the GT engine’s confidence. For the synthetic-data evaluation reported here, the actuator’s role is twofold: (i) record receipt of the command (and acknowledge to the GT engine, which then schedules the post- defense grid-health observation), and (ii) update the SCADA generator’s mitigator state so that subsequent traffic and physics reflect the deployed defense. The responder framework AdCyDER: Online RL with Stackelberg-Optimized Defenses 31 supports rate limiting, IP-based deny, and subnet-quarantine actions; subnet quarantine simulates physical isolation by switching the entire facility to NORMAL traffic for the duration of the mitigation. 3.7 Cross-Cutting Design Choices 3.7.1 Window Cadence and Per-EID Isolation. The 5-minute aggregation window is a de- liberate compromise between feedback latency (shorter is better for online learning) and statistical power per window (longer is better for distinguishing attack from normal). At 5 minutes a single window sees enough netflow and ADS samples to produce stable aggregates, while still being short enough that the training loop — which wakes on the same cadence and trains whenever at least 10 new reward records are available — can update weights frequently. Per-facility isolation prevents one facility’s classifier weights from influencing another’s; this is essential to the original learning hypothesis (per-facility adaptation) but also creates a per-facility data-volume constraint that the supervised LSTM ceiling test (Section 5) shows is consequential. The 8-hour duration of each evaluation iteration in the May 2026 campaign was an arbitrary wall-clock budget chosen to keep individual trial runs tractable, not a system parameter — the deployed classifier would continue training and inferring indefinitely. 3.7.2 Fail-Loud Error Handling. The pipeline policy is to raise on missing or sentinel data rather than substitute defaults. When the OpenSearch aggregator cannot produce an ADS aggregate for a window (e.g., zero samples in the window), it writes a sentinel record (anomaly score = -1.0, anomaly score missing = True) rather than imputing zero. The GT engine raises a custom GridHealthUnavailableError on encountering a sentinel rather than treating it as “healthy.” The RL engine’s grid-outcome registration explicitly skips sentinel records so they cannot contaminate the reward distribution. Diagnostic logging fires a loud ERROR after three consecutive all-sentinel windows so that pipeline degradation is visible in the operations log rather than silent. 3.7.3 AI-Auditor Correctness Remediation. Prior to the May 2026 evaluation campaign, the integrated pipeline was subjected to three independent AI-driven correctness audits using Claude Opus, Gemini 2.5 Pro, and GPT 5.2 as adversarial reviewers. The audits identified six independent classes of pipeline bugs spanning data integrity (reward sign inversions, JSON double-encoding severing the GT→RL feedback link), feedback-loop construction (cross-facility model contamination via patterned id format mismatch), command delivery (lost OpenC2 acknowledgements), and resilience (in-memory pending-outcome state lost on container restart). All findings were resolved across four structured remediation phases with regression tests added for each fix; the integrated system now passes 335+ automated tests across the main repository and two service submodules. The remediation history is the audit trail demonstrating that the experimental campaign reflects the corrected pipeline rather than a configuration with known correctness bugs. 32 Blakely et al. 3.7.4 Software Development Lifecycle Controls. The AI-auditor remediation described above was a one-time, event-driven correctness sweep. Behind it sits a continuous software devel- opment lifecycle (SDLC) apparatus applied across all three repositories (main + two service submodules) for the duration of the project, intended to keep the pipeline continuously in a known-correct state between audits. Each repository carries an identical make cistage target that aggregates the four manda- tory pre-merge checks: code formatting (black), strict static type checking (mypy --strict), security scanning (bandit for code-pattern issues plus pip-audit for dependency CVEs), and the unit test suite. Two additional make targets (make test-integration, make test-synth) cover end-to-end and synthesis-pipeline behaviors that are heavier than the per-merge unit suite. The same pipeline is enforced automatically on every merge request and main-branch push by the GitLab CI/CD configuration in each repository’s .gitlab-ci.yml, so that no code reaches main without passing the full check-set on a clean container build. Process controls layered on top of these mechanical checks include: feature-branch devel- opment with merge-request review (no direct main commits), conventional-commit messages ((scope): description) for searchable history, semantic versioning of pinned de- pendencies (X.Y.*), no-fallback / fail-loud error handling as documented in Section 3.7 (Section 3.7.2), and a project-wide policy against silent default substitution that is en- forced by both code review and the integration tests. For research-code-specific concerns, every machine-learning component carries verification scaffolding (shape assertions at each pipeline stage, parameter-count validation, gradient-norm logging where applicable) so that broken-but-passing configurations are detectable from logs alone. The combination of pre-merge gates, end-to-end integration coverage, and process discipline is what made the late-stage diagnostic effort possible: when figures revealed unexpected behavior, the under- lying pipeline could be trusted as the corrected configuration rather than a configuration drifting unobserved between audits. 3.7.5 Independent Manual Security Review. In addition to the automated security tooling described above (bandit for static code analysis, pip-audit for dependency CVE scanning), the integrated pipeline underwent a manual red-team security review by Synack (synack.us) prior to the May 2026 evaluation campaign. The review identified no security findings against the integrated pipeline itself. The only items reported were two notes recommending that HTTPS be enabled on a peripheral monitoring service that sits outside the primary data path. The full report is available from the project team upon request. 3.8 Deployment and Infrastructure The integrated pipeline runs as a set of Docker-containerized services orchestrated by an Infrastructure-as-Code (IaC) configuration that targets either AWS (the original deployment target) or LocalStack (the local emulator that became the primary deployment target after the AWS-access pivot described in Section 3.2). The same source tree produces both deployments; the choice is parameterized by a single configuration value. 3.8.1 Containerization. Every service in the pipeline is packaged as a Docker container with a dedicated Dockerfile located under src/stack/dockerfiles// in the AdCyDER: Online RL with Stackelberg-Optimized Defenses 33 source tree. Pipeline services include the SCADA generator, Kafka, OpenSearch, Fluent Bit consumer, OpenSearch aggregator, MongoDB, RL classifier, GT mitigator, MQTT broker, the actuator framework, and a RabbitMQ broker for the actuator-acknowledgement RPC channel. We follow the one-process-per-container convention, with each service ex- posing only the ports it needs for declared inter-service interfaces. Container images use officially-tagged base images and run as non-root users where possible. Service environ- ment variables are injected at container startup from the IaC configuration; service code reads its own configuration through a pair of files — settings.py (committed defaults) and settingslocal.py (gitignored local overrides) — which the container loads from its working directory. Container behavior is therefore fully reproducible from the committed stack definition plus the recorded environment-variable values. 3.8.2 Infrastructure as Code (CDKTF). The deployment topology — service definitions, networking, storage volumes, and inter-service connections (e.g., RL classifier → MongoDB, mitigator → MQTT) — is declared in CDKTF (Cloud Development Kit for Terraform), a Python-based IaC framework that compiles to standard Terraform JSON manifests. Each service’s resources are declared once in CDKTF and rendered consistently for whichever target backend the stack is being synthesized against. We chose CDKTF over hand-written Terraform for two reasons: (i) Python lets the stack definition share validation logic and constants with the service code (e.g., the same MongoDB collection names are referenced from both the application and the IaC layer, eliminating a class of name-mismatch bugs), and (ii) the project team is more fluent in Python than in HCL. 3.8.3 AWS / LocalStack Switching. The same CDKTF stack synthesizes against either AWS or LocalStack with a single configuration value: an AWS ACCOUNT ID parameter that, when set to the LocalStack convention value 000000000000, routes every Terraform-defined service endpoint to the local emulator instead of the public AWS API. No service code changes are required to switch between the two; the application layer sees the same boto3 / AWS-SDK interfaces regardless of which backend is responding to the API calls. This property enabled the pragmatic deployment pivot described in Section 3.2: when AWS access proved infeasible, no rewrite or restructuring of the stack was required — only a re-synthesis of the same CDKTF configuration against the alternate endpoint set. The same property also allowed local development of the integrated pipeline without consuming AWS resources or budget, and supports the reproducibility commitments documented in Section 4.6: a future investigator with the source-tree commit and the recorded environment- variable values can re-instantiate the entire pipeline in either environment without further infrastructure work. 4 EXPERIMENTAL METHODOLOGY This section describes the experimental design used to evaluate the integrated RL/GT loop, the analytical pipeline applied to the resulting data, and the rationale for each design decision. Findings produced by this methodology are deferred to Section 5. 34 Blakely et al. 4.1 Evaluation Configurations The loop’s central scientific question — “does the online RL classifier, trained from delayed grid-health feedback, learn to produce probability distributions that drive effective defense selection through the Stackelberg layer” — decomposes into three falsifiable sub-questions. We evaluated one configuration per sub-question; together the three configurations form a control structure that allows the contribution of each component to be isolated. Mode Configuration Sub-question isolated online Full RL/GT loop. Per-facility LSTM Does continuously-updated online learning trains continuously from grid-health- produce defense-driving distributions? This delta reward; GT engine consumes classi- is the configuration of primary interest. fier output for every defense decision. frozen RL classifier trains for the first Does continued online training improve over RL FREEZE AFTER HOURS hours (set a static snapshot? If online performs no bet- to 3 for this campaign), then weights are ter than frozen, the online updates are not permanently frozen; GT engine continues adding value beyond what an early bootstrap operating with the frozen classifier as a captured. static prior. gt only LSTM inference bypassed entirely. The Does the classifier add value beyond the SME- classifier emits a uniform 1/12 probability encoded Stackelberg layer alone? If online distribution for every window; GT engine performs no better than gt only, the RL selects defenses against this uninforma- component is not contributing measurable tive prior. No training occurs. information. Table 5. The three baseline modes and the sub-question each isolates. Mode is selected by RL BASELINE MODE; all other pipeline components are identical. The three modes share an identical pipeline downstream of the classifier. Only the classifier’s training behavior and inference output differ. This is essential for the comparisons to be valid: any difference in measured outcomes can be attributed to the classifier’s contribution rather than to incidental configuration drift. We deliberately did not include a shuffled reward mode (which would randomize the reward-to-classification association at training time, testing whether the loop is learning from the reward structure or from spurious correlations). The mode was specified and partially implemented during planning, but removed before the campaign because the results from online versus gt only already provide the relevant control: if online produces no meaningful improvement over gt only, there is nothing for the shuffled-reward control to falsify. 4.2 Experimental Campaign The May 2026 evaluation campaign comprised 28 trial runs of 8 hours each (approxi- mately 224 hours of total runtime), distributed across the three modes: ∙ online: 16 iterations ∙ frozen: 8 iterations ∙ gt only: 4 iterations The allocation reflects the relative information content of each mode under the loop’s design. online received the largest budget because it is the configuration of primary interest AdCyDER: Online RL with Stackelberg-Optimized Defenses 35 and the only one whose classifier behavior evolves over time — maximizing its trial count gives the most opportunity to observe learning, if any. frozen received an intermediate budget because the frozen-weight regime is the principal comparison anchor for “does continued online learning add value beyond a static snapshot?” — multiple iterations sample distinct trained-then-frozen realizations rather than re-sampling a single fixed classifier. gt only received the smallest budget: its outputs are by construction stable (uniform priors, no training), so a smaller number of trials is sufficient to characterize its grid-outcome distribution. Trial duration. Each iteration ran for 8 hours of wall-clock time. This duration was an arbitrary trial budget chosen to (i) be long enough to accumulate ∼96 reward records per (EID, source-IP) tuple at the 5-minute window cadence, providing multiple per-EID training passes within the trial; (ii) be short enough to fit several iterations into a continuous evaluation window without exhausting compute; and (iii) start each iteration with a freshly- initialized classifier container, so that any cross-iteration state leakage in per-facility LSTM weights, sequence buffers, or pending-outcomes state was prevented by construction. The 8-hour duration is not a system parameter — the deployed classifier is designed to run indefinitely — it is solely an experimental-design choice for sampling. Attack scenarios. Every iteration exercised all twelve attack classes enumerated in Sec- tion 3.2 across all five simulated facilities. Attack campaigns were generated by the SCADA generator’s orchestrator with persistent attacker IPs per (campaign, attack type, subnet) so that windowed aggregation could collapse multiple flows from the same attacker into well- populated per-(EID, src ip) windows. Coordinated multi-facility campaigns ran concurrently across facilities to maximize attack diversity per training hour. Data products per iteration. Each iteration wrote classification, reward, GT-decision, ground-truth, and per-window feature records into the MongoDB collections described in Table 2. Aggregated across the 28 iterations, the campaign produced 6,031 classification records suitable for analysis: 3,429 from online, 1,700 from frozen, and 902 from gt only. The disparity in record counts versus iteration counts reflects per-iteration variation in the volume of (EID, src ip) windows that produced both a classification and a paired ground-truth record (some windows produced one without the other and were excluded from the analyzable record count). 4.3 Data Extraction and Analysis Pipeline 4.3.1 Per-iteration extraction. At the end of each evaluation block, the analyzable records for that block are extracted from MongoDB into a per-host JSON file by the helper script scripts/local results extraction.py. The extraction performs three joins: (i) rl classifications joined with attack ground truth on (eid, src ip, window id) to attach the true attack class to each classifier output; (ii) the same record set joined with rl rewards to attach the grid-health-delta reward and the GT decision-scenario tag; (iii) the joined record set augmented with three ADS aggregates re-computed from the features collection at per-classification-window granularity — ads mean all, ads max, 36 Blakely et al. and ads nonzero rate — because the LSTM-input ADS features (computed across the entire (eid, src ip, 5-min-window) feature record) collapse the bimodal autoencoder output into a single mean that masks attack signal. Records are filtered by an analysis cutoff timestamp to ensure consistent end-of-window inclusion across the analysis pool. 4.3.2 Pipeline-ordered diagnostic figures (Q1–Q7). Labeling convention. We use the project-internal labels Q1–Q7 throughout this report to refer to seven sequential pipeline-stage diagnostic figures. These labels appear consistently in the project’s tracking documents and supporting code, so we preserve them here for traceability. They are orthogonal to the three baseline modes of Table 5: a baseline mode (online, frozen, gt only) is a pipeline configuration, while a Q-figure is a diagnostic question about a specific pipeline stage that can be asked of any configuration. Each Q- figure in Section 5 therefore presents the same stage-specific measurement separately for each mode, so the mode comparisons (Table 5) and the stage diagnostics (Q1–Q7) can be read jointly. Pipeline ordering. Diagnostic figures are produced from the per-iteration JSON files by the central analysis script scripts/central analysis.py. They are deliberately ordered to follow the data path through the loop, so that each figure tests a distinct stage and a failure at any one stage explains failures downstream of it without contaminating the interpretation of upstream stages: Q1 (ADS upstream signal): does the autoencoder produce attack-vs-normal sepa- ration per classification window? Q2 (IDS upstream signal): does the network IDS produce attack-vs-normal sepa- ration per classification window? Q3 (per-class P(true) elevation): given that an attack of type 𝑡 is active, does the classifier elevate 𝑃 (𝑡) above the random baseline? Q4 (output distribution per true class): is the classifier’s output distribution in- formative about the true class, or is it concentrated on a small set of overconfident wrong classes? Q5 (classifier learning trajectory): does P(correct) improve over training-elapsed time within an iteration? Q6 (reward–correctness causal link): is there a meaningful correlation between P(correct) and grid-health-delta within the loop? Q7 (grid outcome trend): does grid-health-delta improve across training itera- tions? The pipeline ordering matters operationally as well as expositionally. If Q1 or Q2 reveals broken upstream signal, Q3–Q7 are uninterpretable and the diagnostic effort localizes to the upstream stages first. Conversely, if Q5 is flat but Q1, Q2, and Q3 show signal, the failure is downstream of the features — in the training procedure or the reward channel. We treat this falsification structure as a methodological contribution; it is the basis for the same-architecture supervised LSTM ceiling test that follows. AdCyDER: Online RL with Stackelberg-Optimized Defenses 37 4.4 Same-Architecture Supervised LSTM Ceiling Test The Q1–Q7 figures characterize what each pipeline stage produces but do not, on their own, distinguish between two distinct failure modes for the loop: ∙ Data-quality failure: the per-window features simply do not carry enough attack- discriminative signal for any per-EID LSTM to extract meaningful distinctions. Online learning would then be expected to fail regardless of the training procedure or the reward formulation. ∙ Training-procedure failure: the features carry signal that the LSTM architecture could in principle extract, but the online RL training procedure — with its delayed, sparse, soft-clamped grid-health-delta reward — fails to extract it. The same-architecture supervised LSTM ceiling test discriminates between these. It is im- plemented in scripts/supervised baseline.py and run via make supervised-baseline. We train an offline supervised LSTM whose per-facility architecture matches the production classifier exactly — all layers, optimizer, loss, sequence length, and feature width as defined in Section 3.4 — on the same per-window feature vectors that the production classifier saw, using gt attack type as the target label. The only variable that differs between the production classifier and the supervised ceiling is the training procedure itself. Choice of comparator. Same-architecture supervised offline training is the conservative comparator for this question. A different architecture would conflate “training procedure” with “architecture choice”; by matching the architecture exactly and changing only the train- ing paradigm, the gap between supervised and online macro F1 is attributable to the training procedure with as few confounders as possible. The same-architecture ceiling is therefore a lower bound on the training-procedure deficit. To bound the deficit from above we addition- ally report a feature-signal ceiling: a multinomial logistic-regression classifier trained on the same per-window features with no temporal context (scripts/supervised baseline.py). The linear ceiling answers a different question — how much signal do the features them- selves carry, independent of any specific architecture — and bounds the training-procedure deficit from above by the deficit attributable to all procedure choices including architecture. The two ceilings together bracket the deficit; the same-architecture ceiling identifies the conservative training-procedure-only contribution, the linear ceiling identifies the maximum contribution if architecture choice at the available data scale is itself part of the deficit. We report both under matched per-EID + chronological handicaps so the architecture comparison is fair, and additionally report the pooled-cross-facility variant of each to isolate the per-facility-data-isolation contribution. Train/test split. We use a chronological 70/30 split per facility: for each (mode, EID) pair, sequences are time-ordered by trial index and elapsed minutes, and the last 30% are held out as test. Within the training portion, the last 20% is used as a validation set for early stopping. Chronological split is essential for time-series data: a stratified random split would risk train/test pairs being adjacent in time and sharing confounding context (e.g. the same attack campaign window straddling the split boundary), inflating the apparent 38 Blakely et al. ceiling. We additionally report a stratified-random-split version of the same protocol for legacy comparison only — the chronological-split numbers are the publishable result. Training schedule. For each (mode, EID) pair with at least 50 sequences and at least 2 represented classes, we train for up to 200 epochs with batch size 16 and Adam at lr = 10−3 , with early stopping on val loss (patience 15, restore best weights). The early-stopping protocol allows each per-EID model to train to its own convergence point regardless of dataset size — a single fixed epoch budget would either undertrain the data-rich EIDs or overtrain the data-poor ones. Per-EID stopping epochs in practice ranged from 16 to 115, confirming that early stopping engaged correctly rather than simply running to the cap. Pooling and metric. Per-EID test predictions are pooled across EIDs within each mode, and per-class F1 is computed once on the pooled set — the same statistic shape produced by the production classifier’s argmax F1 in compute per attack f1, so the supervised ceiling and the actual classifier are directly comparable. The headline metric is macro F1 (unweighted mean per-class F1) rather than accuracy or weighted F1, because the loop’s objective is to produce attack-discriminative distributions across all classes — a weighted metric would underweight rare attack classes (e.g. MALWARE C2) that are nonetheless important for defense selection. Hard labels rather than soft targets. The supervised LSTM is trained against hard one-hot labels with sparse categorical cross-entropy. The ceiling question is “can the architecture extract attack-discriminative signal from these features at all?” — which is most cleanly answered with the standard hard-label setup. A soft-target loss (e.g. matching a target probability distribution that reflects which defenses each attack admits) would test a specific loss formulation rather than the underlying architectural ceiling, and is left as future work. Per-EID and pooled variants. The per-EID supervised LSTM described above matches the production classifier’s per-facility isolation: one model per (mode, EID) trained only on that EID’s data. We additionally report a pooled cross-facility variant of the same supervised LSTM, in which a single model per mode is trained on the (sequence, label) pairs pooled across all EIDs (sequences are still constructed within each EID — no cross-facility sliding windows — but model weights are shared across facilities). The pooled variant breaks the per-facility isolation constraint to isolate a separate question: is per-facility data isolation itself a contributor to the supervised ceiling, distinct from the training-procedure bottleneck the per-EID ceiling identifies? The chronological-split protocol is applied per-EID before pooling so the no-leakage property is preserved. Both variants are reported in Section 5.9; the per-EID variant remains the apples-to-apples comparator for the production loop, and the pooled variant is interpreted as a higher upper bound that quantifies what cross-facility weight sharing would add. 4.5 Statistical Methods P(correct) per record was tested against the random baseline of 1/12 ≈ 0.0833 using a two-sided Mann-Whitney U test. We chose Mann-Whitney rather than a parametric 𝑡-test because per-record P(correct) values are bounded in [0, 1] and heavily right-skewed — when AdCyDER: Online RL with Stackelberg-Optimized Defenses 39 the classifier is uncertain, the softmax output spreads probability across many classes and the true class receives only a small slice — so the Gaussian assumption a 𝑡-test requires does not hold. Per-class F1 was computed without averaging across modes; macro F1 is the unweighted mean of per-class F1 across the 11 attack classes (NORMAL excluded from the mean to avoid the dominant-class case skewing the average). Effect sizes for upstream signal comparisons (Section 5) are reported as Cohen’s 𝑑 on the per-record signal distributions. All statistical tests use a 0.05 significance threshold; significance is annotated explicitly in the relevant figures and tables of Section 5. 4.6 Reproducibility All experimental artifacts referenced in Section 5 are reproducible from the per-iteration JSON exports under version control in the project repository. The full chain is: (i) de- ploy the integrated pipeline at the source-tree commit recorded in the campaign meta- data; (ii) run scripts/run trials machine.sh on each evaluation host with the assigned mode; (iii) extract per-iteration MongoDB state with scripts/local results extraction.py; (iv) run make graphs to regenerate the Q1–Q7 figures and the distributional accuracy report from the pooled per-host extracts; (v) run make supervised-baseline to regenerate the supervised LSTM ceiling figures and per- mode summaries. Random seeds are pinned to 42 in the central analysis and supervised baseline scripts. Library versions are pinned in the per-service environment.yml files. 5 FINDINGS This section presents the per-figure diagnostic outputs of the methodology described in Section 4. Each subsection covers one Q-figure (Q1–Q7) or the supervised LSTM ceiling test, presents the corresponding figure with summary statistics, and gives the immediate quantitative reading. Cross-figure interpretation, limitations, and the methodology-versus- data bottleneck argument are deferred to Section 6. 5.1 Q1 — Upstream Physical-Layer Signal (ADS) Figure 2 shows the per-classification-window distribution of the embedded autoencoder’s anomaly score, partitioned by attack-class ground truth and by mode. The diagnostic question is whether the autoencoder produces attack-vs-normal separation at the per- window granularity that the RL classifier consumes. The autoencoder produces partial but real attack-discriminative signal. Across all attack-class records, attack-vs-normal Cohen’s 𝑑 ≈ +0.46 at the per-classification- window granularity (Welch 𝑡-test attack-mean 0.078 vs. normal-mean 0.009 on ads max, 𝑛attack = 6,030, 𝑛normal = 1,426). The per-class above-threshold-rate distribution is concentrated: CONTROL OVERRIDE (42% of records score above the 0.3 reconstruction-error threshold) and PROTOCOL VIOLATION (37%) detect at meaningful rates; MALWARE C2 (15%), UNAUTHORIZED CONTROL (9%), PRIVILEGE ESCALATION (9%), and DOS DDOS (9%) detect at intermediate rates; and the remaining classes score near zero. Network-only attack classes (PROBE SCAN 0.4%, LATERAL MOVEMENT 5.9%, BRUTE FORCE 2.1%, DATA EXFILTRATION 0.0%, 40 Blakely et al. Fig. 2. Q1 — ADS attack-vs-normal distributions per attack class, by mode. REPLAY ATTACK 0.0%) correctly score near zero, since the autoencoder monitors inverter physics rather than network traffic. Why aggregate ADS firing is intentionally low. The aggregate ADS firing rate (≈ 9% of classification windows carry a nonzero anomaly score) is bounded by the structure of the attack catalog rather than by detector calibration. Of the twelve attack classes (Table 1), only three (CONTROL OVERRIDE, UNAUTHORIZED CONTROL, severe DOS DDOS) have mechanisms that perturb inverter physics in a way the autoencoder is positioned to detect. The remaining nine are network-layer events with no expected physical-layer footprint — the autoencoder should stay silent on those. The per-feature reconstruction-error thresholds are derived from clean-traffic noise during autoencoder training so normal inverter variation does not false-positive, and the per-attack-type effect magnitudes in the SCADA generator are calibrated for physical realism rather than detectability. Both are principled choices; the consequence is that aggregate ADS firing is a function of attack-catalog composition (3 of 12 classes physically observable) and per-class detector sensitivity, not a target metric to optimize. The per-class detection rates reported above are consistent with this expectation — ADS fires on the physics-touching classes (CONTROL OVERRIDE 36.5%, PROTOCOL VIOLATION 38.5%, MALWARE C2 13.5%, PRIVILEGE ESCALATION 11.8%, UNAUTHORIZED CONTROL 10.2%) and stays near zero on pure network attacks (DATA EXFILTRATION 0.0%, REPLAY ATTACK 0.0%, PROBE SCAN 0.6%, LATERAL MOVEMENT 0.6%) — so the 9% aggregate rate reflects principled per-class signal, not undirected detector noise. The Q1 figure presents the headline aggregate signal but does not exhaust the autoen- coder’s diagnostic surface. As noted in Section 3.2, the embedded autoencoder also emits a per-feature reconstruction-error vector covering its two reconstruction-target channels (AphA and DCA) on every ADS event. This per-feature data is captured at the generator and persists in the OpenSearch raw-event index, but was not propagated through the aggregator into the MongoDB feature store and is not analyzed in this report. Per-channel analysis of those reconstruction errors — which channel is responsible for the anomaly score on a given window, and how channel attribution varies by attack class — would refine the AdCyDER: Online RL with Stackelberg-Optimized Defenses 41 per-attack-class detection-rate readings reported above and is left as future work (Section 7). The raw events remain in OpenSearch and can be re-extracted to support such an analysis. 5.2 Q2 — Upstream Network-Layer Signal (IDS) Figure 3 shows the per-classification-window distribution of network IDS alert counts, partitioned the same way. Fig. 3. Q2 — IDS alert count attack-vs-normal distributions per attack class. The IDS pipeline produces a strong attack-vs-normal signal: Cohen’s 𝑑 ≈ +1.38 between attack and normal records on the alert-count distribution (attack mean 2,916 alerts/window, normal mean 125 alerts/window, 𝑛attack = 6,031, 𝑛normal = 1,429). Per-class mean alert counts span roughly 1,260 (PROBE SCAN) to 4,162 (PRIVILEGE ESCALATION) per window, with category labels matching attack types — e.g., ET SCAN signatures fire on PROBE SCAN flows, ET TROJAN on MALWARE C2. Of the two upstream signal sources, IDS is the stronger by effect-size standard — the network-layer signal is now both functional and large. 5.3 Q3 — Per-Class P(True) Elevation Figure 4 is a heatmap of mean 𝑃 (𝑡 | ground truth = 𝑡) per (attack class, mode) cell. The reference value is the random baseline 1/12 ≈ 0.083; cells above 0.083 indicate the classifier is correctly elevating the true class’s probability when that class is active. The colormap is centered on the random baseline so cells appear red (below random) or green (above random). The online RL/GT loop produces only modest above-random elevation. Six attack classes exceed random in the online mode: CONTROL OVERRIDE (1.83×), REPLAY ATTACK (1.68×), PRIVILEGE ESCALATION (1.46×), LATERAL MOVEMENT (1.43×), BRUTE FORCE (1.08×), and UNAUTHORIZED CONTROL (1.02×). The remaining five classes are at-or-below random — DATA EXFILTRATION (0.98×), MALWARE C2 (0.85×), DOS DDOS (0.84×), PROBE SCAN (0.57×), and PROTOCOL VIOLATION (0.44×) — meaning the classifier puts less probability mass on the correct class when that class is active than uniform random would. The pattern of which classes are elevated does not cleanly correspond to which classes have strong upstream signal 42 Blakely et al. Fig. 4. Q3 — mean 𝑃 (true class) per (attack class, mode), centered on 1/12. Green: above random; red: below random. in Q1 or Q2: REPLAY ATTACK and LATERAL MOVEMENT are network-only attacks for which ADS is not expected to fire at all yet they are among the most-elevated, and PROTOCOL VIOLATION has the strongest ADS firing rate in Q1 yet is the most depressed in this readout. We do not interpret the elevation pattern further here; cross-figure interpretation is taken up in Section 6. The frozen mode partially overlaps the online pattern at the top — both elevate CONTROL OVERRIDE, REPLAY ATTACK, and PRIVILEGE ESCALATION above random — but diverges sharply on several classes: BRUTE FORCE drops from 1.08× in online to 0.47× in frozen, LATERAL MOVEMENT from 1.43× to 0.58×, while DOS DDOS (0.84× → 1.22×) and DATA EXFILTRATION (0.98× → 1.21×) move in the opposite direction. The two modes’ aggregate accuracy is similar (Section 5.8) but their per-class distributions diverge meaningfully on roughly half the classes — continued online training shifts which classes are elevated rather than uniformly improving or degrading the classifier. The gt only mode is uniform by construction (every cell is exactly 1/12). 5.4 Q4 — Output Distribution Per True Class Figure 5 extends Q3 by showing the full mean output distribution per true class, not just the diagonal 𝑃 (𝑡 | 𝑡). For each true class 𝑡 (rows), the heatmap shows the mean predicted probability assigned to each candidate class 𝑡′ (columns), pooled across online-mode windows. The diagonal is weak: the classifier rarely puts the largest probability on the true class. Off- diagonal mass concentrates on a small set of predicted classes that receive disproportionate probability regardless of the true class — the classifier has collapsed onto a few favored predictions rather than learning class-conditional output distributions. This is the signature AdCyDER: Online RL with Stackelberg-Optimized Defenses 43 Fig. 5. Q4 — mean output probability per (true → predicted) class pair, online mode. of a classifier that has overfit to whatever residual signal it can extract while learning to produce confident-looking distributions, rather than calibrated ones. The pattern is consistent across iterations within the online mode and across online vs frozen. Alternative interpretation: SME effectiveness–implementation mismatch. A second reading of the favored-prediction pattern is worth recording for follow-on work. The classifier’s optimization target is not per-class accuracy but the grid-health-delta produced after the GT layer maps the posterior to a defense action (Section 3.5); the gradient pulled back to the classifier therefore rewards posteriors that nominate whichever attack type’s SME-mapped optimal defense the generator’s mitigation implementation actually produces the largest grid-health improvement against in expectation. If the SME-encoded effectiveness matrix and the generator’s attack-effects implementation are not in tight correspondence — for instance if a defense rated high-effectiveness in the SME matrix produces only a small intensity- suppression in attack effects.py, or vice versa — then the favored predictions may be tracking implementation effectiveness rather than recognition signal: classes whose mapped defense the generator is most responsive to would attract probability mass regardless of 44 Blakely et al. the input features, and classes whose mapped defense the generator is least responsive to would be underweighted regardless of the input features. The two interpretations are not mutually exclusive and we cannot disentangle them from the present data, but the SME-vs-implementation calibration check is a candidate diagnostic for follow-on work and is closely related to the SME-prior limitation discussed in Section 6.5. 5.5 Q5 — Classifier Learning Trajectory Figure 6 plots P(correct) per record as a function of training-elapsed time within an iteration, separately for each mode. If the loop is learning, online-mode P(correct) should rise over training-elapsed time within each iteration; if not, it should be flat or noisy. Fig. 6. Q5 — P(correct) over training-elapsed time within an iteration, by mode. Horizontal reference: 1/12 ≈ 0.083. Online-mode P(correct) does not improve over training-elapsed time. The trajectory is flat, reaching no statistically significant departure from the random baseline at any point in the iteration window. The frozen mode (which trains for 3 hours then has its weights locked) is comparable to the continuously-trained online mode — consistent with the overall accuracy reading in Section 5.8 that neither online nor frozen is statistically distinguishable from random. The gt only mode is exactly at random by construction (uniform 1/12). This is the clearest pipeline-internal evidence that the online training procedure is not extracting useful information from the reward signal: continuing to train the classifier yields no observable improvement in the metric most directly tied to its objective. 5.6 Q6 — Reward–Correctness Causal Link Q6 tests whether the reward signal driving online training is causally coupled to the correctness metric the loop is trying to improve. The Q6 distribution is heavy-tailed and zero-inflated (71% of composite deltas are exactly zero, with the remainder spread AdCyDER: Online RL with Stackelberg-Optimized Defenses 45 across [−1, +1]), and single-statistic correlation summaries on this shape are fragile: at the campaign sample size (𝑛 = 3,008 online-mode records with both classifier output and reward decomposition available), Spearman 𝜌 and Pearson 𝑟 on the same data can disagree in sign at small effect sizes, and binned-median visualizations with sparse high-confidence bins can manufacture apparent monotonic trends from non-monotonic data. We therefore evaluate Q6 with three orthogonal lenses: a top-decile vs. bottom-decile threshold split (Mann-Whitney U on delta distributions), a hurdle decomposition that separates “does any change happen?” from “what direction when it does?” (Fisher exact on the binary), and trimmed-Pearson sensitivity at 5/10/20% trim (to test whether observed Pearson signals are driven by magnitude tails). A Q6 finding is reportable only if the three lenses converge in direction. Figure 7 reports the convergent reading. The threshold split (right panel) shows top-decile P(true) records carrying a small mean shift in the methodology-expected (more-negative) direction across all three reward channels (composite mean diff −0.046, Mann-Whitney 𝑝 = 0.029; ADS −0.035, 𝑝 = 0.041; IDS −0.057, 𝑝 = 0.058). The shifts are population-level and small at the per-record scale (top-vs-bottom Cohen’s 𝑑 ≈ −0.14 to −0.19); per-record distributions overlap heavily and the visual emphasis in the figure is on the population means with their 95% confidence intervals, not on record-level separation. The hurdle decomposition (left panel) yields the strongest result in the entire Q6 picture: top-decile P(true) records produce observable grid-state changes much more often than bottom-decile records (composite 69% → 89%, Fisher exact 𝑝 < 10−10 ; ADS 16% → 25%, 𝑝 < 10−6 ; IDS 67% → 87%, 𝑝 < 10−9 ). Trimmed-Pearson sensitivity confirms that the IDS sub-channel carries a small but trim-robust right-signed signal while composite and ADS Pearson signals are tail-driven and collapse under modest trimming. Fig. 7. Q6 multi-lens reward-correctness coupling, online mode (𝑛 = 3,008, decile 𝑛 = 301 each). Left (hurdle): top-decile P(true) records produce observable grid-state changes much more often than bottom-decile records, across all three reward channels. Right (threshold split, mean ± 95% CI): top-decile mean delta is shifted in the methodology-expected (more-negative) direction relative to bottom-decile mean. Population-level mean shift only; per-record distributions overlap heavily. The convergent reading is therefore: the reward signal is weakly aligned with classifier correctness in the methodology-expected direction. Effect sizes are small (Pearson 𝑟2 < 0.5% 46 Blakely et al. on all three channels). In practical terms: an above-average prediction explains less than half a percent of the variance in the grid-health-delta the optimizer sees as its training target, so the gradient pointing toward “be more correct” is dwarfed by the per-iteration noise in the reward stream itself, and 28 iterations is too few for that signal-to-noise ratio to drive a learning trajectory. Per-class breakdown. We additionally ran the threshold-split + hurdle test per attack class (the same convergence requirement, applied to each class’s record subset). At per-class sample sizes (93–892 records per class) no individual attack class registers as a convergent right-signed effect under the same multi-lens criterion, and any per-class significance threshold would additionally need to be Bonferroni-corrected across the 11 classes (i.e., the nominal 𝑝 threshold divided by 11, the standard guard against false positives accumulating when the same hypothesis is tested independently on each of 𝐾 subsets). The Q6 finding is therefore a property of the pooled record set, not of any single class. We do not make per-class coupling claims. Methodological note. A single-method reading of the same data — global Spearman 𝜌 = +0.016 (𝑝 = 0.39) on the pooled records and a per-class breakdown showing 9-of- 11 “wrong-sign” classes — would invite a stronger and incorrect interpretation, because Spearman’s average-rank handling of the heavy mass of tied zero deltas produces apparent per-class wrong-signed ordering on subsets where no directional effect actually exists, and a binned-median visualization of the same data is misleading at this campaign’s sample size because the bins with negative medians are small (𝑛 = 10–12) high-confidence bins surrounded by large null low-confidence bins, a pattern that binning manufactures into apparent monotonicity. Reporting the multi-lens convergence reading rather than any single statistic is what avoids that over-interpretation. 5.7 Q7 — Grid Outcome Trend Figure 8 plots grid-health-delta over training iterations within each mode. If the loop produces operationally improving outcomes, the trend should be downward (better grid health) over iterations. Grid-health-delta shows no meaningful trend across training iterations in any mode. The system is not producing improving operational outcomes as training proceeds; the per-iteration distribution is broadly stationary across the campaign. This is the loop-level confirmation of Q5 and Q6: with no learning trajectory in the classifier (Q5) and no causal coupling between correctness and reward (Q6), the operational outcome metric (Q7) has no mechanism by which to improve. Note on metric direction. Grid-health-delta is the per-window difference post-defense − pre-defense (negative values denote defense-driven improvement); it is not the absolute trajectory of grid health over the campaign. A flat trend therefore is not a “defenses are succeeding so nothing happens” reading: Q1 and Q2 confirm that the pre-defense windows in attack periods carry distinguishable autoencoder reconstruction error and IDS alert density above normal background (ADS Cohen’s 𝑑 ≈ +0.46, IDS 𝑑 ≈ +1.38), so the loop AdCyDER: Online RL with Stackelberg-Optimized Defenses 47 Fig. 8. Q7 — grid-health-delta across training iterations, by mode. is being asked to drive the post-defense windows below those elevated pre-defense values. Flat-and-near-zero deltas with non-silent pre-defense signal mean the defenses — as actually realized in the SCADA generator’s mitigation implementation — are not systematically reducing the post-defense ADS/IDS signal across the campaign. The favored-prediction collapse documented in Q4 (Section 5.4) and the SME-effectiveness-versus-implementation alternative interpretation discussed there are both consistent with this Q7 picture. 5.8 Distributional Accuracy Summary Table 6 aggregates the per-record correctness statistics across the campaign for each mode. P(correct) per record is tested against the random baseline 1/12 ≈ 0.0833 using a two-sided Mann-Whitney U test (Section 4.5). Mode 𝑛 records Argmax accuracy Mean P(correct) 𝑝 vs random online 3,429 0.089 (1.06× random) 0.087 (1.05× random) 0.30 (n.s.) frozen 1,700 0.077 (0.92× random) 0.080 (0.96× random) 0.40 (n.s.) gt only 902 0.000 (uniform argmax) 0.083 (1.00× random) exact (by construction) Table 6. Distributional accuracy by mode. Mann-Whitney U tests P(correct) per record against the random baseline 1/12. Two readings from this table. First, neither the online nor the frozen mode is statistically distinguishable from random at the 0.05 significance threshold; both mean P(correct) values fall within standard sampling variation of 1/12. The point estimate for online is marginally above random (1.06×) and for frozen marginally below (0.92×), but neither effect rises to statistical significance and the absolute gap between the two is small. Continued online training does not improve over a static early-bootstrap snapshot, but neither does it materially degrade it — both regimes sit at chance. Second, the small lift the online mode 48 Blakely et al. achieves on argmax (1.06× random) is small enough to be consistent with the few attack classes for which Q1 or Q2 provide strong upstream signal — i.e., the classifier is reading information directly off the upstream channels rather than learning a compositional model of attack types. 5.9 Supervised Ceilings We report two complementary supervised ceilings on the same per-window feature vectors the production classifier saw, with gt attack type as the target label and a chronological 70/30 train/test split for both (Section 4.4). The same-architecture LSTM ceiling holds architecture, features, and per-facility isolation fixed and varies only the training procedure (a conservative lower bound on the training-procedure deficit). A feature-signal ceiling using multinomial logistic regression on the same features (no temporal context) bounds what the features themselves can support. Headline numbers per mode appear in Table 7; per- mode per-class F1 for the LSTM ceiling appears in Figure 9, with the LSTM-vs-actual-RL comparison in Figure 10. Table 7. Supervised ceilings on online-, frozen-, and gt only-mode per-window features. All rows: chronological 70/30 split, predictions pooled across facilities before computing macro F1. The “× deployed” column expresses the ratio against the deployed RL classifier in the same mode. Classifier (handicaps held) Mode Macro F1 × deployed Deployed online RL online 0.07 1.0× Same-arch LSTM, per-EID online 0.34 4.7× Linear (LogReg), per-EID online 0.70 10× Linear (LogReg), pooled online 0.79 11× Deployed frozen RL frozen 0.06 1.0× Same-arch LSTM, per-EID frozen 0.29 4.6× Linear (LogReg), per-EID frozen 0.67 11× Deployed GT-only (uniform prior) gt only 0.00 — Same-arch LSTM, per-EID gt only 0.22 — Linear (LogReg), per-EID gt only 0.68 — Same-architecture LSTM ceiling (lower bound). The per-EID supervised LSTM reaches macro F1 ≈ 0.34 on the online mode versus 0.07 for the deployed online RL classifier — a 4.7× gap. The frozen-mode comparison is 0.29 vs 0.06 (≈ 4.6×), and the gt only-mode per-EID ceiling is 0.22 versus a constructed-zero baseline for the uniform-prior gt only classifier. The pooled cross-facility variant reaches ≈ 0.41 on online-mode features (an additional ≈ 1.2× above the per-EID variant), isolating per-facility data isolation as a real but secondary contributor to the gap. Architecturally identical, the same-architecture ceiling attributes the entire gap to the training procedure alone. Feature-signal ceiling (upper bound). A multinomial logistic-regression classifier (with standard feature scaling, trained to convergence with lbfgs) on the same per-window features under matched per-EID + chronological handicaps reaches macro F1 ≈ 0.70 on online- mode features — ≈ 10× higher than the deployed classifier and ≈ 2× higher than the same-architecture LSTM ceiling under identical handicaps. Pooling across facilities lifts AdCyDER: Online RL with Stackelberg-Optimized Defenses 49 Fig. 9. Same-architecture supervised LSTM ceiling — per-class F1 by mode (predictions pooled across EIDs within each mode). this to ≈ 0.79 (≈ 11× deployed). The matched-handicaps comparison is the diagnostically informative one: with 𝑛 ≈ 250 training records per facility and 11 attack classes, the production LSTM architecture (˜50k parameters with length-3 sequence input) is over- capacity for what the per-window features deliver, and the recurrent machinery contributes little because the features alone are highly discriminative. The same-architecture ceiling therefore underestimates what the features can support; the linear ceiling provides the more permissive upper bound. The training-procedure failure is bracketed between 4.7× (same-architecture) and 10–11× (linear) depending on whether architecture choice is counted as part of the training procedure or as a separate contributor at this data scale. Reading the bracket. The 4.7× same-architecture gap is the conservative claim: holding the LSTM architecture and per-EID isolation fixed, the online RL training procedure extracts ≈ 21% (0.07/0.34) of what offline supervised training extracts from the same features under the same architectural constraints. The 10×–11× linear gap is the more permissive claim: a simpler classifier that is appropriate for this data scale extracts substantially more. Both readings agree on the qualitative conclusion — the training procedure is the dominant bottleneck and the features are not the limiting factor — and disagree only on how to attribute the LSTM-vs-linear gap (training-procedure choice including architecture vs. training-procedure choice excluding architecture). 5.10 Summary of the Findings Read together, the seven Q-figures and the supervised LSTM ceiling test give a consistent story along the data path of the loop: 50 Blakely et al. Fig. 10. Same-architecture supervised LSTM ceiling — macro F1 per mode: offline supervised LSTM (chronological split, early-stopped) vs. actual online RL classifier. Same architecture, features, and per-EID structure — only the training procedure differs. The 4.7× online-mode gap shown here is the conservative lower bound on the training-procedure deficit; the linear feature-signal ceiling (Table 7) puts the upper bound at 10–11×. ∙ Upstream signal works (Q1, Q2): both ADS and IDS produce attack-vs-normal separation, with IDS large-effect and ADS partial and attack-class-selective. ∙ Classifier output is at chance (Q3, Q4, Q5, Table 6): per-class P(true) elevation is modest and confined to the attack classes with the strongest upstream signal; output distributions are off-diagonal-concentrated; P(correct) does not improve over training-elapsed time; argmax accuracy is statistically indistinguishable from random in the online mode. ∙ Reward and correctness are weakly coupled in the methodology-expected direction but underpowered (Q6): multi-lens convergent reading (threshold split, hurdle, trimmed-Pearson) shows top-decile P(true) records produce more frequent AdCyDER: Online RL with Stackelberg-Optimized Defenses 51 state changes and slightly larger improvements than bottom-decile records; effect magnitudes (𝑟2 < 0.5%, top-vs-bot Cohen’s 𝑑 ≈ −0.19) are too small to drive gradient-based learning at the campaign sample size. ∙ Grid outcome is flat (Q7): no operational improvement over training iterations. ∙ The dominant bottleneck is the training procedure (paired supervised ceil- ings): the same-architecture supervised LSTM reaches ≈ 4.7× higher macro F1 than the deployed RL on the same features (training-procedure gap with architecture held fixed — lower bound), and a linear classifier on the same features under matched per- EID handicaps reaches ≈ 11× higher (training-procedure-plus-architecture-choice gap — upper bound). Pooled cross-facility variants contribute a further ≈ 1.2× (data-isolation gap). Cross-figure interpretation, the methodology-versus-data argument in detail, and the implications for the original research hypothesis are taken up in Section 6. 6 DISCUSSION This section interprets the per-stage findings of Section 5 jointly: what the evidence supports and what it does not, where the loop’s failure to learn is localized, what the reward-correctness diagnostic implies for the proxy reward, what generalizable lessons the diagnostic methodology carries beyond AdCyDER, and the principal limitations that bound the conclusions. 6.1 What the Evidence Supports Three claims are well-supported by the per-stage data presented in Section 5. The upstream detection layers carry attack signal. IDS produces a large-effect attack- versus-normal separation (𝑑 ≈ +1.38, Q2), and ADS produces a partial, attack-class-selective separation (𝑑 ≈ +0.46, Q1). Neither signal is null, and the IDS signal in particular is large enough that downstream ML stages have an information-theoretic basis on which to discriminate. The supervised ceilings (Section 5.9) confirm this from the model side: under matched per-EID + chronological handicaps a linear classifier on the same features reaches macro F1 ≈ 0.70, and the same-architecture LSTM reaches ≈ 0.34. Whatever the failure of the production loop is, it is not that the features are signal-free. The dominant bottleneck is the training procedure. The paired supervised ceilings dis- criminate between two hypotheses: that the per-window features lack the signal to support attack-type discrimination (data-quality failure) versus that the signal is present but the online RL training procedure fails to extract it (training-procedure failure). The same- architecture LSTM ceiling, which matches the production classifier’s architecture and per-facility isolation exactly and changes only the training paradigm, reaches macro F1 ≈ 0.34 on online-mode features versus 0.07 for the actual online RL classifier — a 4.7× gap attributable to the training procedure alone, with the production architecture held fixed. The feature-signal (linear) ceiling under the same handicaps reaches ≈ 0.70 (10× deployed), indicating the per-window features support substantially more discriminative 52 Blakely et al. signal than even the same-architecture offline-trained variant extracts at this data scale. The training-procedure deficit is bracketed between 4.7× (same architecture) and 10× (matched handicaps, simpler classifier), with the gap between the two attributable to the produc- tion LSTM architecture being over-capacity for the data scale rather than to additional procedure shortcomings. The pooled cross-facility variant of the same-architecture LSTM lifts per-facility isolation as a separate constraint and contributes a further ≈ 1.2×. Both gaps point the same direction: the dominant bottleneck is the training procedure, not the features. Continued online training does not improve over a static snapshot. The online and frozen baseline modes produce statistically indistinguishable aggregate accuracy (Section 5.8), and the per-iteration trajectory of P(correct) is flat-to-slightly-degrading over training-elapsed time (Q5). In the continual-learning vocabulary of Parisi et al. and Delange et al. [37, 38], the online configuration shows no measurable improvement over the lower-bound baseline that holds parameters near their initial values — a standard control these surveys recommend for assessing whether ongoing parameter updates contribute beyond a static snapshot. AdCyDER’s online-versus-frozen comparison answers this question in the negative for the configuration tested. 6.2 What the Evidence Does Not Support Several claims are explicitly not supported by the data, and we want to be precise about which. The data does not support the original research hypothesis as it was stated — that an online RL classifier coupled to a Stackelberg game-theoretic defense selector, trained on delayed grid-health-delta reward, will learn per-facility attack discrimination from operational outcomes alone. In the configuration evaluated, the classifier’s argmax accuracy is statistically indistinguishable from random in the online mode (Section 5.8), per-class P(correct) does not improve over training time, and grid outcomes are flat across iterations. The data does not support the alternative interpretation that the pipeline is fundamentally broken at the data layer. Both upstream signals are present, the supervised ceiling reaches non-trivial F1 on the same features, and the GT defense selector produces diverse defense allocations rather than collapsing onto a single dominant action. The pipeline’s individual stages function; the closed-loop training procedure is the failing component. The data also does not support a strong claim that confidence trajectory is a misleading metric in general. What it does support is the specific finding that, on this pipeline, classifier confidence converged from low to high while classification correctness did not improve — meaning that confidence convergence on this pipeline did not correspond to discrimination learning. This is consistent with the more general result of Guo et al. [33] that confidence and accuracy are conceptually distinct quantities and that modern neural networks can be confidently wrong, but it does not by itself demonstrate that confidence is uninformative in all RL classifier settings. It does, however, motivate the use of correctness-based diagnostics rather than confidence-based ones for any future evaluation of this pipeline. AdCyDER: Online RL with Stackelberg-Optimized Defenses 53 6.3 The Reward Signal: Weakly Aligned, Underpowered The Q6 reading reported in Section 5.6 is that the reward signal is weakly aligned with classifier correctness in the methodology-expected direction across all three robustness lenses (threshold split, hurdle decomposition, trimmed-Pearson sensitivity), with effect magnitudes too small to drive gradient-based learning at the campaign sample size. The loop-level consequence is that the reward channel is informative but underpowered: it carries usable gradient information about whether the classifier is producing more-correct posteriors, but at a magnitude (top-vs-bottom decile Cohen’s 𝑑 ≈ −0.19, Pearson 𝑟2 < 0.5%) that is small relative to the per-step variance the optimization procedure must contend with over 28 iterations. In the framework of Skalse et al. [32], this is not a misspecified-reward configuration but an underpowered one: the proxy (composite grid-health-delta) and the true objective (correct attack-type classification) exhibit positive directional coupling, just at a magnitude small enough that optimizing the proxy in expectation produces little improvement on the true objective at this sample size. The hurdle finding (more-correct predictions are associated with much more frequent observable grid-state changes, Fisher exact 𝑝 < 10−10 on the composite) is the strongest signal in the Q6 picture and indicates that the loop is producing differential operational effects driven by classifier output — the magnitude direction of those effects, however, is small enough that the gradient pulled out of any individual reward record is comparable to the noise floor of nearby records. The mechanism by which a grid-health-delta reward could be underpowered relative to the optimizer’s noise floor is plausible from the architecture documented in Section 3. The reward observed in window 𝑡 + 1 reflects the combined effect of (i) the classifier’s posterior at window 𝑡, (ii) the GT defense selected from that posterior, (iii) the actuator’s physical effect, and (iv) the synthetic generator’s mitigation propagation. Each link in this chain attenuates the signal-to-noise of the eventual reward. The credit-assignment problem documented in the RL literature [27] additionally means the classifier has no per-stage feedback to disentangle “my classification was wrong” from “my classification was right but the defense’s physical effect was small.” The composite construction of grid-health (Section 3.4.2) is a second candidate location for additional signal-to-noise improvement. The composite is a 50/50 blend of an ADS sub- score (autoencoder reconstruction error) and an IDS sub-score (rolling-baseline-normalized alert density), and gt engine writes the pre- and post-defense values of both sub-scores into every rl rewards document so that per-channel coupling can be examined. Under the same multi-lens robustness check as the composite, the ADS sub-channel shows a small right-signed coupling that is robust under trimming (Spearman 𝜌 ≈ −0.10 on nonzero records, 𝑝 ≈ 0.04) while the IDS sub-channel and composite show right-signed Pearson signals that collapse under modest trimming and hurdle effects with the same direction and significance as the composite. Reward-reformulation candidates worth exploring in follow-on work include weighting the composite to favor whichever sub-channel proves to carry more trim-robust signal in a given deployment, lengthening the IDS lookback window from 5 minutes to match the longer end of the defense observation delay (5–60 minutes), and 54 Blakely et al. replacing the asymmetric clip-at-zero IDS normalization with a symmetric normalization that lets defenses driving IDS density below baseline register as positive contributions. None of these reformulations are predicted by the present data to flip the sign of the coupling — the coupling is already in the right direction — but each is a candidate for raising the magnitude of the gradient signal the optimizer sees per record. 6.4 Methodology Generalization: Confidence Convergence Is Not Learning A practical lesson from the project’s own internal trajectory is worth surfacing. Earlier project status reporting tracked classifier confidence convergence as the headline learning- progress metric and reported that confidence reached 0.95 ± 0.03 across independent trials — a reading consistent with what an RL classifier “looks like when it works.” The May 2026 diagnostic round showed that confidence convergence on this pipeline corresponded to overconfident wrong predictions rather than to discrimination learning, and that the classifier’s argmax accuracy and per-class P(correct) had not actually improved over time. This is a domain-specific instance of the general result of Guo et al. [33] that confidence and accuracy are conceptually distinct and can decouple in modern neural-network classifiers, particularly under training procedures that do not directly optimize for calibration. The methodological recommendation we draw from our own experience is operational rather than theoretical: any evaluation of an RL or weakly-supervised classifier in this domain should report correctness-based metrics (per-class P(correct), distributional accuracy, calibration error, comparison against a same-architecture supervised ceiling) before reporting confidence trajectories, and confidence trajectories should be interpreted as evidence of training-process activity rather than of discrimination learning. The pipeline-stage diagnostic protocol used in Section 5 (the Q1–Q7 sequence) is the second methodological contribution we believe generalizes beyond AdCyDER. The protocol forces the analyst to verify upstream signal presence, classifier output behavior, training- time learning trajectory, reward–correctness coupling, and end-to-end operational outcome independently before drawing model-level conclusions. This is a direct application of the production-ML engineering principle that data-quality and pipeline-stage failures are com- monly misattributed to model-level causes when stage-wise diagnostics are absent [34, 35]. For online RL/GT loops trained on delayed environmental reward, where the chain from classification decision to feedback observation passes through several stages each of which can introduce its own failure mode, this kind of stage-wise inspection is, in our experience, the difference between catching the failure at the right layer and chasing model-architecture changes that cannot fix a reward-misspecification problem. The supervised same-architecture ceiling test (Section 5.9) is the third generalizable component. The construction — match the architecture exactly, change only the training paradigm, and read the F1 gap as an upper bound on what the training procedure can extract from the available features — isolates the training-procedure-versus-feature-signal question with as few confounders as possible. This is consistent with the methodological recommendation in Amodei et al. [30] to compare semi-supervised RL against a baseline trained only on labeled episodes to isolate the effect of label scarcity, applied here to isolate the effect of the online delayed-reward training paradigm rather than label scarcity per se. AdCyDER: Online RL with Stackelberg-Optimized Defenses 55 6.5 Limitations Several limitations bound the conclusions. Synthetic environment. The campaign was conducted entirely against the containerized synthetic SCADA generator described in Section 3.2, not against an HIL testbed integrating real inverters or RTUs with simulated grid dynamics. The generator captures attack-type variety, time-alignment across data streams, and an autoencoder-based physics anomaly signal, but it does not substitute for the operational realism that an HIL deployment of the type used by Nguyen et al. [15], Sangoleye et al. [16], or Mogilicharla et al. [5] would provide. We do not claim that the negative result generalizes to all possible feature signals: the conclusion that the training procedure rather than the data is the bottleneck is conditional on the per-window feature distribution this generator produces. A higher-fidelity environment could change the supervised ceiling and would change the absolute scale of grid-health-delta variability that the reward sees. Single experimental campaign. The May 2026 evaluation campaign comprised one set of trial runs per baseline mode, with random seeds pinned for reproducibility but no replication across independent random seeds at the campaign level. The per-mode aggregate statistics in Section 5.8 are stable in the sense that they pool many windows across many facilities, but the conclusion that the training procedure is the bottleneck rests on the observed gap between the supervised ceiling and the production classifier in this campaign. We do not present multi-seed or multi-campaign variability for this gap. Single architecture and reward formulation. The conclusions apply to the per-facility LSTM(128)→LSTM(64)→Dense(32) architecture, the 5-minute aggregation cadence, and the soft-clipped grid-health-delta reward documented in Section 3.4. Within this combination the training procedure does not learn discrimination, but the report does not establish that no online RL classifier with delayed environmental reward can learn this task. The pooled supervised LSTM ceiling (Section 5.9) shows that cross-facility weight sharing would raise the achievable upper bound by ≈ 1.2× on these features — meaningful but not dominant. A different architecture (longer sequence length, attention), a different reward formulation (reward shaping with intermediate signals from the IDS or ADS layers, longer reward windows, dense per-decision reward), or a different exploration schedule could produce different results. The Q6 reading that the reward signal is directionally aligned but underpowered is the strongest reason to investigate reward-magnitude reformulation specifically. Calibration of the GT model parameters. The GT defense selector’s effectiveness, baseline impact, and cost parameters are SME-encoded priors (Section 3.5), not empirically calibrated. The Stackelberg loss minimization is mathematically well-defined, but its outputs are only as defensible as those priors. We did not evaluate the sensitivity of the GT layer’s defense allocations to perturbations in those priors, and we cannot rule out that part of the reward- misspecification picture in Q6 originates in the GT layer rather than in the reward signal itself. 56 Blakely et al. Generator fidelity to operational telemetry. The synthetic generator’s autoencoder ADS, IDS, and netflow streams are designed to be physically plausible and time-aligned but were not validated against operational telemetry from a real DER deployment. The IDS effect size we observe (𝑑 ≈ +1.38) is sufficiently large that it would be surprising for a real-world signal to be smaller, but the ADS effect (𝑑 ≈ +0.46, attack-class-selective) is in a regime where generator characteristics matter more, and its per-feature reconstruction-error structure (Section 5.1) was instrumented but not analyzed. 6.6 Implications for the Original Research Hypothesis The original hypothesis tested in this work was that an online RL/GT loop trained on delayed grid-health-delta reward would learn per-facility attack discrimination from operational outcomes alone. The evidence in Section 5 is inconsistent with this hypothesis as stated, in the configuration evaluated. We do not believe the evidence is consistent with the negation of the hypothesis at the same level of generality; the supervised ceiling test in particular shows that the features carry signal that the architecture can extract under a different training paradigm, and the Q6 multi-lens reading identifies an underpowered-but-aligned reward channel as a specific, plausibly fixable contributor. We therefore read the result as a falsified configuration of the hypothesis rather than a refuted class of approaches. The configuration includes: per-window features as currently aggregated, a per-facility LSTM as currently sized, a delayed grid-health-delta as the sole reward, a soft-clipped reward magnitude as currently scaled, and an SME-prior- parameterized Stackelberg defense selector as currently calibrated. Section 7 consolidates the recommended next steps in priority order, with reward reformulation at the top given the Q6 finding. 7 CONCLUSIONS AND FUTURE WORK 7.1 Conclusions This report has documented the design, implementation, and evaluation of an integrated online RL / Stackelberg game-theoretic defense pipeline for distributed energy resources, and has presented a falsified configuration of the underlying research hypothesis. We summarize the conclusions in three parts. The hypothesis was tested cleanly and is not supported by the data. The integrated loop was deployed end-to-end across three baseline configurations (online, frozen, gt only), evaluated through a pipeline-ordered diagnostic protocol (Q1–Q7) and a same-architecture supervised LSTM ceiling test, and analyzed against a chronological hold-out partition with random seeds pinned. The hypothesis tested — that an online RL classifier coupled to a Stackelberg defense selector, trained on delayed grid-health-delta reward derived from physically-grounded inverter telemetry, would learn per-facility attack discrimination from operational outcomes alone — is not supported in the configuration evaluated. The classifier’s argmax accuracy in the online mode is statistically indistinguishable from random (𝑝 = 0.30), per-class P(correct) does not improve over training-elapsed time, and grid outcomes are flat across iterations. AdCyDER: Online RL with Stackelberg-Optimized Defenses 57 The failure is localized to two specific mechanisms. The supervised LSTM ceiling test (Section 6.1) localizes the failure to the training procedure rather than to the feature signal: a same-architecture supervised LSTM extracts substantially more F1 from identical features than the production online classifier, with the only variable being the training paradigm. The Q6 reward–correctness diagnostic (Section 6.3) identifies a second mechanism: the reward signal driving online updates is anti-correlated with classifier correctness on a majority of attack classes — the diagnostic signature, in the framework of Skalse et al. [32] and Pan et al. [31], of a misspecified or hacked proxy reward. The contributions worth carrying forward sit in three areas. First, a transferable diag- nostic methodology. The pipeline-ordered Q1–Q7 protocol paired with a same-architecture supervised ceiling test (Section 6.4) is proposed as a falsification protocol for online RL/GT defense pipelines that learn from delayed environmental reward. We believe it generalizes to other online-learning security ML pipelines where the chain from classification decision to evaluable outcome passes through several stages, each of which can introduce its own failure mode. Second, an open and reproducible experimental infrastructure. The integrated pipeline (synthetic SCADA generator with embedded Fronius inverter physics and autoencoder ADS, Suricata IDS, time-windowed feature aggregation, per-facility LSTM with three baseline modes, Stackelberg defense selector with SME-encoded priors, OpenC2 actuator framework) passes 335+ automated tests across the main repository and two service submodules. The diagnostic scripts that produced every figure in this report regenerate from the per-iteration JSON exports under version control, with random seeds pinned and library versions pinned. The synthetic SCADA generator and the attack simulator (Section A) are released publicly. Independent groups can re-run the diagnostic protocol against their own configurations of the loop without rebuilding the infrastructure from scratch. Third, a characterization of the loop’s signal ceiling. The per-EID and pooled supervised- ceiling readings together with the per-class P(true) data establish what the loop can and cannot do on the synthetic-environment features as currently aggregated, and decompose the gap from the actual classifier into a 4.7× training-procedure contribution and a 1.2× per-facility-data-isolation contribution. This characterization is itself a contribution: it removes “the data is fine, the model just needs to train longer” as a viable interpretation, names per-facility isolation as a real but secondary constraint, and scopes the search space for productive next steps. 7.2 Aspirational Future Work The directions below are written as aspirational rather than planned: at the time of writing, the project does not have funding for follow-on work, and these directions are recorded here for the record so that future investigators (within the project team or elsewhere) can pick up the thread without rediscovering the same priorities. We list them in priority order based on the bottleneck localization of Section 6. Reward reformulation (highest priority). The Q6 multi-lens reading (Section 6.3) is that the reward signal is directionally aligned but small in magnitude relative to the optimizer’s 58 Blakely et al. noise floor at the campaign sample size. Reward-reformulation work should therefore focus on raising the magnitude of the gradient signal the optimizer sees per record rather than on changing its direction. Candidate reformulations include (i) reweighting the composite to favor whichever sub-channel proves to carry more trim-robust signal in a given deployment; (ii) reformulating the IDS sub-channel construction — replacing the asymmetric clip-at-zero normalization with a symmetric one that lets defenses below baseline register as positive contributions, lengthening the IDS lookback window to match the defense-dependent grid-health observation delay (5–60 minutes), or filtering out defense-induced alert spikes (session resets, scan-detection firings on blocked sources); (iii) adding denser per-decision intermediate-stage signals (direct per-classification credit assignment from IDS or ADS layers in addition to grid-health-delta); (iv) revisiting the per-defense observation-delay schedule and the soft-clamp scale together with whichever composite is adopted. Pan et al. [31] additionally argue that reward-misspecification problems do not improve and may worsen with increased agent capability; the corollary for the underpowered-but-aligned setting we observe is that simply scaling the classifier within the current low-magnitude reward structure is unlikely to extract more learning, so reformulation should precede architectural scale-up. Hardware-in-the-loop testbed evaluation. The campaign reported here was conducted entirely against the containerized synthetic SCADA generator, not against an HIL testbed integrating real inverters or RTUs with simulated grid dynamics. An HIL deployment in the lineage of Nguyen et al. [15], Sangoleye et al. [16], or Mogilicharla et al. [5] would test whether a higher-fidelity feature signal — in particular, the operational cross-channel coupling between inverter physics, network telemetry, and IDS alerts that a real testbed produces — changes the supervised ceiling and the absolute scale of grid-health variability that the reward sees. We do not predict that HIL evaluation alone would resolve the underpowered-reward issue (which is structural to the proxy-versus-true-objective magnitude mismatch rather than to data fidelity), but it would broaden the supervised ceiling and might raise the achievable upper bound that any training procedure could reach. Per-feature ADS reconstruction-error analysis. The embedded autoencoder emits per- feature reconstruction errors covering its two reconstruction-target channels (AphA and DCA) on every ADS event, and these were captured at the generator and persist in the OpenSearch raw-event index but were not propagated through the aggregator into the MongoDB feature store and were not analyzed in this report (Section 5.1). Re-extracting and analyzing this data would refine the per-attack-class detection-rate readings of Q1, identify which channel is responsible for the anomaly score on a given window, and characterize how channel attribution varies by attack class. This would also provide a finer-grained diagnostic surface for any reward reformulation work that incorporates ADS sub-signals into the credit-assignment pathway. External validation of the diagnostic methodology. The Q1–Q7 + supervised-ceiling proto- col was developed inside this project and applied to one pipeline configuration. Its claimed generality (Section 6.4) would be substantively tested by applying it to other online RL/GT AdCyDER: Online RL with Stackelberg-Optimized Defenses 59 defense pipelines that learn from delayed environmental reward — whether other DER cybersecurity pipelines or analogous systems in adjacent domains. Such an external applica- tion would either confirm that the protocol localizes failures to the same stages it did here, or surface failure modes it does not currently catch and motivate protocol extensions. Workflow-orchestrated simulator extension. The Prefect-based simulator rewrite (Appen- dix A.3) is functional at unit-test scope but was not deployed in any operational use within the project. As a follow-on direction, the prototype could be developed into the operational generation infrastructure for any successor evaluation campaign — enabling per-task error handling, structured logging, multi-protocol coverage extensions, and Grid-STIX-format threat-intelligence interchange that the current generator does not support. Architecture and exploration alternatives (lower priority). The same-architecture su- pervised LSTM ceiling already places an upper bound on what the current per-facility per-window data can support, so architectural changes are unlikely to resolve the headline negative result without an accompanying improvement in either the feature signal or the training procedure. That said, longer LSTM sequence lengths, attention mechanisms, cross- facility weight sharing (which would relax the per-facility data-volume constraint), and alternative exploration schedules are all natural perturbations to the current configuration that would inform a more complete picture of the configuration space if and when reward reformulation work establishes a coherent training signal first. 7.3 Closing The strongest single result reported here is a falsified configuration of an online RL/GT defense pipeline that, on its face, looked like it was working — and a diagnostic methodology that revealed the failure cleanly enough to localize the bottleneck and identify the next-step priorities. Reporting null and negative results in security ML is, in our view, a contribution in itself: the configurations that do not work are part of the search space that follow-on work needs to know about, and the absence of such reports causes resources to be spent rediscovering the same dead ends. The diagnostic methodology and the open infrastructure released alongside this report are, we hope, the more durable contributions of this work. ACKNOWLEDGEMENTS This work was supported by the U.S. Department of Energy, Office of Cybersecurity, Energy Security, and Emergency Response (CESER). The authors thank the CESER program office for its sustained support of distributed-energy-resource cybersecurity research. Large language model assistants were used extensively throughout the project for literature review, code generation, code review, debugging, documentation drafting, and the AI-auditor correctness audits described in Section 3.7.3. The specific systems used — across multiple model versions over the project’s lifetime — include Anthropic Claude (Opus and Sonnet), OpenAI ChatGPT, Google Gemini, and the open-weights families Qwen, Gemma, and Llama. All model-generated artifacts were reviewed and verified by the authors before incorporation; the authors take full responsibility for the technical content of this report. 60 Blakely et al. REFERENCES [1] Y. Li and J. Yan, “Cybersecurity of Smart Inverters in the Smart Grid: A Survey,” IEEE Transactions on Power Electronics, vol. 38, pp. 2364–2383, Feb. 2023. [2] I. Zografopoulos, N. D. Hatziargyriou, and C. Konstantinou, “Distributed Energy Resources Cyberse- curity Outlook: Vulnerabilities, Attacks, Impacts, and Mitigations,” IEEE Systems Journal, vol. 17, pp. 6695–6709, Dec. 2023. [3] C. Carter, C. Lai, N. Jacobs, S. Hossain-McKenzie, P. Cordeiro, I. Onunkwo, and J. Johnson, “Cyber Security Primer for DER Vendors Aggregators and Grid Operators,” Tech. Rep. Sand2017-13113, Sandia National Laboratories, Nov. 2017. [4] M. Liu, F. Teng, Z. Zhang, P. Ge, M. Sun, R. Deng, P. Cheng, and J. Chen, “Enhancing Cyber-Resiliency of DER-Based Smart Grid: A Survey,” IEEE Transactions on Smart Grid, vol. 15, pp. 4998–5030, Sept. 2024. [5] S. Mogilicharla, M. Tripathy, and M. Kanabar, “Integrating Cyber and Physical Analysis for Intrusion Detection in Plant-Level DER Microgrids,” IEEE Transactions on Industry Applications, vol. 61, pp. 5430–5444, July 2025. [6] J. Mao, M. Zhang, and Q. Xu, “CNN and LSTM based Data-driven Cyberattack Detection for Grid- connected PV Inverter,” in 2022 IEEE 17th International Conference on Control & Automation (ICCA), (Naples, Italy), pp. 704–709, Ieee, June 2022. [7] J. Chen, A. J. Gallo, S. Yan, T. Parisini, and S. Y. R. Hui, “Cyber-Attack Detection and Countermeasure for Distributed Electric Springs for Smart Grid Applications,” IEEE access : practical innovations, open solutions, vol. 10, pp. 13182–13192, 2022. [8] A. M. Y. Koay, R. K. L. Ko, H. Hettema, and K. Radke, “Machine learning in industrial control system (ICS) security: Current landscape, opportunities and challenges,” Journal of Intelligent Information Systems, vol. 60, pp. 377–405, Apr. 2023. [9] “IEEE Standard for Electric Power Systems Communications-Distributed Network Protocol (DNP3),” IEEE Std 1815-2012 (Revision of IEEE Std 1815-2010), pp. 1–821, Oct. 2012. [10] S. A. Oyedotun, G. P. Oise, and C. E. Ozobialu, “Towards Intelligent Cybersecurity in SCADA and DCS Environments: Anomaly Detection Using Multimodal Deep Learning and Explainable AI,” Journal of Science Research and Reviews, vol. 2, pp. 20–31, July 2025. [11] F. Alharbi, “Intelligent Adaptive Cyber Defense Framework for Securing Renewable Energy Grids: A Hybrid AI-Driven Approach for Industry 4.0,” IEEE access : practical innovations, open solutions, vol. 13, pp. 195763–195784, 2025. [12] N. Abdi, A. Albaseer, and M. Abdallah, “The Role of Deep Learning in Advancing Proactive Cyber- security Measures for Smart Grid Networks: A Survey,” IEEE Internet of Things Journal, vol. 11, pp. 16398–16421, May 2024. [13] S. J. Plathottam, K. Jhala, and S. Fenu, “Power data anomaly detection package,” 2025. OSTI Code ID: 172190 (SF-25-150). [14] M. Luay, S. Layeghy, N. Noorbin, M. Sarhan, G. Kulatilleke, N. Moustafa, and M. Portmann, “Time Matters: Temporal NetFlow Features for ML-Based Network Intrusion Detection,” IEEE Access, vol. 14, pp. 66899–66913, 2026. [15] T.-T. Nguyen, R. Kadavil, and H. Hooshyar, “A Real-Time Cyber-Physical Simulation Testbed for Cybersecurity Assessment of Large-Scale Power Systems,” IEEE Transactions on Industry Applications, vol. 60, pp. 8329–8340, Nov. 2024. [16] F. Sangoleye, J. Johnson, and E. Eleni Tsiropoulou, “Intrusion Detection in Industrial Control Systems Based on Deep Reinforcement Learning,” IEEE access : practical innovations, open solutions, vol. 12, pp. 151444–151459, 2024. [17] M. Farsi, M. Alwateer, S. A. Alsaedi, A. I. AlSahafi, H. M. Balaha, M. M. Aboelnaga, M. Badawy, and M. A. Elhosseini, “Detection of disturbances and cyber-attacks in smart grids using explainable machine learning,” Scientific Reports, vol. 16, p. 9834, Feb. 2026. AdCyDER: Online RL with Stackelberg-Optimized Defenses 61 [18] F. Zhang, Y. Cui, J. Yu, J. Li, L. Li, G. Cui, Q. Li, and K. Shi, “Advances in Attack–Defense Game Models for IIoT: A Review,” IEEE Internet of Things Journal, vol. 12, pp. 46417–46430, Nov. 2025. [19] J. B. Clempner, “Learning Deceptive Tactics for Defense and Attack in Bayesian–Markov Stackelberg Security Games,” Mathematical and Computational Applications, vol. 30, p. 29, Mar. 2025. [20] A. Ait Temghart, M. Marwan, and M. Baslam, “Stackelberg Security Game for Optimizing Cybersecurity Decisions in Cloud Computing,” Security and Communication Networks, vol. 2023, pp. 1–13, Dec. 2023. [21] S. Sengupta and S. Kambhampati, “Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense,” July 2020. [22] Y. Zhang and P. Malacaria, “Bayesian Stackelberg games for cyber-security decision support,” Decision Support Systems, vol. 148, p. 113599, Sept. 2021. [23] H. Li and Z. Zheng, “Robust Moving Target Defense Against Unknown Attacks: A Meta-reinforcement Learning Approach,” in Decision and Game Theory for Security (F. Fang, H. Xu, and Y. Hayel, eds.), vol. 13727, pp. 107–126, Cham: Springer International Publishing, 2023. [24] X. Deng, P. Li, R. Wang, J. Tan, Y. Liu, W. Han, and Z. Tian, “Learning Sequential Deception Defense Strategy Against APT Using Stackelberg Markov Game,” IEEE Transactions on Information Forensics and Security, vol. 21, pp. 2492–2504, 2026. [25] L. A. T. Cox, Jr., “Game Theory and Risk Analysis,” Risk Analysis, vol. 29, pp. 1062–1068, Aug. 2009. [26] Z. Lian, P. Shi, and M. Chen, “A Survey on Cyber-Attacks for Cyber-Physical Systems: Modeling, Defense, and Design,” IEEE Internet of Things Journal, vol. 12, pp. 1471–1483, Jan. 2025. [27] A. Y. Ng, “Shaping and policy search in Reinforcement Learning,” tech. rep., UNIVERSITY of CALIFORNIA, BERKELEY, 2003. [28] A. Babar, T. Halabi, and M. Zulkernine, “Autonomous and Adaptive Cyber Incident Detection and Response in Industrial Cyber-Physical Systems Using Hierarchical Reinforcement Learning,” ACM Trans. Cyber-Phys. Syst., vol. 10, pp. 8:1–8:27, Jan. 2026. [29] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg, “Specification gaming: The flip side of AI ingenuity,” Apr. 2020. [30] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mane, “Concrete Problems in AI Safety,” Tech. Rep. arXiv:1606.06565v2, arXiv, 2016. [31] A. Pan, K. Bhatia, and J. Steinhardt, “The Effects Of Reward Misspecification: Mapping And Mitigating Misaligned Models,” Tech. Rep. arXiv:2201.03544v2, arXiv, 2022. [32] J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger, “Defining and Characterizing Reward Hacking,” in NeurIPS 2022, 2022. [33] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017. [34] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison, “Hidden Technical Debt in Machine Learning Systems,” in NeurIPS 2015, 2015. [35] E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ML test score: A rubric for ML production readiness and technical debt reduction,” in 2017 IEEE International Conference on Big Data (Big Data), (Boston, MA), pp. 1123–1132, Ieee, Dec. 2017. [36] S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zim- mermann, “Software Engineering for Machine Learning: A Case Study,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), (Montreal, QC, Canada), pp. 291–300, Ieee, May 2019. [37] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, pp. 54–71, May 2019. [38] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021. 62 Blakely et al. [39] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, vol. 114, pp. 3521–3526, Mar. 2017. [40] S. Fenu, K. Jhala, and S. Plathottam, “Perturbation-based cybersecurity for virtual power plant operated DERs using reinforcement learning,” in 2025 IEEE Power & Energy Society General Meeting (PESGM), pp. 1–5, 2025. Appendices 64 Blakely et al. A ADJACENT PROJECT CONTRIBUTIONS Beyond the integrated RL/GT loop pipeline that is the subject of this report, the project produced two related software contributions and started a third. They are not part of the data path evaluated in Sections 4–5, but they are substantive artifacts of the project effort, partly open-sourced, and relevant to the future-work directions in Section 7. A.1 adcyder-attack-simulator (Public Release) The adcyder-attack-simulator is a Python framework for executing cyber attacks against industrial-control-system targets running the Modbus TCP and DNP3 protocols. Its purpose, distinct from the synthetic SCADA generator described in Section 3.2, is to drive real protocol-level attack traffic against operational ICS endpoints — an emulated outstation in development, or an OPAL-RT-hosted simulated facility in the planned hardware-in-the-loop environment. Where the synthetic generator produces simulated netflow records, IDS alerts, and ADS samples consistent with an attack pattern, the attack simulator constructs and transmits live protocol frames to a target on the network and observes the target’s actual response. The two are complements: the synthetic generator made evaluation tractable in the absence of HIL access (Section 3.2, Origin and motivation), while the attack simulator was the intended attack source for the HIL-deployed framework. The simulator implements TCP-based persistent-connection clients for both protocols with proper transaction-ID handling, MBAP (Modbus Application Protocol) header con- struction for the Modbus client, and connection-lifecycle management. The DNP3 client combines the FreyrSCADA dnp3protocol library for standard function-code operations with a custom raw-protocol module (dnp3 raw.py) for attack-specific frame crafting that requires fields the library does not expose, including manual CRC computation and Wireshark-compatible frame generation for protocol analysis. Device discovery uses UDP broadcast to 255.255.255.255: Modbus discovery via Read Device Identification (func- tion code 0x2B/0x0E) extracts vendor and model information from responsive devices; DNP3 discovery uses basic read requests to identify outstations and probe protocol char- acteristics. Standalone command-line entry points (src/standalone modbus attack.py, src/standalone dnp3 attack.py) execute attacks directly without infrastructure depen- dencies, with configurable point/register address ranges to match the OPAL-RT target’s addressing scheme (the address-range configurability and OPAL-RT-specific notes are documented in the simulator’s docs/OPAL-RT-COMPATIBILITY.md). The simulator implements four attack-vector families: (i) a Watering Hole Attack chaining initial system compromise via SSH and Metasploit-driven payload execution; (ii) Modbus attacks including command injection, denial-of-service via resource exhaustion, false data injection to control registers, and information exfiltration from device memory; (iii) DNP3 attacks including binary and analog point manipulation, control-sequence injection, and exfiltration across multiple object groups; and (iv) data-streaming attacks targeting the historian and gRPC data-injector components included in the same codebase. The framework was open-sourced as ANL-CEEESA/adcyder-attack-simulator on GitHub. It was handed off to the second university partner’s testbed team for integration with their OPAL-RT-based AdCyDER: Online RL with Stackelberg-Optimized Defenses 65 infrastructure, but, as noted in Section 3.2, that integration was not completed within the project timeline. A.2 test-outstation (Internal) The test-outstation is an internal-only DNP3 protocol test environment built as a debugging aide for the attack simulator (Section A.1) during its preparation for the partner- testbed OPAL-RT integration. It allowed the team to iterate on the simulator’s DNP3 client behavior against a known-good local outstation without requiring concurrent access to the OPAL-RT testbed. The environment is complete and functional — it is not under active development — and remains available as an internal tool for any future DNP3 client work. The environment bundles three outstation implementations at varying levels of fidelity (a full pydnp3-based outstation with pre-populated binary and analog input values, a light- weight raw-socket DNP3 server with no external library dependency for basic READ-request handling, and a minimal echo server for link-layer inspection); two client implementations matching those (a pydnp3 master, and a raw-socket client that constructs DNP3 frames by hand with manual CRC computation, useful for validating the attack simulator’s own raw-socket logic against the same wire format); and a packet-capture-and-analyze toolkit (capture pydnp3.py for parsing live traffic into DNP3 link / transport / application layers, and a tcpdump-driven shell wrapper for hex-dumping captures alongside the test runs). The environment is not part of any public release; it is documented here as a substantive contribution to the project’s debugging tooling. A.3 Workflow-Orchestrated Simulator Rewrite (Prototype) A second-generation rewrite of the attack simulator was prototyped using the Prefect 3.5 workflow-orchestration framework, with Pydantic 2.10 type-safe configuration, structured logging, and integration with Grid-STIX1 for threat-intelligence interchange. The prototype reorganizes the simulator’s attack execution as composable DAG workflows with automatic retry, contextual logging, and per-task error handling; protocol coverage is extended to Modbus TCP, DNP3, SSH, Metasploit, and generic TCP/IP, and outputs can be streamed to file, Kafka, Kinesis, or gRPC sinks. The prototype is functional at unit-test scope but was not deployed in any operational use within the project. We document it here for traceability and as a candidate starting point for future workflow-orchestrated extensions of the attack simulator (Section 7). 1 Grid-STIX is a canonical data model for power-grid cybersecurity, developed at Argonne under the Iowa State University-led ZTCARD program (DOE CESER funded), available at https://github.com/argonne- citadel/grid-stix. www.anl.gov ARGONNE NATIONAL LABORATORY ∙ U.S. Department of Energy research facility ∙ Operated by the UChicago Argonne, LLC ∙ Midwest’s largest federally funded R&D facility ∙ Located in Lemont, IL, about 25 miles (40 km) southwest of Chicago, IL (USA) ∙ Conducts basic and applied research in dozens of fields ∙ Unique suite of leading-edge and rare scientific user facilities CONTACT Argonne National Laboratory 9700 South Cass Avenue Lemont, IL 60439 630-252-2000 Argonne National Laboratory is a U.S. Department of Energy laboratory managed by UChicago Argonne, LLC.