SAND2025-12333R SANDIA REPORT SAND2025 Printed September 2025 InterGraph-CPS: A Graph- Theoretic Approach to Characterize Cross-Domain Cyber- Physical Interdependencies and Uncertainties in Electric Grid Systems for Improved Decision- Making in Operation and Response Principal Investigators: Shamina Hossain-McKenzie and Nicholas Jacobs Sandia Team: Christopher Goes, Michael Livesay, Adam Summers, Logan Blakely, Georgios Fragkos Texas A&M University: Shining Sun, Emily Payne, Astrid Layton, Katherine Davis Southwest Research Institute: Erin DeCarlo Emera Technologies: Daniel Bauer Prepared by Sandia National Laboratories Albuquerque, New Mexico 87185 and Livermore, California 94550 CONTENTS Abstract ............................................................................................................................................................... 3 Acknowledgements............................................................................................................................................ 4 Executive Summary ......................................................................................................................................... 13 1. Introduction and Problem Statement .................................................................................................... 16 1.1. Introduction and Problem Statement .......................................................................................... 16 2. Background ................................................................................................................................................ 17 2.1. Interdependency Analysis for Resilience Assessment ............................................................... 17 2.2. Interdependency Analysis for Cyber-Physical Situational Awareness .................................... 17 2.3. Cyber-Physical Interdependency Analysis Gaps and Challenges ............................................ 19 3. Cyber-Physical Emulation Models ......................................................................................................... 20 3.1. Emulation......................................................................................................................................... 20 3.2. Western System Coordinating Council (WSCC) 9-bus power system .................................... 20 3.2.1. WSCC 9-bus scenario runs .............................................................................................. 20 3.3. Direct Current Emera Microgrid .................................................................................................. 21 3.3.1. Initial Emera emulation .................................................................................................... 23 3.3.2. Simplified Emera Emulation ........................................................................................... 25 3.3.3. OPAL-RT integration with SCEPTRE ......................................................................... 26 3.3.4. Disruption Scenarios for the Emera emulation ............................................................ 27 3.4. Experiment Data Collection ......................................................................................................... 29 3.4.1. Scenario Orchestration ..................................................................................................... 29 3.4.2. Data post-processing with Harmonize .......................................................................... 31 3.5. Open-source Contributions .......................................................................................................... 36 4. Structural and Temporal Analysis Real-Time Techniques Framework ............................................ 37 4.1. Background ...................................................................................................................................... 37 4.2. Clustering Techniques .................................................................................................................... 37 4.3. Graph Embedding Technique ...................................................................................................... 39 4.3.1. Introduction of Graph Embedding Technique ............................................................ 39 4.3.2. Methodology - DeepWalk ................................................................................................ 40 Node2Vec-based Risk Analysis .................................................................................................... 41 4.3.3. Results ................................................................................................................................ 45 4.4. Bio-Inspired Cyber-Physical Interdependency Network Analysis .......................................... 48 4.4.1. Bio-inspired Design .......................................................................................................... 48 4.4.2. Bipartite and Tripartite Networks................................................................................... 49 4.4.3. Ecological Network Analysis (ENA) ............................................................................. 88 4.5. Dynamic Mode Decomposition – .............................................................................................. 107 4.5.2. Stability Analysis with Dynamic Mode Decomposition ............................................ 110 4.5.3. Forecasting With Dynamic Mode Decomposition .................................................... 111 4.6. Long Short-Term Memory Autoencoder Neural Networks .................................................. 112 4.6.1. LSTM AE Classification of Emera Microgrid Disturbances .................................... 113 4.7. Power Resilience Inspection of Synchronous Motor Model.................................................. 119 4.7.1. Introduction of PRISM .................................................................................................. 119 4.7.2. Modeling Cyber Events and Control with PRISM .................................................... 120 4.7.3. Phase Shift Stability and Sensitivity Analysis .............................................................. 126 4.7.4. Performance ..................................................................................................................... 133 5 5. Cyber-Physical Uncertainty Quantification ......................................................................................... 134 5.1. Developing Initial UQ Methodology Plan ................................................................................ 134 5.2. Testing of InterGraph-CPS Approach Within Emulation ..................................................... 136 5.2.1. UQ Studies around Ecological Network Analysis (ENA) ........................................ 136 5.2.2. UQ Studies around Dynamic Mode Decomposition (DMD) .................................. 140 6. Integrated Cyber-physical interdependency inights Using the START Framework ..................... 149 6.1. Overall START Integration Takeaways..................................................................................... 150 6.2. Integrated UQ Analysis for Cyber-Physical Systems............................................................... 151 7. Conclusions and Next Steps .................................................................................................................. 154 References ....................................................................................................................................................... 155 Distribution..................................................................................................................................................... 167 LIST OF FIGURES Figure 1: Integrated Cyber-Physical System Exemplar. .............................................................................18 Figure 2: WSCC 9-bus and Corresponding Notional Control Network. ................................................21 Figure 3: Nanogrid Architecture and Grid Connections, from [19] ........................................................21 Figure 4: Emera Microgrid layout, from [19] ...............................................................................................22 Figure 5: The Simulink model for the Emera MG .....................................................................................22 Figure 6: The initial Emera emulation communication network ..............................................................24 Figure 7: The simplified Emera communication network .........................................................................25 Figure 8: Data flow diagram of SCEPTRE Provider for the OPAL-RT simulator...............................27 Figure 9: Screenshot of a Scorch run for the WSCC 9-bus system with the RTDS simulator ............30 Figure 10: Example of a Scorch run in progress for a run of the Emera DoS scenario .......................31 Figure 11: Screenshot of a portion of the human-readable report in HTML format............................32 Figure 12: Screenshot of part of the CSV header for the file “experiment_results.csv” ......................32 Figure 13: Example directory structure of results for a DoS run .............................................................33 Figure 14: Screenshot of a portion of the Kibana dashboard for the Emera MG.................................34 Figure 15: Screenshot of Kibana Dashboard for Emera MG displaying Load Voltage and Current .35 Figure 16: Screenshot of Kibana Dashboard for Emera MG displaying Battery Voltage, Current, and State of Charge ...................................................................................................................................35 Figure 17: Screenshot of Kibana Dashboard for Emera MG displaying PV and Bus Voltage, Current, and Power ...................................................................................................................................36 Figure 18: CPG of WSCC 9-bus system, color coded for graph components. (Right) Condensation Graph of WSCC 9-bus system. From [27]. .................................................................38 Figure 19: K-Shell Decomposition of WSCC 9-bus CPG. From [27]. ....................................................39 Figure 20: Node2Vec-based Risk Analysis Framework .............................................................................42 Figure 21: Colored, one-line Diagram for WSCC 9-bus Physical System by 3 Areas. Figure from [27]. ..............................................................................................................................................................50 Figure 22: Cyber Network for WSCC 9-bus Physical System by 3 Areas. Figure from [27]. ...............51 Figure 23: Bipartite Network shown as two groups of actors “D” indicating digital components and “P” indicating physical components. Figure from [60]................................................................52 Figure 24: Bipartite Cyber-physical Model, with Different Colors Indicating Each Module. Figure from [34]. ....................................................................................................................................................54 Figure 25: Modules found within the cyber-physical interface of WSCC 9-Bus case study for (a) the original, Scenario 1, and Scenarios 4–6; and (b) for Scenarios 2 and 3. There are no connections at the interface that fall outside of a module. Figure from [27]. ..................................56 6 Figure 26: A color-coded one-line diagram of the WSCC 9-bus physical system, distinguishing different substations for better visualization. Figure from [60]. ........................................................60 Figure 27: A hypothetical bipartite network model highlighting the interface connections between two groups of actors: "C" for cyber components and "P" for physical. Figure from [60]. ...........61 Figure 28: A hypothetical tripartite network model with three types of actors: "C" for digital/cyber components, "P" for physical components, and “CP” for cyber-physical components. Figure from [60]. ...............................................................................................................62 Figure 29: The bipartite network for the WSCC 9-bus system under normal conditions with two groups of actors and the total count of interactions per module. Figure from [60]. ......................65 Figure 30: The normal tripartite network for the WSCC 9-bus system with three groups of actors and the total count of interactions per module. Figure from [60]. ....................................................67 Figure 31: Distributions of Hub, Non-Hub, and Peripheral node classes among cyber and physical devices in the WSCC 9-bus system generated from the tripartite network. Figure from [60]. ....................................................................................................................................................69 Figure 32: The plotting of node elimination for the tripartite network for the WSCC 9-bus system. Figure from [60]. .........................................................................................................................70 Figure 33: A tripartite graph model for the ACTIV 200-bus power system. Physical, cyber physical, and cyber nodes are connected via the initial node and target link structure generated from the initial graph simulation. The count color scale is representative of the frequency of connections at the physical device. Figure from [60]. ..................................................71 Figure 34: Distributions of Hub, Non-Hub, and Peripheral node classes among cyber and physical devices in the ACTIV 200-bus system generated from the tripartite network. Figure from [60]. ....................................................................................................................................................75 Figure 35: The plotting of a node elimination heatmap for the tripartite network for the ACTIV 200-bus system. Here each square represents the number of nodes remaining for each disturbance scenario for selected runs of node elimination. The color gradient to the right of the figure represents the range of nodes available. Figure from [60]. ...............................................76 Figure 36: A color coded diagram of the WSCC 9-bus case study [139]. Figure modified from [34]. ..............................................................................................................................................................78 Figure 37: Left - An example of a hypothetical cyber-physical power system bipartite network model of the cyber-physical interface, where “C” represents cyber components and “P” represents physical components. Right – a hypothetical plant-pollinator bipartite network model where generalists (left) interact with many partner species, while specialists (right) interact with few partners. In both models, interactions can be described as mutualistic, or mutually beneficial. Figure on right modified using [140]. Figure left modified from [34]............79 Figure 38: An example of different relay protection schemes, including individual relay zones (thin blue dashed circles), local and duplicate backup zones (thick light blue dashed oval), and remote relay backup zones (thick purple dashed circle). The red blocks with R# are the relays and the black lines connecting them are the lines/interactions. Vertical black lines are the physical buses (corresponding to those in Fig. 1). The disturbance is indicated by the lightning bold and “fault” label between relays 1 and 2. .....................................................................82 Figure 39: Bipartite network models of the cyber-physical interface of a) the original WSCC 9- Bus cyber-physical network, b) the remote relay traditional redesign, and c) the bio-inspired optimized redesign. The colored bar legend is the number of new line connections per device. Note: the two redesigns (b, c) go from 1-7 links while the original (a) is only 1-3 links. Actor abbreviations: B-bus, G-generator, L-load, R-relay. Figure from [65]. ..................................84 Figure 40:Cyber-Physical Graph of Emera MG Model .............................................................................86 Figure 41: Emera bipartite graph. ..................................................................................................................87 7 Figure 42: A hypothetical food web directional graph (left) and its structural matrix (right). Edges/connections are drawn and quantified as being from prey to predator. Figure from [145].............................................................................................................................................................88 Figure 43: WSCC 9-bus System Cyber Topology. Figure from [34]. .......................................................93 Figure 44: Information Flow Diagram for One Way Sending Data. Figure from [66]. ........................94 Figure 45: Information Flow Diagram of One-Way Delivering Command. Figure from [66]. ...........94 Figure 46: A high-level directional graph for a cyber network of a power system. The cloud represents the system boundary. Figure from [66]. ..............................................................................95 Figure 47: The flow matrix [T] for the cyber network depicted in Figure 4.25. Figure from [66]. .....95 Figure 48: Normal and disturbed cases plotted on the ecological fitness curve, with the ecological Window of Vitality highlighted in green. “Normal” indicates the fully operational/undisturbed case study, “DS” indicates disturbance scenario 1 or 2. Figure from [66]. ..............................................................................................................................................................97 Figure 49: An example of the WSCC 9-bus power system cyber topology. ........................................ 101 Figure 50: The Western System Coordinating Council (WSCC) 9-Bus power system case study. Figure modified from [60]. ................................................................................................................... 102 Figure 51: A hypothetical flow matrix [T] (right) of a small food web network model (left), red dashed lines represent the selected system boundaries. The matrix (right) highlights the following entries: “In” are system inputs, “Out” are useful system exports, and “Diss” are dissipations or lost usefulness that leaves the system. ...................................................................... 103 Figure 52: TSTp (packets) over time (milliseconds, ms) for normal and disturbance scenarios. Blue and green colored areas in the disturbance plot (right) show times when the TSTp drifted from the maximum and minimum values captured in the normal scenario (left). .......... 105 Figure 53: DoSO over time for normal and disturbance scenarios. The colored section represents the point of occurrence of the disturbance. ....................................................................................... 106 Figure 54- DMD identification of unstable windows of data. ............................................................... 110 Figure 55-DMD Stable and Unstable Window of Data.......................................................................... 110 Figure 56-Stable Windows of data ............................................................................................................. 111 Figure 57-DMD Forecasting Generator 1 Frequency. ............................................................................ 112 Figure 58: Autoencoder encoding-decoding scheme. ............................................................................. 113 Figure 59: Overall data mean and standard variance. .............................................................................. 114 Figure 60: Training Loss .............................................................................................................................. 114 Figure 61: Validation Loss ........................................................................................................................... 115 Figure 62: Normal data reconstruction error distribution. ..................................................................... 115 Figure 63: MITM attack RE distribution. ................................................................................................. 116 Figure 64: Reconstruction results from varying thresholds for MITM attack. .................................... 117 Figure 65: Overall data mean and standard variance. .............................................................................. 118 Figure 66: Abnormal RE distribution. ....................................................................................................... 118 Figure 67: Reconstruction results from varying thresholds for DoS attack. ........................................ 119 Figure 68: SIMULATION OF THE PHASE AND FREQUENCY OF THE WSCC 9-BUS SYSTEM BY THE PRISM MODEL WITH THE MAIN GENERATOR STARTING OUT OF PHASE .................................................................................................................................. 120 Figure 69: THE WSCC 9-BUS SYSTEM ................................................................................................. 121 Figure 70: THE WSCC 9-BUS SYSTEM WITH THE 6-9 POWER LINE DOWN AS INDICATED BY THE LARGE RED ‘X’ ....................................................................................... 121 Figure 71: FREQUENCY AND PHASE FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM WITH THE 6-9 POWER LINE REMOVED AT 3 SECONDS ................................................ 122 8 Figure 72: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 4-5 LINE DOWN ........................................................................................................................................ 122 Figure 73: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 4-6 LINE DOWN ........................................................................................................................................ 123 Figure 74: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 5-7 LINE DOWN ........................................................................................................................................ 123 Figure 75: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 7-8 LINE DOWN ........................................................................................................................................ 123 Figure 76: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 8-9 LINE DOWN ........................................................................................................................................ 124 Figure 77: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 6-9 LINE DOWN ........................................................................................................................................ 124 Figure 78: CONTRASTING THE FREQUENCIES OF THE WSCC 9-BUS SYSTEM AFTER 4-5 POWER LINE GOES DOWN (LEFT) AND 6-9 LINE GOES DOWN (RIGHT) ....... 124 Figure 79: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, AN EVENT OCCURS WHERE ONE OF THE LOADS INCREASES LINEARLY BEFORE HOLDING CONSTANT ................................................................................................................... 125 Figure 80: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, TWO GREEDY CONTROLERS TASKED TO ASSURE THEIR GENERATOR’S FREQUENCY DOESN’T GO TOO LOW. EITHER CONTROLLER IS SAFE FOR THE SYSTEM (LEFT AND MIDDLE), BUT TOGETHER THEY VIOLENTLY BREAK THE SYSTEM (RIGHT)................................................................................................................................ 125 Figure 81: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, EVENT WHERE A LOAD IS INCREASING AS IN FIGURE 81 (LEFT). ADDED IN SUPER POSITION IS A CONTROLLER THAT DOES ACTIVATES TO STABILIZE THE SYSTEM BUT IS NOT A DROOP CONTROLLER AND DOESN’T TAKE INTO ACCOUNT THE MOMENTUM OF THE SYSTEM ................................................................................................... 126 Figure 82: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, SYSTEM RESPONDS TO A SHORT SINUSOIDAL THROTTLE ADDED TO GENERATOR 3 . 126 Figure 83: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 3 .... 127 Figure 84: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 3 .... 127 Figure 85: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 1 .... 128 Figure 86: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 2 .... 129 Figure 87: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 3 .... 130 Figure 88: IEEE 39-Bus Oneline Diagram. .............................................................................................. 131 Figure 89: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 3 .... 132 Figure 90: SCEPTRE-RTDS CPS emulation environment and four InterGraph-CPS modeling approaches to capture cyber-physical interdependencies................................................................. 134 Figure 91: 9-bus system detailing the 3 generators, 9 busses, 21 relays, and 3 load shedding devices...................................................................................................................................................... 135 Figure 92: WSCC 9-bus cyber topology with three sub-stations and one control center communicating with the 21 physical relays. ....................................................................................... 136 9 Figure 93: Degree of System Order (DoSO) vs Ecological Fitness (Reco) for six candidate 9-bus packet flow scenarios developed by TAMU. ..................................................................................... 137 Figure 94: ENA cyber packet flow matrix for normal operations ........................................................ 137 Figure 95: Distributions of DoSO (left) and R-eco (right) after uncertainty propagation with 10% CoV applied to each packet flow value in the nominal operating condition. ............................... 138 Figure 96: Propagated uncertainty around DoSo and Reco relative to the window of vitality for the nominal operating condition.......................................................................................................... 139 Figure 97: First order sensitivity indices for 56 flow parameters to DoSO using the U(-1,1) variation scheme. .................................................................................................................................... 140 Figure 98: First order sensitivity indices for 56 flow parameters using the 10 % CoV variation scheme. .................................................................................................................................................... 140 Figure 99: PMU frequencies (top row) and voltages (bottom 3 rows) for the 8 PMUs in the 9-bus system for a cyber-physical disruption scenario without mitigation............................................... 141 Figure 100: Assessment of system stability using eigenvector values relative to the unit circle........ 141 Figure 101: Mode 1 and Mode 2 eigenvalues on real and imaginary axes from an existing scenario (red) and a new scenario (yellow)......................................................................................................... 142 Figure 102: 2D probability contours from KDEs with a bandwidth of 0.2 for Mode 1 and Mode 2 eigenvalues from an existing scenario (red) compared to new scenario (yellow). ..................... 142 Figure 103: 2D probability contours from KDEs with a bandwidth of 0.5 for Mode 1 and Mode 2 eigenvalues from an existing scenario (red) compared to new scenario (yellow). ..................... 143 Figure 104: (Left) PMU1 frequency output on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues on the unit circle scale on the left axis. The red shaded area depicts the -1 to 1 range on the left-axis to represent the un ..................... 143 Figure 105: 32 voltages and frequency outputs for the 8 PMUs on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues on the unit circle scale on the left axis. The red shaded area depicts the -1 to 1 range on the left-axis.................................. 144 Figure 106: (Left) PMU1 frequency output on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues from a time window of 100 timesteps on the unit circle represent the unit circle range identified on the right axes. (Right) T ................... 145 Figure 107: Phases and frequencies of each node in the WSCC 9-bus system during a disturbance scenario .................................................................................................................................................... 145 Figure 108: (Top row) Optimal numbers of retained modes for Bus 1 across different training data time spans along with (bottom row) the eigenvalues if 11 modes are retained across all time spans ................................................................................................................................................ 146 Figure 109: An example of DMD forecasting and resulting prediction discrepancy. ........................ 147 Figure 110: DMD training and decision-making pathways regarding forecasting, stability, and the dynamically changing system. ............................................................................................................... 147 Figure 111: Summary of the START framework. .................................................................................... 149 Figure 112: Notional relationship between true system structure and observed outputs in a cyber- physical system ....................................................................................................................................... 151 Figure 113. LSTM AE reconstruction producing small errors because the true structure and assumed structure are equivalent ......................................................................................................... 152 Figure 114. LSTM AE reconstruction producing triggering errors because the true structure and assumed structure are not equivalent .................................................................................................. 152 Figure 115.CPS UQ integration with DMD, LSTM AE, and ENA approaches ................................ 153 10 LIST OF TABLES Table 1: Access Complexity Scores provided by CVSS [38] .....................................................................42 Table 2: Assessment of Impact Score: Categorization and Grading Levels ...........................................44 Table 3: Components with High Risks During Disturbance ....................................................................45 Table 4: Risk Assessment Result for High-Risk Nodes with ‘r2‘ as Target for the WSCC 9-Bus Cyber-Physical System..............................................................................................................................47 Table 5: Risk Assessment Result for High-Risk Nodes with ‘SubA1_SW‘ as Target for the WSCC 9-Bus Cyber-Physical System ..................................................................................................................47 Table 6: Common Bio-inspired systems design metrics and methods ....................................................49 Table 7: Bipartite Modularity Analysis Results ............................................................................................53 Table 8: Modules for WSCC 9-bus CPS system. ........................................................................................56 Table 9: Topology evaluation results of normal, and disturbance scenarios for the WSCC 9-bus system..........................................................................................................................................................65 Table 10: Topology evaluation results including the number of interactions, the number of modules, and the interior to exterior ratio of components in the WSCC 9-bus system. ...............67 Table 11: Topology evaluation results of normal, and disturbance scenarios for the ACTIV 200- bus system. .................................................................................................................................................72 Table 12: Topology evaluation results including the number of interactions, the number of modules, and the interior to exterior ratio of components in the ACTIV 200-bus system...........73 Table 13: The three network designs are investigated under normal operational conditions and 2 disturbances (DS1 and DS2). The original WSCC 9-bus model, the traditional remote relay protection redesign, and the bio-inspired redesign are described by their before and after nestedness (NODF), bipartite robustness (BR), and interactions added beyond the original design (∆WSCC). The bolded/red values correspond to the maximum values of NODF and BR across all model versions. ..................................................................................................................83 Table 14: The starting node, direct, and primary indirect interactions for the WSCC 9-bus original case study, remote relay protection redesign, and bio-inspired redesign. .........................................85 Table 4.15 ..........................................................................................................................................................87 Table 16: Connectance, Nestedness, and Modularity metrics for the Emera microgrid system in normal operating state. .............................................................................................................................88 Table 17: Network analysis metrics for the START framework captured for changes in topology and scenario. ..............................................................................................................................................89 Table 18: Network analysis metrics for the Emera microgrid. .................................................................90 Table 19: Shannon Index (H) and Evenness (E) for the normal, DS1, and DS2 scenarios for both topologies. ..................................................................................................................................................97 Table 20: Topology Evaluation Results of Normal, DS1, and DS2.........................................................98 Table 21: Performance evaluation (components impacted and PP) for the two disturbance scenarios tested. .........................................................................................................................................99 Table 22: Generator sensitivity analysis for IEEE 39-bus system without any topology changes. .. 131 Table 23: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 3 .... 132 Table 24: IEEE 39 BUS SYSTEM WITH POWERLINE ADDED FROM BUS 1 TO 29........... 132 Table 25: PRISM computational runtime and complexity ..................................................................... 133 11 This page left blank 12 EXECUTIVE SUMMARY The electric grid is becoming increasingly cyber-physical, rapidly evolving with smart grid technologies, wide-area monitoring capabilities, and advanced automation. However, these modernization efforts also broaden the grid’s vulnerability surface and increase cyber-physical mutuality. Multi-hazard events such as cyber attacks and climate change-driven extreme weather can cause detrimental, cascading impact. The cyber-physical nature of the grid does not limit cyber disturbances to the cyber system and physical disturbances to the physical system – disturbances can propagate between systems. Similarly, operational system changes can affect both cyber and physical domains; observability is needed for both normal operation and during disturbances. Obtaining cross-domain observability into the cyber-physical system (CPS), including connected critical infrastructure, is an important requirement to informing grid operation and response. It no longer suffices to only monitor the physical system to achieve full observability of the grid. Conversely, network monitors that only process cyber data are not enough to thwart adversaries and/or detect communication failures. Therefore, cyber-physical observability (CPO), the ability to determine both cyber and physical system states with input and output measurements in finite time, is crucial for characterizing the grid’s intricate cyber-physical interactions and enabling detailed feature extraction. Prior work developed an approach to define CPO by combining physical observability algorithms with graph-theoretic network observability methods. However, the interdependencies between cyber and physical graph-nodes and their formal definition were not investigated. The CPO graph- theoretic model, an initial step to leverage for this proposal, must be revised and added to with techniques such as multi-order dependency analysis, network decomposition, principal component analysis, and/or coupled oscillators to gain observability into the CPS interdependencies and assess controllability needs. We hypothesize that a graph-theoretic modeling approach using these numerical analysis and decomposition techniques combined with control-theoretic observability/controllability concepts can be developed to assess cyber-physical interdependencies and dynamics. This InterGraph-CPS approach can then be used to study the alignment of terms and quantities in a mixed system model and how those information flows change across system boundaries. It is crucial to assess how seemingly disparate cyber and physical data sets can be combined in a rigorous and useful manner for improving grid operation and response; this CPO would help identify weak points that require hardening and controllability analyses will improve mitigation strategies to increase grid resilience. Controllability-informed mitigations will be studied with partner TAMU and leverage prior sensitivity-based control characterization work InterGraph-CPS will be developed with a unique set of uncertainty quantification (UQ) methods to assess the critical CPS features and how they can inform operation/response decision-making (utilizing partner SwRI’s UQ expertise). For data collection and disturbance/change modeling, a high-fidelity cyber-physical emulation will be leveraged from existing RES MC HARMONIE-SPS LDRD environment with RTDS and SCEPTRE™; this will enable real-time cyber-physical data extraction and InterGraph- CPS validation using our novel UQ approach. The second stage of InterGraph-CPS validation will include emulating a real system, proposal partner Emera Technologies Kirtland Airforce Base (KAFB) microgrid; SNL’s existing Emera partnership and system model will allow streamlined environment development and real dataset access. 13 Acronyms and Terms Acronym/Term Definition AC Alternating Current AE Autoencoder ANN Autoencoder Neural Network API Application Programming Interface ARP Address Resolution Protocol ATS Automated Transfer Switch CAN Controller Area Network CPG Cyber-Physical Graph CPO Cyber-Physical Observability CPS Cyber-Physical System CPSA Cyber-Physical Situational Awareness CPU Central Processing Unit CSV Comma-Separated Values DC Direct Current DoS Denial of Service DOE Department of Energy DMD Dynamic Mode Decomposition DNP3 Distributed Network Protocol version 3.0 ENA Ecological Network Analysis ICS Industrial Control Systems IT Information Technology IP Internet Protocol KAFB Kirtland Air Force Base LAN Local Area Network LDRD Laboratory Directed Research and Development LSTM Long Short-Term Memory MG Micro-grid 14 Acronym/Term Definition MitM Man-in-the-Middle NG NanoGrid OT Operational Technology OPC Object Linking and Embedding for Process Control OE (Department of Energy) Office of Energy PCAP Packet Capture PMU Phasor Measurement Unit PRISM Power resilience inspection of synchronous motor PV Photovoltaic SCORCH Scenario Orchestration SCADA Supervisory Control and Data Acquisition SECURE Science and Engineering of Cybersecurity by Uncertainty quantification and Rigorous Experimentation SETO (Department of Energy) Solar Energy Technologies Office START Structural and Temporal Analysis Real-time Techniques SwRI Southwest Research Institute SNL Sandia National Laboratories UQ Uncertainty Quantification RE Reconstruction Error RTU Remote Terminal Unit RTDS Real-Time Digital Simulator TAMU Texas A&M University WAN Wide Area Network WSCC Western System Coordinating Council VM Virtual Machine 15 1. INTRODUCTION AND PROBLEM STATEMENT 1.1. Introduction and Problem Statement Cyber-physical systems (CPSs) can be defined as systems that combine sensing, computation, control, and networking into physics-based processes, connecting them to the internet and to each other. Critical infrastructure systems such as the electric grid are increasingly cyber-physical due to recent modernization and expansion efforts. Autonomous and adaptive control as well as cloud- based analytics and data storage are just two examples of capabilities that strengthen the cyber- physical connection. More and more processes require distributed analysis and data storage that leverage internet connections to facilitate these needs. However, despite the cyber-physical characteristics of critical infrastructure systems, the physical process system and communication/control network system are traditionally analyzed in silos. For example, the information technology (IT) and operational technology (OT) systems are often monitored separately with more focus on the IT system at the enterprise system level. Although there is movement towards IT/OT convergence [1] for industrial control systems (ICSs), the physics-based processes and analysis are not considered within the ``converged" situational awareness. Instead, the physics-based system employs state estimation, stability analyses, and other analytics to assess system health and operation [2]. As critical infrastructure systems become more cyber-physical, it is crucial that models and methods are available to assess the cyber-physical system (CPS) interdependencies, characteristics, and event propagation for improved planning, operation, and response. This includes alerting on changes or detected anomalies in the CPSs, whether through logs or visualization dashboards, in a coordinated fashion to relevant stakeholders such as system operators and cyber defenders [3]. However, to streamline this coordination, it is important to first establish consistent definitions of CPSs and methods for analysis to comprehensively understand the CPS operation and important interdependencies. Comprehensive CPS interdependency results can help not only operational needs such as event detection and response but also planning efforts through identifying key cyber- physical relationships that may require additional redundancy or security. In this project, we focus on developing an integrated structural and temporal CPS interdependency analysis framework that provides insight into the CPS function during normal operation as well as disturbances. This structural and temporal analysis real-time technique (START) framework is uniquely designed for assessing CPSs by account for the challenges of analyzing cyber and physical data streams together due to data availability, data type, and time scale differences. By leveraging both structural (e.g., graph analysis) and temporal (e.g., data analytics) techniques, different CPS behaviors and configurations can be accounted for. For example, in the electric grid domain, different power system types such as transmission, distribution, and distributed energy resources as well as communication networks and protocols can be analyzed together using graph techniques that focus on structural connections and incorporate temporal data (e.g., sensor data, IP-based communication packets) where available. Thus, the START framework provides much needed flexibility to analyze CPSs of varying characteristics and sensing capabilities. Additionally, each technique within the START framework can provide cross-correlation of detected changes or events to increase confidence in system characterization and alerting. This includes detection of individual cyber and physical events as well as when those events are connected, correlated. 16 2. BACKGROUND Interdependency analysis is key for analysis of physics-based systems where variations and changes of physical components can influence overall system operation and health immediately. The level of influence or impact depends on the specific physical component, protection and safety mechanisms in place, and overall system conditions. Similarly, in cyber systems (e.g., IT/OT), interdependency analysis can be leveraged to identify network traffic patterns and congestion bottlenecks. In this section, we present existing interdependency analysis approaches and their application, prior work in cyber-physical interdependency analysis, and overall gaps for cyber-physical situational awareness (CPSA). 2.1. Interdependency Analysis for Resilience Assessment Studying interdependence between different system components and their uni- or bi-directional linkages is an effective approach for assessing system resiliency. Pierre et al. developed an analytical framework for identifying electric grid failure trends and critical components by leveraging a cascading outage model and transient dynamic simulations; this effort only focused on the physics- based system layer and to inform grid planning and response [4]. Falahati et al. proposed a framework that accounted for direct and indirect cyber-physical interdependencies which enables more comprehensive assessment of resilience impact on grid disturbances and communication failures [5]. Probabilities of the cyber-physical system states, mappings between cyber-physical states to predict impact, and reliability metrics (e.g., loss of load probability and expected energy not served) are used to assess power system reliability in the presence of cyber or physical failures. Resilience analysis across interdependent critical infrastructure systems is also an important research area but does not always consider the communication network inclusion or impact across the physics-based layers of the interconnected critical infrastructure systems. For example, Wang et al. present a deep learning framework for studying topological interdependencies across power and gas systems to help identify different failure scenarios and coupling modes [6]. With these results, the authors propose different recovery strategies to inform an optimal resilience improvement strategy across the critical infrastructure systems both from structural and functional perspectives. Lastly, Petit and Verner also highlight critical infrastructure interdependence but categorize dependencies as physical, cyber, geographic, or logical [7]. The authors emphasize the complexity of studying such systems and the need for systems of systems analysis approach as well as bottom- up/top-down perspectives for more comprehensive results. 2.2. Interdependency Analysis for Cyber-Physical Situational Awareness The interdependency between communications systems and physics-based systems (e.g., power system) is crucial to the reliable and resilient function of critical infrastructure systems. For example, for the electric grid, fiber optic networks and/or cellular networks are often used as a wide-area network (WAN) to provide monitoring, control, and protection across a transmission system. Additionally, for distribution systems, Ethernet and/or wireless mesh networks are leveraged to enable automated and adaptive operations such as demand response and response actions. These communication systems can be categorized as local area networks (LAN) and can be subdivided into field-areas, home-areas, etc. [8]. 17 An example of the interdependent cyber and physical layers of critical infrastructure systems is shown in Figure 1. Figure 1: Integrated Cyber-Physical System Exemplar. Liu et al. performed a review of different interdependency analysis techniques for cyber-physical systems and found the applications to span across resilience analysis (failure analysis, vulnerability analysis, failure mitigation, failure recovery) and simulation/modeling efforts. Their overall findings indicated that many of the approaches focused on impact in one direction (i.e., cyber system to power system), challenges in accurately modeling the CPS, and lack of mitigation/recovery methods for the integrated CPS. Additionally, the authors highlighted that there is a tradeoff between realism and tractability across modeling techniques (e.g., graph-based models vs. equation-based models vs. executable models) and that more study was needed for assessing interdependent relationships. Pacific Northwest National Laboratory [9] also highlighted the interdependence of information and communication technology and the power grid in 2015. However, their next steps and future directions focused on designing a new reference architecture for control systems, ensuring communication network standalone resilience, and developing new grid management tools to adapt to faster and more adaptive operation paradigms. Thus, this worked focused more on any power system disturbance and impact from the cyber domain. The Department of Energy (DOE) Office of Electricity (OE) also emphasized the importance of communication infrastructure and power grid interdependencies in their report “Power Grid and 18 Communication Interdependencies: Key Challenges for Reliable, Resilient Operations.” The report focused on communication security, prioritization of networking infrastructure for critical services, and coordinated recovery efforts for both communication infrastructure and the power grid for more comprehensive resilience [10]. 2.3. Cyber-Physical Interdependency Analysis Gaps and Challenges Cyber-physical modeling and analysis is an ongoing and rapidly evolving research domain for improving critical infrastructure cybersecurity, as evidenced by the prior work described in the last section. However, there exists gaps for cyber-physical interdependency analysis for comprehensive situational awareness, whether from the cyber or physical perspective. These gaps can be summarized as: o Liu et al. highlighted that there is a gap for assessing interdependent relationships and quantitative, motivating the need for both structural (graph) and temporal techniques to more accurately model CPSs. By incorporating temporal techniques, quantitative metrics can be achieved to differentiate CPS interdependencies types and intensities. o DOE OE emphasized the need for coordinated response across CPSs. Essentially, how do we provide operators and defenders the necessary tools and analytics to effectively make those decisions that affect both cyber and physical parts of a system? o Interdependency analysis is not only important for resilience analysis, as presented by Wang et al. and Fahalati et al., but for more efficient planning and operation. Thus, there is a need to understand bi-directional impact and improvements to the comprehensive CPS. o In this project, structural techniques leveraging graph theory and network analyses were explored to understand interdependencies between cyber-physical components and connections. To dive deeper into behavior and dynamics of the components and connections, temporal analysis leveraging time-series data from the CPS is needed to achieve a more accurate interdependence model for increased depth of CPSA insights for system operators and defenders to leverage. The summarized gaps highlight the crucial need for more comprehensive and integrated structural and temporal technique for analyzing CPSs. Though, it is important to note the challenges of integrating temporal and structural techniques for CPSs with different time-scales, different sensor data availability, and different operational needs across cyber and physical domains. In this report, we propose an integrated structural and temporal analysis real-time technique (START) framework that seeks to mitigate these challenges to provide comprehensive CPSA for planning, operation, and response. 19 3. CYBER-PHYSICAL EMULATION MODELS 3.1. Emulation For analysis of cyber-physical systems, emulation is an incredibly powerful tool. In emulation, components and devices in the system are virtualized for testing and analysis. This may be paired with simulation models running power flow calculations for the physical dynamical system connected to these virtual components. In this project, all of this is done using Sandia developed tools Minimega [11] and SCEPTRE [12], [13]. Minimega is an emulation tool developed at Sandia National Laboratories for fast and scalable Virtual Machine (VM) management, either locally or on server clusters. SCEPTRE is a cyber- physical emulation capability that connects minimega virtual environments to simulation providers for underlying process dynamics. It contains an orchestration tool called Phenix to help run experiments, and native Industrial Control System (ICS) protocol support for Modbus, DNP3, and other ICS protocols as well as common Supervisory Control and Data Acquisition (SCADA) software for use in experimentation and analysis. Both of these fall into a set of capabilities we at Sandia call Emulytics [14], which contain emulation and analysis toolsets to assist in experimentation and analysis for security and resilience of cyber-physical systems. 3.2. Western System Coordinating Council (WSCC) 9-bus power system For initial testing of START, a cyber-physical 9-bus power system dataset with integrated network traffic monitoring was leveraged. The 9-bus system is a simple approximation of the Western System Coordinating Council (WSCC) to an equivalent system with 9 buses and 3 generators, as shown on the left side of Figure 2. For the cyber-side, a corresponding synthetic cyber communication and control network was created for the WSCC 9-bus system, as described in [15], and is shown on the right side of Figure 2. The power system one-line diagram will be referred to as the physical model and the network diagram will be referred to as the cyber model. Together, they comprise the cyber- physical model of this power system, encompassing the integrated power system operational physics, sensing, computation, and communication to and control of power system components. 3.2.1. WSCC 9-bus scenario runs While the Emera emulation was being developed early in the project, the existing WSCC 9-bus power system from HARMONIE-SPS LDRD was leveraged [16]. The existing disruption scenarios from that project were re-used, including baseline, cyber, physical, and cyber-physical. These scenarios were re-run with additional experimental parameters added for to suit the purposes of the InterGraph-CPS project goals. Specifically, a variable packet loss was added to the network links of the target virtual RTUs, in addition to the bandwidth restrictions. This was to simulate a degraded connection quality, e.g. due to faulty equipment or an unreliable remote connection. The packet loss was applied in increments of 10%, going from 0% loss to 100% loss. The same loss was applied for the total duration of a run. There were 4 distinct scenarios (baseline, cyber, physical, and cyber-physical) with 11 permutations of packet loss per scenario (0% loss to 100% loss in increments of 10%), and 3 runs of each permutation to account for noise between runs. This resulted in 132 total runs. 20 Figure 2: WSCC 9-bus and Corresponding Notional Control Network. 3.3. Direct Current Emera Microgrid Another emulation system that was developed for this work is the Emera Microgrid (MG), which is a DC microgrid that services a residential neighborhood and has advanced metering and control infrastructure. This model was developed as part of a Cooperative Research and Development Agreement (CRADA) with Emera Technologies [17] and a simulation of this MG was developed and tested in [18], [19]. It is made up of multiple nanogrids that are connected together to form the microgrid, with each nanogrid containing rooftop PV, Battery Energy Storage Systems (BESS), and an inverter. The nanogrids provide local control and sensing in a decentralized, hierarchical control that coordinates power supply between the residential loads and the microgrid [18], [19]. The MG additionally balances and manages power supply from the PV, BESS, and from the AC grid [18], [19]. The following figures show the nanogrid structure for each modular nanogrid in the MG itself, and the layout and structure of the MG as a whole as it serves power for a residential neighborhood. Figure 3: Nanogrid Architecture and Grid Connections, from [19] 21 Figure 4: Emera Microgrid layout, from [19] Figure 5: The Simulink model for the Emera MG The simulation for this microgrid is run using a Simulink model, developed for a U.S. DOE Solar Energy Technologies Office (SETO) project #36533, “Adaptive Protection and Control for High Penetration PV and Grid Resilience”, and described in [19]. This Simulink model has Modbus points added for each nanogrid control box that can be used to connect dataflows into and out of SCEPTRE for emulation. The model is then run on an OPAL-RT instance for real-time simulation and connected to a control network emulated in SCEPTRE using Minimega. Further description of the Emera MG itself and its structure can be found in [19]. 22 3.3.1. Initial Emera emulation The initial implementation of the Emera emulation aimed to replicate the Emera network with the highest possible fidelity. This effort utilized information from multiple sources, including network diagrams, inventories, system reports, and interviews with engineers. However, the available data did not include software or configuration details for the devices within the Emera network, such as the Heila controllers. This limitation arose from the proprietary nature of the devices and restricted access to the network. The Heila nanogrid controllers consist of proprietary software operating Ubuntu Linux with an ARM CPU. These controllers employ a custom economic dispatch algorithm to optimize power injection and withdrawal within the MG. Each nanogrid incorporates a single embedded Heila controller, which communicates with other controllers and a coordinator Heila within the same subnet via a network switch. The subnet is protected by a firewall that also serves as a gateway to the Internet. In addition to the networked devices, the Emera system includes several non-networked components connected via serial interfaces such as CAN bus, RS-485, RS-422, and fiber-optic links. These devices perform various functions related to the photovoltaic (PV) and battery systems, including inverters, controllers, sensors, and protection mechanisms. The system also integrates a fast charger and data aggregation devices. 23 Figure 6: The initial Emera emulation communication network 24 All devices within the Emera system operate proprietary software, and the team did not have access to their configuration files or software binaries for use in the emulation. Consequently, it was not feasible to replicate the full functionality of the Emera microgrid. The proprietary nature of the Heila software and economic dispatch algorithm, combined with time and resource constraints, precluded the accurate reproduction of these elements in the emulation. The simulation implemented in OPAL-RT was limited to modeling the sensors and control points for the nanogrids, along with a subset of other functions. It primarily generated high-level data and power set-points for the battery and photovoltaic (PV) systems. As a result, even if the proprietary software and configuration files had been accessible, the absence of a physical process simulation would have prevented their integration into the emulation. 3.3.2. Simplified Emera Emulation To mitigate the limitations of the initial emulation, a simplified version was developed. This version included only the Heila controllers and the firewall from the Emera system, supplemented by an OPC server virtual machine (VM) to serve as a data aggregator and a Kali Linux VM for launching cyberattacks. The OPC server establishes TCP connections using the DNP3 protocol to communicate with the virtualized Heila controllers. These controllers are represented by VMs running Ubuntu Linux and the Bennu virtual field device software. Bennu emulates the functionality of a generic field device, such as a Remote Terminal Unit (RTU), by collecting data from the simulation and transmitting it over the network to the OPC server as though it were a physical system. The OPC server polls each Bennu VM every second to retrieve all available values. Figure 7: The simplified Emera communication network Additional functionality was incorporated into the emulation to enable actuation and control of various parameters within the Emera microgrid (MG) simulation in OPAL-RT. For each nanogrid, these control functions include the photovoltaic (PV) power set-point, automatic transfer switch (ATS) state, DC bus power set-point, and generator (Genset) power. For the Central Box, control functions include DC bus voltage, DC grid power, power output for Genset 1 and Genset 2, and PV 25 power. Although these capabilities were implemented, they were not utilized due to time constraints. Future work will focus on leveraging these control functions to simulate cyberattacks on the MG. The simplified emulation was limited to data collection from each Heila node and did not include logic for system management or control. Furthermore, it lacked cross-node communication, such as Heila-to-Heila interactions, which are integral to the real Emera system. Each node communicated exclusively "upward" with the OPC server, without direct communication between nodes. 3.3.3. OPAL-RT integration with SCEPTRE The Emera microgrid (MG) simulation was developed in Simulink and executed on an OPAL-RT dynamic simulator. To integrate this simulation into the emulation framework, a SCEPTRE Provider specific to the OPAL-RT simulator was created. This Provider operated similarly to the existing RTDS Provider but employed distinct communication protocols to accommodate differences in functionality between the RTDS and OPAL-RT simulators. The communication protocol used for the OPAL-RT simulator was Modbus/TCP, which supported both read operations (polling values) and write operations (e.g., modifying power setpoints or toggling the ATS switch). The interconnection between the emulated control system and the simulated power system is facilitated by a SCEPTRE Provider. Bennu virtual field devices, functioning as controllers, communicate with the Provider using a text-based UDP multicast protocol implemented via ZeroMQ. The Provider transmits updated values to the Bennu virtual machines (VMs), labeled as "RTU" in the accompanying image (Figure 8), which subsequently relay these values to the OPC server using the DNP3 protocol. Similarly, control commands—such as toggling the automatic transfer switch (ATS) or modifying power setpoints—are transmitted in the reverse direction. The ZeroMQ channels serve as the equivalent of real-world grid measurements being fed into the field 26 devices. Figure 8: Data flow diagram of SCEPTRE Provider for the OPAL-RT simulator 3.3.4. Disruption Scenarios for the Emera emulation Two disruption scenarios, simulating cyber attacks, were designed for the Emera emulation, in addition to a baseline scenario characterized by the absence of disruptions. The disruption scenarios consisted of a packet flood Denial of Service (DoS) attack and a Man-in-the-Middle (MitM) data manipulation attack, implemented via an Address Resolution Protocol (ARP) spoofing technique. These scenarios were designed to test the robustness and effectiveness of the analysis techniques developed for InterGraph-CPS. 3.3.4.1. Baseline scenario The baseline scenario consists of the Emera MG simulation operating without any cyber attacks, thereby generating a dataset representative of normal operational conditions. This dataset serves as a reference point for evaluating the efficacy of analysis techniques and facilitating comparisons with datasets that include disruptions. The baseline scenario was executed in three separate 3-hour runs, during which all data was collected but no cyber attacks were executed. 3.3.4.2. Denial of Service scenario The first disruption scenario developed was a Denial of Service (DoS) attack designed to interrupt communications between the Heila controllers and the OPC server. This attack involved a packet 27 flood that saturated the communication bandwidth of the targeted field devices. The packet flood was implemented using the hping3 tool [20], executed on an "attacker" virtual machine (VM) running Kali Linux [21]. The targets of the attack were three nanogrid (NG) controllers—NG1, NG4, and NG7—with IPv4 addresses of 10.1.32.45, 10.1.32.100, and 10.1.32.99, respectively. To ensure the attack was effective, an artificial bandwidth limitation of 500 kbps was applied to the network links associated with the three target controllers. This restriction was implemented using the Quality of Service (QoS) functionality provided by minimega. The bandwidth limitation ensured that the packet flood could reliably overwhelm the communication capacity, thereby disrupting DNP3 communication between the target controller VMs and the OPC server. These artificial constraints were necessary due to the high efficiency and bandwidth capabilities of the software-defined networking technology used for VM communication, as well as the ample CPU resources available to the target VMs. In real-world systems, physical devices typically operate with limited hardware resources and constrained network bandwidth. The DoS disruption scenario was executed in three separate runs, each lasting exactly three hours (10,800 seconds). Each run consisted of three distinct phases: the first 10 minutes involved normal operations, followed by a 30-minute packet flood disruption, and concluding with 140 minutes of normal operations after the disruption. 3.3.4.3. Man-in-the-Middle scenario The Man-in-the-Middle (MitM) disruption scenario caused a loss of communication by dropping DNP3 packets exchanged between the target nanogrids and the OPC server. This was achieved through an Address Resolution Protocol (ARP) poisoning attack, which redirected traffic between the targets and the OPC server through the attacker VM. The attacker VM, running Kali Linux, utilized a custom Python script built with the advanced Scapy [22] packet manipulation library to inspect each packet and determine the appropriate action. Possible actions included forwarding the packet unmodified, dropping the packet (not forwarding), or manipulating the packet (e.g., false data injection). In this scenario, the attacker exclusively dropped DNP3 protocol packets between the targets and the OPC server. As with the baseline and DoS scenarios, the MitM disruption scenario was executed in three separate runs, each lasting exactly three hours (10,800 seconds). Each run consisted of three distinct phases: an initial 10 minutes of normal operations, followed by a 2.5-hour (9,000-second) MitM attack dropping DNP3 packets, and concluding with 20 minutes of normal operations after the disruption. Future scenarios could leverage the attack script's functionality to perform more sophisticated attacks, such as false data injection. For example, the attacker could display an incorrect battery state of charge or selectively block specific types of DNP3 packets, such as control command packets (e.g., ATS switch actuation), while allowing other packets to pass unmodified. 3.3.4.4. Limitations The OPAL-RT simulation was not restarted between runs within the same scenario due to the absence of automation for starting and stopping cases on the OPAL-RT platform, as well as time constraints. This limitation was deemed inconsequential for the current experiments, as the physical system was not impacted. However, it is expected to become significant in future scenarios where disruptions directly affect the microgrid (MG). Another challenge was the delay introduced by the Modbus connection between the SCEPTRE Provider and the OPAL-RT simulator. Polling all 198 data points over Modbus/TCP from the 28 OPAL-RT required several hundred milliseconds per cycle, and this cumulative delay became significant over the course of a 3-hour experiment run. To address this issue, a potential solution is to increase the polling rate of the Provider to multiple times per second. During post-processing, the collected data could be down-sampled to the desired rate of one data point per second. This approach would preserve the required temporal resolution while minimizing the impact of cumulative delays. The iperf3 tool [23] was initially employed to collect detailed network quality measurements for the WSCC 9-bus system. However, the packet flood DoS attack disrupted the iperf3 control connection, resulting in errors during data collection. Additionally, several technical challenges arose when integrating iperf3 into the SCORCH orchestration framework, leading to reliability issues. Consequently, the decision was made to exclude iperf3 data collection from the Emera MG scenarios. Future work will focus on resolving these technical challenges to enable the inclusion of iperf3 data in subsequent experiments. One potential solution involves running the iperf3 control connection over an alternate network interface, such as the existing management interface used by SCEPTRE for VM management. 3.4. Experiment Data Collection 3.4.1. Scenario Orchestration Scenario Orchestration (SCORCH) is a capability for conducting complex real-time emulated experiments, originally developed under the Science and Engineering of Cybersecurity by Uncertainty quantification and Rigorous Experimentation (SECURE) Grand Challenge LDRD [24]. SCORCH was utilized to automate key processes, including initiating and terminating data collection and disruption scenarios, cleaning up virtual machines (VMs) between experiment runs (e.g., deleting files and resetting network interfaces), and performing basic validation of results to ensure successful execution. Validation steps included confirming that all expected data parameters were collected, verifying their formats and correctness, and ensuring both the disruption scenario and power simulation executed successfully. For the WSCC 9-bus experiments, SCORCH also automated the starting and stopping of the RTDS simulator, ensuring no timestep overruns occurred. 29 Figure 9: Screenshot of a Scorch run for the WSCC 9-bus system with the RTDS simulator The automation provided by SCORCH was critical to the success of the experimentation due to the large number of runs required. Manual execution, resetting, and validation of runs would have been excessively time-consuming, error-prone, and could have introduced inconsistencies in command timestamps, thereby increasing noise in the data. For instance, the WSCC 9-bus experiments, which involved variable packet loss, required 132 total runs, while the Emera scenarios consisted of 9 runs with a cumulative runtime of 27 hours. 30 Figure 10: Example of a Scorch run in progress for a run of the Emera DoS scenario 3.4.2. Data post-processing with Harmonize A substantial amount of data was collected from diverse sources, as outlined above. Manual processing of this data would have been both time-consuming and prone to errors. To address this challenge, the Harmonize tool was developed to automate data post-processing, including validation. The tool aggregated outputs from various collection tools and Scorch components, such as network packet capture files (PCAP), power system measurements from Elasticsearch, VM resource utilization metrics, metadata from attack scripts (e.g., start/stop times, targets, parameters), experiment start/stop times, and experiment metadata (e.g., SCEPTRE topology, scenario details, IP addresses, hostnames, and versions of Phenix and Minimega). The collected data was collated, organized, and key values were extracted into three primary outputs: a consolidated data file in Comma-Separated Values (CSV) format (experiment_results.csv), experiment metadata in a machine-readable JSON file (experiment_results.json), and a human-readable report in HTML and Markdown formats. 31 Figure 11: Screenshot of a portion of the human-readable report in HTML format The primary output of the post-processing workflow was a CSV file named experiment_results.csv. This file consolidated network packet capture data with power system measurements, providing a unified dataset for testing and evaluating InterGraph-CPS analysis techniques. Each row in the CSV file represented a single time step in the power system. For the WSCC 9-bus experiments, the dataset contained 30 rows per second, while the Emera MG data included 1 row per second. To generate this combined dataset, Pandas [25] DataFrames were created from the network and power system data. Network-related values included Round Trip Time (RTT) measurements for initial and acknowledgment TCP packets, as well as packet counts. Power system values included voltages, currents, frequency (for the WSCC system), battery state of charge (for the Emera MG), and other metrics representing the power grid state. The two DataFrames were merged into a single DataFrame by mapping network data timestamps to the nearest corresponding timestamps in the power system measurements. Figure 12: Screenshot of part of the CSV header for the file “experiment_results.csv” Given the organizational and geographical dispersion of the team, along with varying technical backgrounds, data sharing posed significant challenges. Efforts were made to ensure datasets were manageable in size, easily transferable via file-sharing tools, and compatible with a variety of systems. The Harmonize tool addressed these challenges by creating a compressed archive in Zip format, which included the processed experiment data and checksum files with SHA256 hashes for validating the archive and its contents. Each dataset archive was accompanied by a human-readable report that detailed the specific scenario, the types of data collected, and instructions for using the 32 data. Additionally, the Pandas DataFrame corresponding to the final experiment_results.csv file, along with intermediate DataFrames generated during post-processing, were included in compressed format to facilitate further research. Figure 13: Example directory structure of results for a DoS run Future work in this area could expand the consolidated data file to include additional metrics, such as detailed measurements from iperf3, VM resource utilization (e.g., CPU, memory, network, disk), and more granular network traffic data (e.g., average protocol counts within specific time ranges). Moreover, automatically generating visualizations such as histograms, tables, and graphs within the report would enhance the immediate interpretability of the data. 33 3.4.2.1. Kibana Dashboards Figure 14: Screenshot of a portion of the Kibana dashboard for the Emera MG To facilitate data visualization and monitor the operation of the emulated system, a dashboard was developed in Kibana. This dashboard provides a comprehensive overview of the state of the Emera MG as captured by the OPAL-RT SCEPTRE provider. Key metrics displayed include voltage, current, power setpoints, battery state of charge, and ATS status for both the overall grid and individual nanogrid systems. The dashboard served as a critical tool during experiments, allowing researchers to verify system functionality and ensure operational parameters remained within acceptable ranges without direct access to the OPAL-RT interface. This capability enabled the identification and resolution of potential issues prior to conducting final validations after extended experimental runs. Additionally, the dashboard was designed to visualize grid disturbances, particularly those induced by cyberattacks targeting grid operations. Although no disruption scenarios resulting in grid disturbances were implemented due to time constraints, future work will focus on simulating such disturbances. Examples include disconnecting nanogrids to deplete their battery reserves or rapidly toggling the ATS interconnect. The Kibana dashboard will support the efficient development and testing of these scenarios by providing immediate visual feedback on grid status. 34 Figure 15: Screenshot of Kibana Dashboard for Emera MG displaying Load Voltage and Current Figure 16: Screenshot of Kibana Dashboard for Emera MG displaying Battery Voltage, Current, and State of Charge 35 Figure 17: Screenshot of Kibana Dashboard for Emera MG displaying PV and Bus Voltage, Current, and Power 3.5. Open-source Contributions The majority of the code for the SCEPTRE emulation platform, including phenix, bennu, and other tools, is published on GitHub under open-source GNU Public License Version 3 (GPLv3). The platform is used by many members of the community. As part of the work on InterGraph-CPS, many improvements were contributed to the open-source codebases. These contributions included the integration with the OPALRT simulator and numerous bug fixes and enhancements that were required to make the system work efficiently and reliably over the large number of permutations ran for InterGraph-CPS. Additionally, 7 new SCORCH components were contributed that enabled the experimental orchestration and data collection, including components for applying packet loss (quality of service), collecting PCAP data leveraging minimega APIs, collecting and processing data from the experiment, executing and collecting iperf3 measurements, and orchestration of complex cyber-physical disruption scenarios (the cyber-physical part leveraged for the WSCC 9-bus executions). The contributions made to the open-source codebase as part of the work on InterGraph-CPS have already had a significant positive impact on many Sandia projects and LDRDs and will continue to have an impact after the project ends. The work on this project significantly advanced the capabilities of SCEPTRE and contributing the work back to the community ensures those improvements live on beyond the end of the LDRD. 36 4. STRUCTURAL AND TEMPORAL ANALYSIS REAL-TIME TECHNIQUES FRAMEWORK Various structural and temporal analysis real-time techniques (START) were explored to assess cyber-physical interdependencies. These techniques and their results are presented next. In Section 5, we discuss the integration of the different START approaches and the comprehensive insight they provide for cyber-physical systems. 4.1. Background When starting to look at analysis for structural and temporal properties of cyber-physical systems, a good place to start is by examining graph structure. This can be done by constructing a Cyber- Physical Graph (CPG) of the system, where each node of the graph is a component or device in either the communications / control network (Field Devices, Routers, etc...) or in the physical system (Generator, Load, etc...). The control network and the physical system network can then be connected through a parallel composition of the two graphs, where the added links are from sensors, such as Phasor Measurement Units (PMUs), and actuators in the system. These are directed edges (this as a directed graph) that connect information flow from the physical system into the communications and control network via sensor devices (PMUs, AMI, etc.) and the actuators (generators, motors, pumps, etc.). This type of structure is more fully described in [26]. 4.2. Clustering Techniques Before examining additional ways to study graph structure for important nodes and edges, we can start by utilizing well known metrics for graph connectivity and structure from graph theory, as well as additional clustering techniques to characterize groups of nodes in the system. These methods were examined over the entire CPG to help assess graph structure and isolate important elements of the system. The methods examined were: (1) Graph Components and Condensation Graphs (2) K-Shell Clusters (3) Graph Embedding Techniques (4) Bipartite Graphs Graph embedding techniques will be fully discussed in Section 4.3, while Bipartite graphs and their relation to graph clustering are discussed in Section 4.4.2. In this section, we will briefly discuss graph components and K-Shell decomposition. This section covers material from [27]. Graph components are subgraphs of a graph that are strongly connected. The graph itself is only strongly connected if it has one component, and if there is more than one component the graph is not strongly connected. A strongly connected graph means that from any node in the system you can traverse the graph to any other node in the system. A weakly connected graph is a graph that would be strongly connected if you convert all directed edges into undirected edges. In a weakly connected graph, there may be graph components that are sinks or sources, where there is an edge that only leads in or out of that component. A condensation graph is a graph of the components and the connections between them, which is helpful for visualizing those important edges. The following figure shows a CPG of the WSCC 9-bus system with a control network, as described in this report, as well as the corresponding condensation graph. 37 Figure 18: CPG of WSCC 9-bus system, color coded for graph components. (Right) Condensation Graph of WSCC 9-bus system. From [27]. In the above figure, we see that the WSCC 9-bus system in its configuration as described in Section 3.2. In this system, there is a central main graph component that is strongly connected, with the 3 generators acting as sources that are only weakly connected. This is because this network does not incorporate communications and control back to the generators, but mainly incorporates sensor connections for situational awareness, while the generator controls are separate. The additional or removal of graph nodes or edges, such as in planning for new power system additions or in the case of N-1 contingencies, can drastically change this system structure. The use of examining graph components and condensation graphs helps to see how specific lines and components can become critical for the connectivity of the graph. More complete discussion and further examples can be found in [27]. K-Shell Decomposition [28] examines and decomposes a graph by looking the node degree for each node in the system and separating them based on that metric. Node degree is a value that counts the number of edges coming into (in-degree) or out (out-degree) of a node, if the number of edges coming in and out are the same, then that is the degree of the node. K-shell decomposition separates nodes into separate cores based on their degree. An example figure is shown below. 38 Figure 19: K-Shell Decomposition of WSCC 9-bus CPG. From [27]. The above figure color codes nodes in the CPG by their K-shell core, with red for 1-shell, orange for 2-shell, and grey for 3+. These values represent the degree of these nodes in an undirected version of the graph (as we are not separating out in-degree versus out-degree). The K-shell for each node helps to identify isolated nodes (1-shell) or nodes that are more central to the graph (higher shell value). For 1-shell nodes, the loss of a single link may disconnect the graph, while a more central component (higher shell) may be more critical for overall data flow and more access to the information flowing through the system (such as a router). More complete discussion on this topic can be found in [27]. As can be seen in this section, analyzing graph structure can assist in identifying critical nodes and edges in the system and isolate where individual could cause disconnected components of the network and loss of connectivity. This is important as a base for analyzing interdependencies in CPS and can be extended through further analysis as we go into throughout the rest of this report. 4.3. Graph Embedding Technique 4.3.1. Introduction of Graph Embedding Technique Graph embedding techniques are applied in numerous application fields including natural language processing (NLP) [29], biology [30], and financial fraud detection [31]. The technique extracts the graph structural information while reducing the high-dimensional graphic data, which could extend the graph theoretic capabilities to machine-learning approaches. The recent application of graph embedding to power systems reflects the increasingly data-driven approach to outage prediction [32], network anomaly detection [33], and node clustering [34], [35]. For example, Xiao et al. demonstrated how graph embedding techniques can facilitate anomaly detection by extracting low- dimensional information from high-dimensional graphs without significant information loss [33]. 39 Among the various graph embedding techniques, DeepWalk method can extract the information from a given graph as input. Millions of random walks are generated to extract the structure of a target vertex. By seeking the possibilities of learning interdependencies of a vertex by capturing neighborhood similarities and using skip-gram method to encode the interdependencies in a vector [36], the training result could be used as risk analysis to presume the components most likely to be attacked next (from DeepWalk) The Node2Vec method, from [29], stands out for second-order biased random walks, which equips it with notable flexibility in capturing neighborhood relationships. Additionally, the Node2Vec technique learns continuous feature representations by capturing more complex interactions between the nodes, going beyond the traditional measurements of node connectivity. The capability is illustrated in [35], where Node2Vec is applied to identify the interdependencies between the critical nodes. 4.3.2. Methodology - DeepWalk DeepWalk DeepWalk method is an unsupervised deep learning method on graphs [36]. It is a statistical model that learns probabilities from the data of randomly generated walking sequences. The estimated likelihood is utilized as an equation for the starting vertex 𝑣𝑖 can be denoted as: 𝑃𝑟( 𝑣𝑖 ∣∣ (𝑣1 , 𝑣2 , … , 𝑣𝑖−1 ) ). In order to get the latent variables, a mapping function: Φ: v ∈ 𝑉 ↦ 𝑅 |𝑉|×𝑑 is introduced, by which the interdependencies could be represented as a 𝑅 𝑑 vector. Hence, the likelihood is estimated by: Pr( 𝑣𝑖 ∣∣ Φ(𝑣1 ), Φ(𝑣2 ), … , Φ(𝑣𝑖−1 ) ) This could also yield an optimization problem: min(− log Pr( {𝑣𝑖−𝑤 , 𝑣𝑖−𝑤+1 , … , 𝑣𝑖−1 , 𝑣𝑖+1 , … , 𝑣𝑖+𝑤 } ∣∣ Φ(𝑣𝑖 ) )) Φ Skip-gram is an unsupervised deep learning method, using the target node to predict the related vertices. In this algorithm, hierarchical softmax is built to summarize the conditional probability by assigning the vertices to be the leaves of a binary tree. Given that 𝑢𝑘 ∈ 𝑉, leaves of the tree could denote as (𝑏0 , 𝑏1 , … , 𝑏⌈log|𝑉|⌉ ), where 𝑏0 is the root of the tree and 𝑏⌈log|𝑉|⌉ = 𝑢𝑘 equals to 𝑢𝑘 . The conditional probability of being attacked next transforms into searching the maximum probability of a particular path in a hierarchical tree [36]. Then the conditional probabilities for given target node could be calculated as: ⌈log|𝑉|⌉ Pr ( 𝑢𝑘 ∣∣ Φ(𝑣𝑗 ) ) = ∏ Pr ( 𝑏𝑙 ∣∣ Φ(𝑣𝑗 ) ) 𝑙=1 Based on the DeepWalk method, latent variables embedded in the interdependencies could be generated for each vertex. 40 Node2Vec-based Risk Analysis Risk is typically defined as the likelihood of a given threat, evaluated by multiplying the potential vulnerability by the impact of the resulting incidents, as outlined by NIST [37]: Risk = Likelihood × Impact. To accurately quantify the likelihood of a specific threat, it is imperative to conduct a comprehensive analysis of the vulnerabilities associated with each system component. Meanwhile, the impact scores linked to adversarial events are considered to evaluate potential consequences. The Node2Vec approach is a graph embedding technique that extracts the structure and relational information of the nodes in a continuous vector space [29]. As described in [29], the technique is capable of learning node representations based on their network roles or the communities they are part of. One of the key advantages of Node2Vec is its ability to balance between different searching strategies in its biased random walk process and be perfectly applied to a weighted bi-directional graph [29]. The success inspires the potential use in CPS. Applying this to cyber-physical power systems, the Node2Vec technique can similarly unravel complex interdependencies between various system components. It starts with the creation of random walks, simulating graph exploration. In this method, we mirrored the random walks as the access paths within the graph. The skip-gram model is then applied to predict neighboring nodes for a target node within these walks. The skip-gram technique is especially important in representative learning [29], which could generate latent variables for uncovering hidden structures in data and for representing abstract concepts that are not directly measurable. It enables researchers and data scientists to make inferences about unseen or underlying processes that affect observable phenomena. The utilization of random walks in the Node2Vec technique serves exploration of potential access pathways that an adversary might exploit or intrude between nodal communication. In the previous section, we focused on employing the Deepwalk-based method to cluster nodes based on their risk level [34]. Deepwalk technique operated under the assumption of homogeneous linkage between each node, treating each path with equal possibility regardless of its characteristics. Such representations are particularly beneficial for capturing the structural similarities and interdependencies between the nodes. Assigning exploitability scores to edges can effectively model complexity of an intruder moving from one node to another. An intruder’s potential ease or difficulty in utilizing a vulnerability to travel from one place in the network to another can be efficiently measured by assigning the exploitability scores to the edges in a network graph. Exploitability-weighted graphs can be used in more advanced analytics like access simulation, where potential access pathways are simulated to understand the most likely routes an adversary might take. In this paper, we referenced the Common Vulnerability Scoring System (CVSS) and assigned the Access Complexity (AC) value as the exploitability scores to the edges within the network [38]. AC represents the degrees of difficulty an adversary could encounter to exploit a vulnerability in the system [38]. A lower AC score indicates that it is more challenging for adversaries to exploit the component. According to [38], the edge weight values in the network graph are clarified based on the AC classes of Hard, Moderate, and Easy, along with their associated AC values: • Hard: These threats are highly complex, often requiring elevated privileges and social engineering. They are less effective against informed individuals, depend on rare configurations, and have limited opportunities due to narrow race condition windows. 41 • Moderate: These threats require specialized access conditions and target specific groups of authorized users or systems. They also require prior information gathering and exploit rare non-default configurations. • Easy: These threats are broadly applicable and require minimal expertise or specific conditions. The adversaries often target common or default configurations and wide user ranges, with easily exploitable conditions. Table 1: Access Complexity Scores provided by CVSS [38] Access Complexity Weights Access Paths Hard 0.35 Local Devices to Control Center Moderate 0.61 Within Local Devices Easy 0.71 Control Center to Local Network Overall, the importance of the graph-based CPS model lies in its integration of physical and cyber components into unified structures as well as in the incorporation of cyber topologies drawn from realistic scenarios. Additionally, by leveraging graph embedding techniques and incorporating exploitability scores based on the CVSS, our models provide a general understanding of potential adversarial pathways and system responses. Figure 20: Node2Vec-based Risk Analysis Framework The random walks are generated on the weighted CPS graph. As previous described in DeepWalk section, a random walk is defined as a sequence of steps originating from a vertex, denoted as vi . The first-order and second-order transition probabilities work in a sequence to integrate the searching algorithm in contract with the DeepWalk technique, where unbiased random walks are applied [36]. In general, random walks simulate paths that data or an intruder might take through the network. By generating random walks from various nodes, we can effectively map out the potential pathways that could be exploited in cyber threats. First and second-order transition probabilities determine the likelihood of moving from one node to another, influenced by edge weights. For instance, in a DoS scenario, intruders might exploit nodes or paths with higher transition probabilities to disrupt significant portions of the network. Analyzing these probabilities helps identify which nodes are more 42 likely to be traversed and therefore, are more critical and vulnerable. Using random walk simulations with different bias settings (p and q) can predict potential attack paths which may cause damaging consequences and inform defense strategies. Suppose we initiate a random walk on each node thousands of times. This process generates a large set of access paths. Node embeddings are formulated to symbolize the graph structure and represent the biased random walks by optimizing the embeddings to preserve neighborhood similarities. The feature learning approach extends the skip-gram model used in the Deepwalk approach, to predict the neighborhood context based on the central node [29]. The skip-gram method is a novel approach for calculating vector representations from large datasets [29], [39]. This adaptation, as utilized in approaches like Deepwalk and Node2Vec, effectively treats nodes and their sequences generated from random walks as analogous to words and sentences in text. Since all nodes are represented by the latent variables, the likelihood of entities behaving similarly or being exposed to similar risks can be accessed by examining the "proximity" of nodes in the embedding space through cosine similarity. Cosine similarity could effectively measure the similarity between two non-zero vectors and provide insights into likelihood calculation. For any two nodes u and v in the graph, their cosine similarity is calculated using their vector embeddings 𝑧u and 𝑧v , and the likelihood is estimated by: 𝑧𝑢 ⋅ 𝑧𝑣 |𝑧𝑢 | |𝑧𝑣 | The likelihood computation yields a similarity score between -1 and 1. Adversaries often exploit vulnerabilities in one part of the network to move laterally to other regions. Nodes that are similar in terms of vector embedding may be part of connected or related subsystems in the CPS, making it easier for threats to propagate from one subsystem to another. When a node’s score is close to 1, it suggests their vector embeddings in the latent space are closely aligned, indicating that they possess similar attributes or are highly involved within the network. A number close to -1 denotes significantly divergent or opposing embeddings, whereas a score close to 0 denotes no discernible resemblance. Assessing the impact score of components in CPS is another fundamental aspect of risk analysis. This process involves assessing the potential consequence of a device being compromised. In this paper, we propose a comprehensive metric to determine the impact score, considering the multiple integral roles of each device. These dimensions are Operational Importance (OI), Security and Confidentiality (SC), Maintenance and Reliability (MR), Network Influence (NI), and Exploitation Complexity (EC) level. ( IS′ = OI · SC · MR · NI · EC )The product of these dimensions represents the combined impact of these factors on the system’s overall risk or importance. Using multiplication can be seen as a way to amplify the risk assessment, where each component’s risk level exponentially increases the overall risk if all factors are high. Besides, a low score in any single aspect affects the total score. This is ideal in the CPS environment where a single point of failure is unacceptable. Equation normalizes the preliminary Impact Score (IS’) to produce the final Impact Score (IS) by applying min-max normalization. The min-max normalization technique ensures that the impact score is a proportion of the maximum possible score, keeping it within a normalized range [40]. The normalization is scaled relative to the maximum possible score, keeping the result within (0, 1), and facilitating a standardized assessment across different components and systems. 43 OI ⋅ SC ⋅ MR ⋅ NI ⋅ AC Impact Score = ∑5𝑖=1 all_score𝑖 Table 2 demonstrates a structured framework in assessing impact sore by various categories. Each category is defined at three levels: High, Medium, and Low with corresponding scores of 0.9, 0.6, and 0.3 respectively. Selecting 0.3, 0.6, and 0.9 divides the full score range into three distinct intervals, ensuring a clear and structured gradation within the total score of 1. Table 2: Assessment of Impact Score: Categorization and Grading Levels Overall category High (0.9) Medium (0.6) Low (0.3) Critical Minor Operational Importance Standard Operations Operations Operations Security and Highly Public Restricted Access Confidentiality Confidentiality Information Maintenance and Frequent Regular Minimal Reliability Maintenance Maintenance Maintenance Top 25 nodes of Bottom 25 Network Influence Middle of the BC BC nodes of BC Exploitation Complexity Easy to Access Average Difficulty Hard to Access • Operational Importance: OI scores devices based on their significance in network operations and integration complexity. Devices at the high OI level are labeled as critical to operations. The failure of such a device could lead to significant disruptions in network operations or trigger cascading failures. The medium OI level indicates devices that are involved in regular operations, where their loss is manageable. Devices at the low OI level are considered less critical within the system. • Security and Confidentiality: SC score measures the level of the device’s role in network security and its handling of sensitive data. High SC refers to the device with highly sensitive data or components that require strict access control. "Restricted Access" stands for the components which are not public but less sensitive. Low SC includes the components that are potentially accessible to the public. • Maintenance and Reliability: The MR metric evaluates the frequency and criticality of maintenance required for a device within a CPS, emphasizing how cyber threats can exacerbate maintenance challenges and affect system reliability. For example, in the case of DoS attacks where control systems are overwhelmed with excessive requests or data, the system may slow down or crash, requiring immediate action. This could involve maintenance activities like repairs, replacements, or resetting parameters, particularly for cyber components. This could involve deploying security patches, updating software, and re-configuring network settings to prevent future damage. These actions would contribute to a higher MR score. "Frequent Maintenance" indicates the necessity for regular and critical maintenance activities. Devices at this level are at high risk since OT or IT disruptions can lead to increased and unscheduled 44 maintenance needs. Medium level of MR score refers to standard maintenance at routine intervals. While devices at this level may still be susceptible to cyber attacks, their impact on maintenance needs is typically less severe compared to those requiring frequent maintenance. While minimal maintenance suggests infrequent maintenance requirements. • Network Influence: Betweenness centrality (BC) is used to indicate the nodal network influence. It measures the frequency at which a node appears on the shortest paths. Nodes with high BC imply critical communication paths and information flow paths, acting as vital bridges or junctions. We defined the top 25 nodes of BC scores as the high-level network influence. A low NI score is denoted as the bottom 25 nodes with the least influence on network dynamics. Medium level of NI is the modes in between, referring to the components with moderate network centrality. • Exploitation Complexity: This score varies based on the difficulty of accessing components. A high value of the EC score suggests that the component is easily accessible. A low-value EC score suggests that the component is secure and difficult to breach because of extensive security or sophisticated design, whereas a medium EC score implies a more balanced accessibility challenge. 4.3.3. Results DeepWalk Risk Assessment of N-1 Disturbance Scenarios We assumed DoS occurred to physical components and cyber components. One component whether physical or cyber is in a state of failure which gives us an N-1 state for the system. We assess the risks of the structure and guess the most probable location of the next attack. Several scenarios are assumed as followed: • DoS of physical components: generator 1 outage and branch 6-9 • DoS of cyber components: substation C router 3 • Cyber-physical disturbance: a combination of physical disturbance and cyber disturbance During the malicious DoS attack, one device is targeted and then the attack propagates through the system. In this paper, we are interested in knowing whether a DeepWalk-based detection method could explore the interdependencies of each community. We are seeking the adversary’s next step during the disturbance scenarios assuming that the adversary will commonly target one component and keep the attack proliferating. In this case study, we assume several N-1 disturbance scenarios including physical and cyber components. The probability of each node being attacked next. We ranked the eight components which are most likely to be attacked next in the event of the N-1 disturbance. After training the skip-gram model, the result is shown in Table 3. Table 3: Components with High Risks During Disturbance Components facing DoS High Risks Components Related G1 {B1,B4,L5,R15-R20,SubB2 SW} Physical Components B6 {B3,G2,B3,R9,R11,R12-R15} 45 B7 {B2,G3,L8,R1-R4,SubA1 SW} B8 {G2,B3,L8,R3,R7,R8,R10,R11} B9 {G2,B3,L8,R7,R8,R10,R11,SubC2 SW} Cyber Components Router 3 {r0-r2,SubA_FW-SubC FW,CC_SW,CC_HMI} Table 3 reveals the devices at a high-risk level related to the components attacked already. Suppose Generator 1(G1) is facing the DoS attack, the result shows Bus 1 (B1), Bus 4 (B4), Load 5 (L5), Relay 15 to 20 (R15 − 20), Substation B’s switch (SubB2 SW ) are the components most likely to be the next target. In comparing this to the original network the result with the physical and cyber network is shown in previous section. G1, B1, and B4 are connected with G1; R15 − R20 are protecting the buses and lines within substation B; SubB2 SW is the Ethernet switch to monitor and transfer data for G1 and related buses. By inspection, all the components in high risks are highly associated with G1. We then compared the rest of the circumstances, all the results are in consistent with the cyber- physical network. This is an important finding in risk mitigation. During the disturbance scenario, DoS of one component, physical or cyber, could result in failure of other pinpoints with high risks and cause severe problem. If the system operators could estimate which component is most likely to be attacked next, we could start an action and mitigate the disturbance. Node2Vec Risk Analysis We assume that DoS attacks can occur on any physical and cyber component in the CPS network and can be recognized. As a result of a malicious attack, we assume that one component (either physical or network) is in a state of failure. Consequently, we are interested in predicting the subsequent target of the adversary’s intrusion. Suppose we are dealing with a DoS situation and the target has been identified. The two scenarios are as follows: • Scenario 1 – DoS attack on r2: This specifies an attempt to make the router, a critical network component, unavailable to its intended users by overwhelming it with a flood of traffic or requests, thereby disrupting services. • Scenario 2 – DoS attack on Substation A’s SW1 (SubA1_SW): This indicates a similar attack targeting a key piece of network hardware within Substation A, which could potentially disrupt internal communications and operations. 46 Table 4: Risk Assessment Result for High-Risk Nodes with ‘r2‘ as Target for the WSCC 9-Bus Cyber-Physical System 1. r2 as Target: Nodes SubC_FW and SubB_FW exhibit the highest cosine similarity scores (greater than 0.9), suggesting they are highly susceptible to being targeted. Conversely, the lowest cosine similarity score is for SubA0_SW, indicating it has a lower probability of being targeted shortly. Notably, six nodes’ (CC_SW,SubB0_SW, SubC0_SW, SubC2_SW, SubA0_SW) impact scores are higher than 0.8, representing the significant potential consequences of compromised nodes. SubC_FW and SubB1_HMI, which have a lower score of around 0.4, are still considered high-risk levels due to their high likelihood of being targeted. In this circumstance, the r2 mostly affected cyber components like CC_SW, SubB0_SW, SubC0_SW, etc. These components showed a significant susceptibility due to their high cosine similarity scores, indicating that they share characteristics common to high-risk profiles and impact scores within the network simultaneously. For example, CC_SW with a cosine similarity of 0.875 and 1.0 impact score, indicates a significant vulnerability to compromise in the event of an attack on r2. Table 5: Risk Assessment Result for High-Risk Nodes with ‘SubA1_SW‘ as Target for the WSCC 9- Bus Cyber-Physical System 2. SubA1_SW as Target: Higher cosine similarity scores in the table (nodes SubA0_SW, R3, R2, R4, and SubA_HMI) suggest that the characteristics of these nodes are closely associated with high risk and are therefore more likely to be targeted by adversaries. As for the impact scores, nodes like SubA0_SW and CC_SW have an impact score greater than 0.8, suggesting that a security incident at 47 these nodes could have the most severe consequences for the network. R2 has an exceptionally high cosine similarity score of 0.951 but a moderate impact score of 0.569, yielding a risk score of 0.541. The high cosine similarity suggests R2’s characteristics make it highly susceptible to threats, but its impact on the system is less severe compared to nodes like SubA0_SW. However, CC_SW has lower risk scores (0.463), due to low cosine similarity despite CC_SW’s high impact score of 1.0. The nodes SubA0_SW, R3, and SubA2_SW have the highest risk scores, indicating not only are they likely targets, but an attack on them would also result in a significant impact. In this case, the cyber threat to SubA1_SW extends beyond the cyber network, affecting physical infrastructure and interconnected cyber-physical devices. For instance, physical components (B5,B7) and cyber-physical devices (like R3) are listed as high risk vertices, which could pose a significant risk to overall operational integrity. 4.4. Bio-Inspired Cyber-Physical Interdependency Network Analysis 4.4.1. Bio-inspired Design Bio-inspired systems design is one way the problems of sustainability and resilience are addressed in engineering design. Biological ecosystems are of interest because they represent systems that integrate the need of individual species to survive with the need for the overall ecosystem to survive. The hypothesis of ecosystem-inspired systems design is that those ecosystems in existence now represent “successful” sustainable and resilient designs [41], [42], [43]. Food webs (FW), or predator-prey models of ecosystems, have been found to have system-level characteristics that may be desirable to engineers seeking a system that can both thrive when conditions are good and survive when conditions are bad [42], [44], [45], [46]. FW and other ecological techniques rely heavily on graph theory to construct desired models of the network in order to analyze them later on via ecological metrics. Pollination networks are a type mutualistic ecosystem analyzed using bipartite networks and includes, neural networks, transportation centers, and these serve as suitable models for studying the interdependencies in complex systems [47], [48]. Mutualistic networks like plant-pollinator networks have attracted attention due to their resistance to disturbances and mutually beneficial interactions [49], [50]. Mutualistic networks involve the cooperation and exchange of goods or services between various actors [47], [49], [51], [52], [53]. This interaction in nature has formed through self- organization, and new insight suggests that the network structure can be interpreted as the consequence of abundance matching [48], [51], [54], [55]. This abundance matching can be directly applied to cyber-physical systems organized by engineering planning but lacking the self- organization found in ecology. For each of these ecological networks quantitative metrics have been developed to help analyze a systems behavior, stability, resilience, robustness, and sustainability (Table 6). 48 Table 6: Common Bio-inspired systems design metrics and methods Method Metrics Used Requirements Ecological References, Papers, References Projects made in this work Directed and Pr, Ps, Binary, 1 or 0 [56], [57], START (Section Error! Unweighted Vulnerability, connections [58], [59] Reference source not Ecological Generalization, found.), Emera (Section Network Cyclicity 4.4.3.1.2) Analysis Directed and TSTp, AMI, H, Continuous [44], [45], [60], [61] Weighted DoSO, Reco variables, matrix [58] Ecological inputs, outputs, Network dissipations, Analysis actor to actor flows Bipartite and NODF, Qb, Two unique [62], [63], [27], [34], [60], [65], Tripartite Connectedness, groups of actors, [64] Emera (Section 4.4.2.5) Networks node extinctions, 1 or 0 connection system robustness The following sections consist of work taken from the following published works. Section 4.4.2.1 contains content from [34], 4.4.2.2 contains content from [27], 4.4.2.3 contains content from [60], 4.4.2.4 contains content from [65], 4.4.3.2.1 contains content from [66], lastly 4.4.3.2.2 contains content from [61]. 4.4.2. Bipartite and Tripartite Networks 4.4.2.1. Bio-inspired and AI DeepWalk Based Approach to Understand Cyber-Physical Interdependencies of Power Grid Infrastructure* *This section contains content from [34]. 4.4.2.1.1. Background and Problem Formulation Numerous structural imbalances can present challenges to power system security [67]. As [68] illustrated, the power outage caused by a cyber-attack on the Ukrainian power grid was created by an adversary placing malware in the communication system and opening the target circuit breakers. DoS can result in a component failing to provide service, causing an unscheduled power outage resulting from communication disruptions [67]. Malware attacks could cause configuration problems, communication devices impacted, sending hazardous remote control, etc. [68]. Prior applied Ecological Network Analysis (ENA) shows the normal operation of the WSCC 9-bus power system’s synthetic cyber network presents a less cyclical nature than food webs [69], [70]. A look at the ratio of specialized consumers to producers, known as the specialized predator ratio, is seen to be high for the power network, indicating that power system components have a high dependency on each other and there is less redundancy in the overall network [70]. In this paper, we continued 49 working on disturbances scenarios of cyber-physical systems to aid in nodal clustering methods to understand the characterization of CPS and how information flows within and between different cluster [70], [71]. An augmented WSCC 9-bus system is used for the case study, where the physical system is composed of 3 generators (G), 9 buses (B), and 3 loads (L) [72]. Based on the structural data, we have separated the infrastructure into three individual substations. Figure 21 shows the one- line diagram for the WSCC 9-bus physical system; for better illustration, different substations are color coded. Figure 21: Colored, one-line Diagram for WSCC 9-bus Physical System by 3 Areas. Figure from [27]. As indicated in Figure 22, a cyber system is generated based on 3 different substations, comprised of relays (R), Ethernet Switches (ES), human-machine interfaces (HMI), routers (r), and firewalls (FW). The cyber system monitors and transfers real-time data and sends commands that control the physical components. Relays, as defined as cyber-physical components, become the linkage between the physical and cyber networks of the WSCC 9-bus case study. 50 Figure 22: Cyber Network for WSCC 9-bus Physical System by 3 Areas. Figure from [27]. A fully functioning network analysis is investigated here using bipartite network models, structural information on cyber-physical connections help formulate answers on critical nodes and community groupings. Two types of visualizations are used: • Bipartite network visualization: module formation and out-of-node groupings from a fully functioning network • Quantitative information: nestedness, connectance, and modularity regarding a fully functioning network 4.4.2.1.2. Methodology 4.4.2.1.2.1. Ecological Bipartite Modeling and Analysis Previous bio-inspired design work using ENA explored graph visualization and matrix-based depictions to capture interactions between actors in a network [73]. The NxN structural matrix, where N is the number of actors or nodes inside your system boundaries, creates quantitative characteristics from the network. The model uses graph-theory-based methods to quantify the characteristics of the system. Using a bipartite network depiction (Figure 23) enables the use of a modularity analysis to understand the interaction modules in the network. The modularity level of a bipartite network is limited by the total number of interactions possible vs. actualized in a network. Connectance (C, (4.2)) is thus important to a modularity analysis and measures a system’s complexity. Generally, the lower the connectance the higher the modularity. Connectance alone cannot determine network stability, but it does help us understand the modularity of the network since it controls the level of modularity achievable. 51 Figure 23: Bipartite Network shown as two groups of actors “D” indicating digital components and “P” indicating physical components. Figure from [60]. A bipartite interaction matrix is constructed and ecology metrics are analyzed for the WSCC 9-Bus grid case study. Modularity (Qb, (4.1)) calculates the overall network modularity, where E is the total number of interactions in the network. 𝐸𝑖𝑗 is the matrix entries where one represents an edge or zero for none [74], [75]. Where gi and hj are the module indices of the nodes i and j, and ki and dj represent the degrees of i and j. Delta (δ) parses the module indices for pairings between actor groups. In this case being cyber and physical power grid components and then assigns a value of one if they are in the same module and a value of zero if they are in two different modules. This process is recursive and carried out initially by dividing the network into two modules and calculating the modularity. This continues until the network has reached maximum modularity [76], [77]. 1 𝑘𝑖 𝛿𝑗 𝑄𝑏 = ∑ (𝐸𝑖𝑗 − ) 𝛿(𝑔𝑖 , ℎ𝑗 ) (4.1) 𝐸 𝐿 𝑖𝑗 Connectance indicates there are identifiable bounds that depend on the network size of N rows and N columns [75], [78], [79]. Here L is the sum of all the edges or links, and N2 represents the total number of possible connections. A connectance of one means that all possible interactions are occurring, and every element is connected to each other. A connectance of zero indicates there are no interactions in the network. 𝐿 𝐶= (4.2) 𝑁2 Nestedness is another ecology metric prominent in plant pollinator studies. This metric helps to investigate two groups of modules interacting across their respective modules [80], [81]. Further investigations of this metric include studying the stability of bipartite networks to disturbances and analyzing failure rates of large-scale industrial networks [82]. Nestedness is a quantity detailing the structural hierarchy of the overall network [78], [83]. Highly nested networks observed by ecologists, calculated by Nestedness based on Overlap and Decreasing Fill (NODF, (4.4)), help to understand the resilience to disturbances [84]. The overall nestedness ranges from values of 0 to 100 or from 0 to 1 if the values have been normalized. Higher values indicate more nested networks similar to nature [85]. NODF continually compares the existing column value during the calculation to other column values (4). When the decreasing fill condition becomes met, either the first or second pair of 52 interactions are less than the other. When this occurs the lower pair of interactions are represented by a zero in the calculation. Analyzing a network’s nestedness is valuable because it creates more information about the interactions of that network in a diverse environment. 0 𝑖𝑓 𝑐 ≤ 𝑘𝑗 𝑀𝑖𝑗 = { 𝑛 𝑖𝑗 (4.3) min (𝑘𝑖 , 𝑘𝑗 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 ∑𝑖𝑗 𝑀𝑖𝑗 𝑟𝑜𝑤 + ∑𝑖𝑗 𝑀𝑖𝑗 𝑐𝑜𝑙𝑢𝑚𝑛 𝑁𝑂𝐷𝐹 = (4.4) 𝑚(𝑚 − 1) 𝑛(𝑛 − 1) + 2 2 In (4.3), ki is the sum of the row or column i, likewise, kj is the sum of the row or column j. Meanwhile,𝑛(𝑖𝑗) is the total number of entries matching between the two viewed values, and c is the number of entries that are a value of 1 for all values in kj. In (4.4), NODF is the normalized value for the matrix to compare various matrix sizes together and produces the final NODF value from 0 to 1. Bipartite models and a modularity analysis can provide quantitative values of the interactions between CPS regarding the reliability, serviceability, and sustainability of the network. 4.4.2.1.3. Results and Discussion 4.4.2.1.3.1. Ecology Metrics Modularity and connectance of the cyber-physical system, coupled with contingency analyses, provide critical information about the design of cyber-physical interconnections and a grid’s resilience. Table 7 shows the analysis results. Table 7: Bipartite Modularity Analysis Results Metric Bipartite Network Value Connectance 0.034 Nestedness 0.002 Modularity 0.890 The bipartite results indicate nodal groupings through defined ecological metrics. The bipartite model has a low connectance of 0.034, indicating that preliminary cyber-physical connections have a very low percentage in the overall network. A low nestedness value (0.002) also indicates a short range/lower ability to create inter-node influence within the network. This could be an effect of the low diversification between cyber-physical connection types and appearances. Lastly, modularity (0.890) indicates that the power grid structure is heavily influenced by the original physical network 53 due to distinct, densely connected groupings of subsystems. In this case, the subsystems can be equated to power grid substations or micro-grids. Figure 24 helps to visualize the nature of the bipartite connections. Figure 24: Bipartite Cyber-physical Model, with Different Colors Indicating Each Module. Figure from [34]. The colors represent the defined modules, black lines indicate out-of-module connections, and no lines mean there are no interconnections captured for those nodes. Figure 24 for example shows a module as B7, R2, R3, and R4, while B2 has an inter-module connection with R1 and an out-of- module connection with R13. 4.4.2.2. Leveraging graph clustering techniques for cyber‐physical system analysis to enhance disturbance characterization* This section contains content from [27]. 4.4.2.2.1. Cyber-Physical Emulation Experiment Design 4.4.2.2.1.1. Cyber-physical disturbance scenarios To compare between the different techniques studied in this work, we examined how different modifications to the cyber and physical graph connections affect the results gathered. As each technique is used to examine different aspects of the graph structure, this can result in differences in 54 the intuition gained about the system. The scenarios that will be studied involve changes in the CPS graph structure used for clustering analysis. They are as follows: 1. Generator 1 and lines 6-9 outage 2. DoS against load 5 and 6 relays (R13 and R20) 3. DoS + line and generator outage 4. Double line outage: line 7–5, line 6–4 5. DoS against control center switch (cc_sw) and substation C switch SubC1_SW 6. Double line outage: line 7–5, line 6–4 and DoS against control center switch (cc_sw) and substation C switch SubC1_SW In Scenario 1, generator 1 and lines 6–9 are disconnected, requiring load shedding to prevent over- current faults. Scenario 2 looks instead at a cyber-attack on the control network that is aimed at preventing the relays at loads 5 and 6 from responding to any load shedding commands. Scenario three is the combination of both. These scenarios are represented as modifications to the CPS graph through node or edge deletion. In Scenario 4, a double line outage occurs and no mitigation is deployed. Scenario 5 studies a distributed DoS (DDoS) at two locations and Scenario 6 studies the DDoS in conjunction with the double line outage. 4.4.2.2.1.2. Bipartite graphs Mutualistic networks in nature, such as plant-pollinator or plant-herbivore models, have been found to be resilient to unexpected disturbances [49], [52]. Ecologists have attributed this to their unique hierarchical network structure, which also supports highly specialized actors [86]. Ecological analyses of mutualistic networks use bipartite graphs and clustering analyses to model and quantify interaction structures as related to network function and to allocate conservation efforts amongst species [52]. A bipartite model groups actors into two sets and highlights interactions between the two groups. A modularity analysis is then used to identify generalist and specialist species, interaction clusters or modules between species, and general interaction distribution patterns across all species. These modules may be thought of as like the clusters identified in the other techniques investigated here. Modularity (Qb) quantifies the level to which components are clustered into modules based on interactions. The Newman algorithm [76], [87], produces a value from zero to one for Qb (4.1) where eii is the percentage of edges in module i, and ai is the percentage of edges with at least one end in module i, with zero indicating no clustering or modules based on interaction patterns and one indicating clearly defined clusters with no interactions crossing clusters. A modularity analysis also produces a visual depiction of the clusters, using colors to highlight within module interactions. The level of modularity that can be achieved is controlled somewhat by the overall connectance (4.2) of the interface, or the ratio of actual interactions or edges (L) to total possible edges in the graph. A higher connectance (closer to one) limits the level of modularity that can be achieved while a lower level (closer to zero) makes modularity more likely but not guaranteed. A bipartite network graph is relevant in cyber-physical systems as it uniquely focuses on the interface between the cyber and physical components, helping to identify any patterns that are impacting performance. 1 𝑘𝑖 𝛿𝑗 𝑄𝑏 = ∑ (𝐸𝑖𝑗 − ) 𝛿(𝑔𝑖 , ℎ𝑗 ) (4.5) 𝐸 𝐿 𝑖𝑗 55 𝐿 (4.6) 𝐶= 𝑁2 4.4.2.2.1.3. Results from WSCC 9-bus emulation scenarios This work builds on that of [34] to examine how understanding modules describing the cyber- physical (CP) interface can inform our understanding of disturbance (removal of nodes and edges) propagation between cyber and physical parts of the network. As the entire CP power system is highly complex, understanding what impact the interface design has on the overall network response is highly beneficial. The modularity analysis finds that the WSCC 9-Bus CPS interface is made up of 12 modules, seen in Figure 25 and detailed in Table 8. Three modules contain one of 3 buses (B7, B9, and B4), each having a dedicated set of 3 relays. Three modules have a set of 2 relays per bus (B8, B6, and B5) and the remaining 6 modules are made up of one bus or load connected to one relay. The modularity for the original WSCC 9-Bus CPS and disturbance scenario 1 is Qb = 0.898. This is a high modularity (very close to one), a result of the low connectance (C = 0.083) at the interface. Mutualistic networks in nature have been found to have low levels of modularity, prompting the question: What would a biologically inspired CPS interface look like and how would it perform - both under normal circumstances and during disturbances? Biological ecosystems have been found to be characteristic of high levels of performance in both of these cases, most likely a result of long periods of reformation to be able to both grow and develop as well as survive [46]. Figure 25: Modules found within the cyber-physical interface of WSCC 9-Bus case study for (a) the original, Scenario 1, and Scenarios 4–6; and (b) for Scenarios 2 and 3. There are no connections at the interface that fall outside of a module. Figure from [27]. Table 8: Modules for WSCC 9-bus CPS system. Module Physical components Cyber components 1 B7 R2, R3, R4 56 Module Physical components Cyber components 2 B9 R8, R9, R10 3 B4 R15, R16, R18 4 B8 R5, R6 5 B6 R12, R14 6 B5 R19, R21 7 B2 R1 8 L8 R7 9 B3 R11 10 L6 R13 11 B1 R17 12 L5 R20 • Note: During disturbance scenarios 2 and 3, modules 10 and 12 are taken out, leaving 10 of the original 12 modules remaining. • Abbreviations: B, bus; L, load; R, relay. The first disturbance scenario does not directly impact the interface at all and leaves the 12 modules intact. The second disturbance scenario takes out two of the 1:1 modules: load 6 to relay 13 (module 10 in Table 8) and load 5 to relay 20 (module 12 in Table 8). The third disturbance scenario, which combines both the generator and line outage with a DoS, results in the same 10 modules as the second scenario—losing modules 10 and 12. The fourth, fifth, and sixth disturbance scenarios investigated here (a double line outage of B7-B5 and B6-B4, a DoS against the control and SubC1 switches, and the combination of the two) do not directly impact the cyber-physical interface and therefore maintain all 12 original modules listed in Table 8. The modularity for the second and third disturbances, the only two that directly impacted the interface, drops slightly to Qb = 0.881, down from 0.898. This is still an extremely high modularity (very close to one), that is mostly a result of the low levels of connectance at the interface (0.1 for the second and third disturbance). It should be noted that the disturbance scenarios being tested here are not full disturbance propagation scenarios and therefore we are unable to truly see the impact on the interface of cyber disturbances propagating across the interface to impact the physical network components (which would further impact the modules at the interface) and physical disturbances propagating across the interface to impact the cyber network components. Future work will expand to include these simulations. 4.4.2.2.2. Comparison of Graph Clustering Techniques Bipartite network graphs help to identify patterns arising due to the cyber-physical interface connections. A modularity analysis can aid in identifying interaction patterns and level of clustering (termed modules) to enable tying those to network functioning. The number of original modules identified in the fully functioning CPS network here decrease as a result of disturbance scenarios 2 and 3, which are the only ones that directly impact components at the CP interface. The utility of 57 this method for screening whether disturbances have (or do not have) a cyber-physical (cross-layer) property can be seen directly here. Additional simulations are needed to use the approach to understand disturbance propagation across in the interface. The highly modular findings for the interface of the WSCC 9-bus case study suggests that an interesting future optimization problem could be to design the modularity of the interface closer to what is seen in mutualistic biological ecosystems, making a less modular and more hierarchical interface to increase interface robustness. The results here prompt the further exploration of modularity and interface design, including its quantification in other larger and realistic power systems configurations. We do also see some overlap with the clusters found by the DeepWalk method, for example, module 1, made up of B7, R2, R3, and R4, overlaps are contained within cluster three of the DeepWalk. Similarly, module 9, made up of B3 and R7, is contained within cluster 4 and modules 1 and 3 are both contained within cluster 1. There is less agreement however between the modules and the k-shells, for example, modules 3 and 9 do not line up with the k-shells but module 11 does. That there is some agreement between the clustering techniques is promising to the combined use of multiple techniques to understand different facets of these highly complex cyber-physical networks. 4.4.2.3. Analyzing Cyber-Physical Modularity and Interdependence Using Bio-Inspired Graph Modeling* *This section contains content from [60]. 4.4.2.3.1. Introduction Current and future power grids are created with physical and cyber components creating unique interactions and interdependencies [88], [89]. Interdependencies can be assessed and made a possible target point by adversaries [67]. Recent attacks in Ukraine (2015, 2019, 2023) and in medical facilities have prompted concerns regarding secure cyber-physical connections [68], [90], [91], [92]. Additionally, new concerns are emerging regarding functionality under natural and man-made disasters prompting research and engagement on resilience and defense [93], [94]. A methodology to characterize the Cyber-Physical Interdependencies (CPI) can assist in modeling various error and attack propagation techniques in Cyber-Physical Systems (CPS) [95]. Literature has heavily characterized diverse techniques available to adversaries for CPS [96], [97], [98], [99]. Some works go as far as to address specific improvements in supervisory control and data acquisition (SCADA) systems, and others break down popular attack patterns. Recent and formidable attack patterns include denial of service (DoS) attacks, deception attacks, and replay attacks. Mahmoud et al. gives insight into these specific attacks, highlights the need for an advanced control system, and detail various sequential methodologies for modeling CPS. This includes triggering strategies, game theory, and stochastic secure control approaches [100]. Such analysis and methods provide valuable insights into the propagation and likely flow of information involved with CPS; however, this does not necessarily capture network functionality and interdependence across close devices that were not directly at risk. A method to develop CPI while modeling the entire architecture in retrospect to specific cyber-physical connections can provide further insight into reliance on specific devices, and preferable reliance on cyber or physical systems under disturbance scenarios. 4.4.2.3.2. Introduction of Biological Systems Modeling Previous work has proposed various techniques for molding CPS interactions including Ecological Network Analysis (ENA), bipartite networks (BN) [69], [101], [102], [103]. Bio-inspired network 58 design has emerged as a promising approach for achieving traditional resilience goals in various domains, including power grids, engineering makerspaces, water distribution networks, and industrial resource networks [74], [104]. This innovative approach draws inspiration from ecological modeling and analysis techniques to understand the role of cyber-physical interdependencies in the resilience of complex systems. 4.4.2.3.3. Partite Modeling Methods Techniques developed by ecologists assist in studying complex and multi-actor interactions between diverse species [105], [106]. ENA contains functional metrics that can evaluate structural and flow type characteristics of ecological networks [57], [62], [107], [108], [109], [110], [111], [112], [113], [114]. These characteristics have been used for various control dependent, power dependent, and system systems technology regarding the resilience, and robustness of power grids under disturbances [69], [73], [102], [115], [116], [117]. This technique has been widely used in ecology for conservation efforts and is now being applied to understand the resilience of cyber-physical systems. Bipartite networks, such as neural networks, transportation centers, and mutualistic ecosystems like plant-pollinator networks, serve as suitable models for studying the interdependencies in complex systems [47], [48]. Mutualistic networks like plant-pollinator networks have attracted attention due to their resistance to disturbances and mutually beneficial interactions [49], [50]. This study focuses on the previous works by diving further into partite modeling, which requires understanding mutualistic and antagonistic networks [34]. Mutualistic networks involve the cooperation and exchange of goods or services between various actors [47], [49], [51], [52], [53]. This interaction in nature has formed through self-organization, and new insight suggests that the network structure can be interpreted as the consequence of abundance matching [48], [51], [54], [55]. This abundance matching can be directly applied to cyber-physical systems organized by engineering planning but lacking the self-organization found in ecology. By applying ideas found in different ecological networks including mutualistic and antagonistic networks, this paper utilizes bipartite, and tripartite modeling for the identification and analysis of cyber-physical modularity, nestedness, and community identification in the IEEE WSCC 9-bus and ACTIV200-bus case studies [72], [118], [119]. Additionally, the creation of disturbance scenarios by work at Sandia National Laboratories assists in the vulnerability and cyber characterization of the system. The creation of the ACTIV 200-bus was utilized in the previous work by Sun et.al. [119]. 4.4.2.3.4. Project Goals and Objectives This work expands upon current bio-inspired modeling techniques specifically analyzing cyber- physical, physical-physical, and cyber-cyber connections. By identifying these interactions and their subsequent impacts on nearby at-risk neighbors we can further map the CPS space surrounding power grids. This paper’s contributions include: 1. Introducing a bipartite network analysis to map all system interactions at full network strength and disturbance scenarios developed with Sandia National Laboratories. 2. Presenting a tripartite network analysis to identify novel keystone devices across multiple CPS types. 3. Developing a quantitative approach to measure attack tolerance of a partite network. 4. Expanding these analyses to an ACTIV 200-bus case study. 59 The paper proceeds as follows: 4.4.2.3.5 provides an overview of the methodology of the case studies considered and the details the extent of our methods, followed by the discussion of results in Section 4, and finally the conclusions in 4.4.2.3.6. 4.4.2.3.5. Methods Power system security can face hurdles due to various structural imbalances [67], as demonstrated by [68], when a power outage resulted from an adversary implanting malware in the communication system and opening target circuit breakers. Communication disruptions caused by DoS attacks can provoke unscheduled power outages, interfering with a component’s ability to provide service [67]. Malware assaults can lead to configuration issues, compromised communication devices, and the transmission of hazardous remote control commands, among other problems [68]. Previous research employing network decomposition techniques indicated that the WSCC 9-bus power system’s synthetic cyber network behaves less cyclically during normal operation than food webs, mirroring its non-cyclical real power flows [70]. Examining the specialized predator ratio– the proportion of specialized consumers to producers– reveals a high value for the power network, signifying a strong interdependence between power system components and reduced redundancy in the broader network [70]. In this paper, we expand upon disturbance scenarios within cyber-physical systems, postulating that prior ENA methods including partite modeling, could enhance our understanding of CPS characteristics and information flow dynamics within and between different clusters [71], [94]. Previously, we utilized an augmented WSCC 9-bus system for our case study, comprising 3 generators, 9 buses, and 3 loads in its physical make-up [72]. We divided the infrastructure into three substations based on structural data following previous work from [34]. Figure 26: A color-coded one-line diagram of the WSCC 9-bus physical system, distinguishing different substations for better visualization. Figure from [60]. As shown in Figure 26, a cyber system is created based on three separate substations and consists of relays (R), Ethernet switches (SW), human-machine interfaces (HMI), routers (r), and firewalls (FW). 60 The cyber system is responsible for monitoring and transmitting real-time data, linked to the control of physical components. Relays are the essential cyber-physical component linking physical and cyber networks within the scope of our WSCC 9-bus and ACTIV 200-bus case studies. 4.4.2.3.5.1. Bipartite Networks and Ecological Analysis One key technique employed in this field is modularity analysis, which involves analyzing bipartite network models to identify actor groupings based on their interactions. Modularity analysis can reveal critical system components including hub actors which connect different clusters in the network, and specialized actors at risk. By applying modularity analysis in conjunction with nestedness and connectedness measures, researchers can effectively model the interactions between cyber and physical components of a power system. This integrated approach provides insights into the system’s resilience, as assessed through contingency analyses, and its interconnectivity. Figure 27: A hypothetical bipartite network model highlighting the interface connections between two groups of actors: "C" for cyber components and "P" for physical. Figure from [60]. The findings from this research could inform the design and management of resilient systems in various domains. By identifying critical actors and understanding their interactions, decision-makers can develop strategies to enhance the resilience of cyber-physical systems. Moreover, the application of bio-inspired network design can lead to unconventional routes for achieving traditional resilience goals, offering new perspectives and solutions to address the challenge faced by complex systems. In conclusion, the use of modularity analysis on bipartite network models Figure 27, inspired by ecological modeling techniques, has proven to be a valuable approach to understanding the resilience of cyber-physical systems. By studying mutualistic networks like plant-pollinator networks, researchers can gain insights into the interdependencies between cyber and physical system components. This knowledge can enhance the resilience of various systems, including power grids, engineering makerspaces, water distribution networks, and industrial resource networks. Applying bio-inspired network design opens new possibilities for achieving traditional resilience goals and addressing the complexities of modern systems. 4.4.2.3.5.2. Tripartite Networks Tripartite networks are emerging in recent literature as a method to model a multi-layer network Figure 28. These networks can be found in both mutualistic (plant-pollinator and seed dispersal) and antagonistic networks (herbivore-carnivore and plant interactions) [50], [120], [121], [122], [123], [124]. 61 Figure 28: A hypothetical tripartite network model with three types of actors: "C" for digital/cyber components, "P" for physical components, and “CP” for cyber-physical components. Figure from [60]. Tripartite networks include third-party communities and system perturbations. Considering how ecological networks function can lead to better standardization of robustness and reliability associated with network communities and modules [114], [125], [126], [127], [128], [129], [130], [131]. This modeling will help to identify cascading failure types due to the possibility of the network leaning antagonistic-antagonistic, meaning when an intermediate interaction occurs linking the network it may trigger extinctions of other actors connected by modules. 4.4.2.3.5.3. Ecology Metrics Ecologists have created various metrics to standardize the quantitative meaning of resilient and robust ecological systems. Specific metrics related to community identification such as modularity and nestedness are applied. Earlier research in bio-inspired design with ENA focused on graph visualization and matrix-based illustrations to represent interactions among participants in a network [93]. The N x N structural matrix, consisting of N actors or nodes inside the system boundaries, extracts quantitative features from the network. This model employs graph-theory-driven techniques to quantify the system’s attributes. By utilizing partite network representations, modularity analysis becomes possible, allowing us to comprehend the interaction modules within the network. The modularity level of a bipartite network is constrained by the total potential interactions compared to those actualized within a network. Consequently, connectance (C, (4.8)) plays a crucial role in a modularity analysis and assesses a system’s complexity. Generally, a lower connectance forces a higher modularity. While connectance alone does not solely determine network stability, it aids in understanding the network’s modularity since it regulates the achievable level of modularity. 1 𝑘𝑖 𝛿𝑗 𝑄𝑏 = ∑ (𝐸𝑖𝑗 − ) 𝛿(𝑔𝑖 , ℎ𝑗 ) (4.7) 𝐸 𝐿 𝑖𝑗 Taking the WSCC 9-bus and 200-bus models as our target networks, we construct several partite interaction matrices and examine ecology metrics. Modularity (Qb, ()) calculates overall network modularity—E denotes the total number of interactions in the network. 𝐸𝑖𝑗 signifies matrix entries where one stands for an edge or zero for none [68], [117]. 𝐺𝑖 and ℎ𝑗 are module indices for nodes i 62 and j, while 𝑘𝑖 and 𝑑𝑗 represent their respective degrees. Delta (𝛿) evaluates module indices for pairings between actor groups—in this instance, cyber and physical power grid components—and assigns a value of one if they share a module or zero if they occupy separate modules. This recursive process begins by dividing the network into two modules and determining their modularity. This continues until maximum modularity is reached [101], [103]. 𝐿 𝐶= (4.8) 𝑁2 Connectance reveals discernible boundaries contingent on the network size of N rows and N columns [68], [74], [117], [132]. Here, L represents the sum of all edges or links, while 𝑁 2 indicates the total number of potential connections. A connectance of one implies that all possible interactions are taking place and every element is connected. A connectance of zero denotes an absence of interactions within the network. Nestedness is a widely recognized ecological metric used in plant-pollinator studies. It allows researchers to examine the interactions between two groups of modules within their respective domains [105], [106]. This metric has been further explored to study the stability of bipartite networks in response to disturbances and analyze failure rates in large-scale industrial networks [62]. Nestedness represents the structural hierarchy within a network [109], [132]. Ecologists often analyze highly nested networks using the Nestedness based on the Overlap and Decreasing Fill (NODF) approach, which calculates a numerical value ranging from 0 to 100 or 0 to 1 if normalized [107]. Higher NODF values suggest more nested networks similar to those seen in nature [108]. Throughout the calculation process, NODF continuously compares existing column values with others (4.10). Once the decreasing fill condition is met, either the first or second pair of interactions becomes less than their counterpart. In these instances, a zero replaces the lower pair of interactions in the calculating equation. Gaining insights into a network’s nestedness proves invaluable as it offers additional information on network interactions within diverse environments. The NODF formula is expressed below. 0 𝑖𝑓 𝑐 ≤ 𝑘𝑗 𝑀𝑖𝑗 = { 𝑛𝑖𝑗 (4.9) min (𝑘𝑖 , 𝑘𝑗 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 ∑𝑖𝑗 𝑀𝑖𝑗 𝑟𝑜𝑤 + ∑𝑖𝑗 𝑀𝑖𝑗 𝑐𝑜𝑙𝑢𝑚𝑛 𝑁𝑂𝐷𝐹 = (4.10) 𝑚(𝑚 − 1) 𝑛(𝑛 − 1) + 2 2 In (4.9), 𝑘𝑖 represents the sum of row or column i, while 𝑘𝑗 signifies the sum of row or column j. Additionally, 𝑛𝑖𝑗 is the total number of matching entries between two values under consideration, and c refers to all occurrences where 𝑘𝑗 possesses a value of 1. Equation (4.10) normalizes NODF for matrix comparison across different sizes and generates a final NODF value between 0 and 1. 63 Researchers can acquire quantitative values for interactions between CPS networks by examining bipartite models and conducting a modularity analysis increasing the chances for reliability, serviceability, and sustainability. Additionally, attack tolerance can be observed by utilizing the flow of equations detailed in the series of equations below. Here 𝑆𝑏𝑒𝑓𝑜𝑟𝑒 and 𝑆𝑎𝑓𝑡𝑒𝑟 represent the length or size of the network before and after an attack on the 𝑐𝑒𝑛𝑡𝑟𝑎𝑙𝑛𝑜𝑑𝑒 ultimately providing the number of nodes remaining per deletion (4.11) 𝑐𝑒𝑛𝑡𝑟𝑎𝑙𝑛𝑜𝑑𝑒 = arg max 𝑑𝑒𝑔𝑟𝑒𝑒 (v)𝑣 ∈𝐺 ⬚ (4.12) 𝐺 = 𝐺 − 𝑚𝑜𝑠𝑡 𝑐𝑒𝑛𝑡𝑟𝑎𝑙 𝑛𝑜𝑑𝑒 (4.13) (4.14) 𝑆𝑎𝑓𝑡𝑒𝑟 (4.15) 𝑇𝑎𝑡𝑡𝑎𝑐𝑘 = 𝑆𝑏𝑒𝑓𝑜𝑟𝑒 The above series of equations are repeated until there are no more connected nodes within the graph, and this process is then repeated for each disturbance scenario outlined in Section 3.2.4. This analysis aims to assess the resilience of the network by determining the number of connections that remain intact after the removal of the most highly connected nodes. In short, this could be viewed as the cascading of a virus throughout the network once the devices or elements with the highest degree are removed. Normalizing these values against the baseline scenario could also display interesting insights regarding disturbance similarity before weights are introduced into the graph in future studies. 4.4.2.3.5.4. Case Studies In this study, the WSCC 9-bus, and ACTIVS 200-bus case study models are evaluated via 2 partite modeling methods in 10 scenarios. These scenarios will apply the various methods to compare perturbations the network experiences aiming to provide deeper insight into CPS interdependencies via community and ecological modeling techniques. Some assumptions are required within this work based on the network conditions and the metrics used. All models are evaluated statically using snapshots in time before and after disturbances. Disturbances may affect specific interactions between devices or all interactions the device experiences. Below, each of the disturbance scenarios are outlined. WSCC 9-bus Disturbances: 1. Generator 1 and Bus 6 and Bus 9 outage. 2. Relay 13 and Relay 20 outage. 3. Disturbance 1 and Disturbance 2 combined. 4. Bus 7 and Bus 9 connection outage, and Bus 4 and Bus 6 connection outage. 5. Control Center and Substation C1 outage. 6. Disturbance 4 and Disturbance 5 combined. 64 Transitioning to a 200-bus case study requires scaled similar disturbances inspired by the 9-bus case study. ACTIVS 200-bus Disturbances: 1. B1, B3 connection outage. 2. B137, B4, R-L4-1, R-B4-3-1, HMI-2, router-2 outage. 3. Disturbance 1 and Disturbance 2 combination. 4. Disturbance 2 and B140, B30, SW-30, R-B30-29-1, router-12 outage. 5. SW-193, router-82, router-98, FW-5 outage. 6. G1 through G55 outage. 7. Half of all FWs, Relays, routers outage. 8. SW1 through SW60 outage. 9. Full SW outage. 10. Full FW outage. 4.4.2.3.6. Results and Discussion From this study following the methods section BN, and TN are created to assist in the identification of unseen interactions in a CPS. 4.4.2.3.6.1. WSCC 9 Bus Results Figure 29: The bipartite network for the WSCC 9-bus system under normal conditions with two groups of actors and the total count of interactions per module. Figure from [60]. Table 9: Topology evaluation results of normal, and disturbance scenarios for the WSCC 9-bus system. Graph Scenario Qb C NODF Bipartite Normal 0.898 0.083 0 - 1 0.898 0.083 0 65 Graph Scenario Qb C NODF - 2 0.891 0.100 0 - 3 0.881 0.100 0 - 4 0.898 0.083 0 - 5 0.898 0.083 0 - 6 0.898 0.083 0 Tripartite Normal 0.859 0.060 0.010 - 1 0.886 0.027 0.010 - 2 0.861 0.024 0.008 - 3 0.877 0.050 0.007 - 4 0.868 0.051 0.012 - 5 0.859 0.060 0.010 - 6 0.868 0.051 0.012 Figure 29 was constructed from the interactions of two adjacency matrices where the cyber-physical components are the joining column for the 9-bus system. A tripartite graph allows for a third level of graph connections to be present and connect more graphical components than the BN. Following the bio-inspired methodology the CPS are the herbivores, the cyber devices are the carnivores, and the physical devices are the plants. This relationship in ecology is typically used to model complex interactions, which perfectly suits the purpose of this study: to seek out unseen cyber-physical interactions. The extension into the cyber column (far-right in Figure 29) allows for the interactions between the cyber-physical and cyber devices to be modeled which was previously not present in the BN modeling of Figure 29. Here, intermodular connections are represented by different colors in the TN graph. 66 Figure 30: The normal tripartite network for the WSCC 9-bus system with three groups of actors and the total count of interactions per module. Figure from [60]. Using these higher order connections is one way to discover possible devices at risk because of their shared modular connection. For example, SubC1_SW does not exist in the BN graph, but in the TN graph, SubC1_SW is interconnected with R5, R6, and R7, linking it indirectly to B8 and L8 (load 8) respectively. Additionally, looking at the BN graph in Figure 30 specifically R4, there are module connections to B7 and indirect to R2 and R3. Looking into R4 on the TN graph in Figure 30 there are direct connections to B7 on the physical side and SubA2_SW on the cyber side. Following the physical connections there are initial indirect connections to R2, R3, and deeper indirect connections to SubA1_SW and R1. On the cyber side, there are initial indirect connections to SubA2_SW and deeper indirect connections with R19, R21, and R20. In this way, the TN model can capture possible attack pathways from modular and indirectly connected components. Table 10: Topology evaluation results including the number of interactions, the number of modules, and the interior to exterior ratio of components in the WSCC 9-bus system. Number of Graph Scenario Interactions Int/Ext Modules Bipartite Normal 21 12 1 - 1 21 12 1 - 2 19 10 1 - 3 19 10 1 - 4 19 10 1 - 5 21 12 1 - 6 19 10 1 Tripartite Normal 43 14 0.884 67 Number of Graph Scenario Interactions Int/Ext Modules - 1 37 13 0.946 - 2 39 12 0.923 - 3 31 13 0.939 - 4 33 14 0.909 - 5 43 14 0.884 - 6 33 14 0.909 Results regarding BN interactions, number of modules, and interior-to-exterior connections deepen the knowledge disruptions play in a cyber-physical network. Table 10 shows a decrease in interactions and number of modules during disturbances 2, 3, 4, and 6. Compared to earlier results, this indicates changes are occurring in the BN. However, it does not influence the modularity, connectance, or nestedness values except for during disturbances 2 and 3. Similarly, analyzing the results of the TN graph, we see changes reflective of all disturbance scenarios in Figure 30. As disturbances occur, the modularity was predicted to increase due to the removal of larger modules and the simultaneous formation of smaller modules. Disturbance 1 indicates the largest change in modularity from normal operation (a difference of 0.027) indicating the importance of Bus 6 and Bus 9 in the network for power and communication delivery. Disturbance 3 reports the second largest modularity increase (0.018) this scenario has an outage of Relays 13 and 20 and Bus 6 and 9. Connectance values are more descriptive for each scenario suggest that disturbance 2 is possibly the least connected at 0.024. Disturbance 1 is minimally different indicating the same effect (0.027). The network’s nestedness results also support the conclusion that disturbance 3 is possibly the most influenceable scenario due to a nestedness decrease in the network to 0.007. Understanding the rank and roles of specific modules can further provide insight into the criticality of nodes in the network. In future work, the interior to exterior ratio per node could provide further analysis of node-to-node attack pathways and cyber-physical kill chains. 68 Figure 31: Distributions of Hub, Non-Hub, and Peripheral node classes among cyber and physical devices in the WSCC 9-bus system generated from the tripartite network. Figure from [60]. Figure 31 above shows the hub, non-hub, and peripheral roles of nodes in specific device classes in the TN. Here unlike the BN, the TN displays Hub components in the Bus, Firewall, Relay, and Substation classes. These unseen hubs assist in higher level community grouping. As shown in Figure 31 the relay class hosts the largest number of non-hub actors while maintaining a small amount of hub actors. Hub actors may disappear throughout a disturbance, and the non-hub actors may become hub actors. 69 Figure 32: The plotting of node elimination for the tripartite network for the WSCC 9-bus system. Figure from [60]. Figure 32 highlights the number of runs required for the algorithm mentioned earlier to reach completion when all the nodes are removed. The runs of node elimination by node degree target the highest degree connected component present in the chosen scenario and then continually eliminates the next highest node degree. This simulation of transitions between hub, non-hub, and peripheral actors displays a strong impact specifically from disturbances 3 and 4. In terms of node elimination, the removal of multiple buses has a large effect on available inter-modular connections via the indirect pathways. In this way, a combined outage of 2 relay and 2 bus devices are comparable via node degree to a 4-bus outage. Overall, the TN and node removal process can provide deeper insights regarding CPS interactions by capturing the impacts on direct, and multiple levels of indirect interactions. In the 9-bus specifically, the TN analysis can be equally impacted by cyber and physical disturbances at the network level. When considering applied impacts, cyber disturbances specifically impacting load delivery could pose a significant problem for power delivery to a community. 4.4.2.3.6.2. ACTIV 200 Bus Results Following a similar process as the WSCC 9-bus results, BN and TN for the ACTIV 200-bus disturbance were graphed and evaluated. Here we utilize the labels and nodal relationships from this initial case study generation. 70 Figure 33: A tripartite graph model for the ACTIV 200-bus power system. Physical, cyber physical, and cyber nodes are connected via the initial node and target link structure generated from the initial graph simulation. The count color scale is representative of the frequency of connections at the physical device. Figure from [60]. A tripartite graph model for the ACTIV 200-bus power system. Physical, cyber-physical, and cyber nodes are connected via the initial node and target link structure generated from the initial graph simulation (Figure 33). The count color scale is representative of the frequency of connections for the device. The construction of bipartite and tripartite graphs for the 200-bus followed the same procedures as the WSCC 9-bus. The 200-bus BN graph contains more out-of-module connections than the 9-bus, but this is mainly attributed to scale since the 200-bus is 28x the scale of the 9-bus. Considering this, 71 the results for the 200-bus may produce more resilient results compared to the 9-bus due to size alone. The scaled disturbances (7-10) can assist in evaluating scaled disturbance scenarios. Unlike the 9-bus having a predetermined cyber topology from previous research, the 200-bus does not have an explicitly detailed topology. This is beneficial for studying future CPS models because the creation of the TN model does not rely on a drawn topology, but instead on the connections generated from node-to-node interactions. Thus, the TN is evaluated slightly differently due to the lack of knowledge regarding substation sections that house physical and cyber devices. Like the 9-bus, tracking connected indirect components can show deeper insights regarding the pathway of disturbances based on grouped communities. For example, the 200-bus BN model starting at R_B97_200_1 on the cyber-physical side is directly linked to B97 and indirectly linked to R_B60_1, R_L97_1, R_L97_2, R_L97_3 on the cyber-physical side. The second indirect connections on the physical side are B60, L_97_1, L_97_2, and L_97_3. The TN for the 200-bus allows for even more detailed direct and indirect cyber connections to be visible. R_B97_200_1 displays a cyber connection to SW_97 and links itself to secondary indirect interactions depicted in the BN model. In the TN model, they are seen as the first indirect interaction based on the connection from the cyber component SW_97. Additionally, there are more indirect connections from SW_97 because of a higher physical dependency at B200. Although simpler there are clear examples of higher-level interdependencies in B89 and SW_89. These higher-level interactions aid in recognizing possible attack pathways between three device types. Table 11: Topology evaluation results of normal, and disturbance scenarios for the ACTIV 200-bus system. Graph Scenario Qb C NODF Bipartite Normal 0.780 0.017 0.001 - 1 0.758 0.017 0.001 - 2 0.774 0.017 0.001 - 3 0.794 0.017 0.001 - 4 0.776 0.017 0.001 - 5 0.761 0.017 0.001 - 6 0.780 0.017 0.001 Tripartite Normal 0.811 0.017 0.001 - 1 0.810 0.017 0.001 - 2 0.810 0.017 0.001 - 3 0.815 0.017 0.001 - 4 0.809 0.017 0.001 72 Graph Scenario Qb C NODF - 5 0.805 0.017 0.001 - 6 0.809 0.017 0.001 - 7 0.889 0.002 0.012 - 8 0.850 0.003 0.001 - 9 0.808 0.003 0.009 - 10 0.806 0.003 0.000 Table 11 explores changes in modularity, nestedness, and connectance in original and scaled scenarios. The modularity values for the BN and TN for the 200-bus are lower for each scenario compared to the 9-bus case study. This indicates more out of module connections as seen in Figure 33. The 200-bus during normal operation has a modularity of 0.780 for the BN and 0.811 for the TN. The difference between these values (0.031) is about the same as the 9-bus model (0.039) indicating that the TN demonstrates consistent results even when scaling of case study size occurs. Similar to the 9-bus when a hypothesized higher level disturbance occurs the modularity increases. Within the scaled disturbances the most prominent by modularity is disturbance 7 within the TN (0.889). Although the BN has a lower modularity indicating less segregated modules, the TN joins some of these separated modules into the same substations seen on the very right of Figure 33. This extension allows for more realistic modularity values in the overall cyber-physical network after regrouping occurs. This specific disturbance targeted half of the existing firewalls, relays, and routers randomly as a test to see the effects on the larger network. A change in connectance also occurs in all the scaled scenarios by approximately 0.015 for scenarios 7 through 10. The connectance for the overall 200-bus network is low compared to the 9-bus network, but this was predicted because the 200-bus topology was developed as a scaled 9-bus topology. An interesting result regarding the nestedness is the increase in disturbances 7, and 9 as typically an increase in modularity indicates a decrease in nestedness. This indicates the formation of new random formation of new module groupings after a disturbance. Table 12: Topology evaluation results including the number of interactions, the number of modules, and the interior to exterior ratio of components in the ACTIV 200-bus system. Graph Scenario Interactions Number of Int/Ext Modules Bipartite Normal 1411 27 0.637 - 1 1397 43 0.566 - 2 1401 36 0.606 - 3 1388 24 0.682 - 4 1392 32 0.622 73 Graph Scenario Interactions Number of Int/Ext Modules - 5 1411 31 0.589 - 6 1392 32 0.622 Tripartite Normal 1935 76 0.647 - 1 1921 80 0.640 - 2 1923 66 0.649 - 3 1910 101 0.678 - 4 1912 82 0.654 - 5 1928 71 0.632 - 6 1912 82 0.654 - 7 1312 202 0.811 - 8 1020 209 0.938 - 9 1672 53 0.656 - 10 1411 34 0.673 The 200-bus saw the largest decreases in the number of interactions from 1411 to 1388 in disturbance 3 for the BN and from 1935 to 1020 in disturbance 8 for the TN (Table 12). Decreases in interaction size correspond with an increase in the number of modules to 32 and 209 respectively. Erasing critical interactions between modules of a larger power grid network can cause further disconnection and modularity than in a smaller power grid network. In a space with limited connections supporting redundancy, a disturbance can further separate network interconnections. The largest increase in the number of modules for the BN is a difference of 14 between the normal operation and disturbance 1. This suggests that B1 and B3 in the 200-bus network is of high degree or are highly connected components. Even though there are only 2 buses removed, the network has to drastically alter itself, hence the additional strong change in the ratio of interior to exterior components (0.566) compared to the normal scenario (0.637). Disturbances 7 and 8 also have a high number of modules 202 and 209 respectively compared to the normal scenario (76) for the TN. Their interior to exterior ratios are the largest change seen in any of the modeling in this study (from 0.647 normal to 0.811 and 0.938). This transition indicates that the network is unbalanced because one source may contribute more connections than the other. These scenarios have high cyber-physical and cyber component outages. In this case, disturbance 8 is more likely to be more decentralized into smaller modules causing the spike in modularity. It does retain some uniformity in the overall structure and arrangement of nodes creating stable nestedness values relative to the normal. Since disturbance 7 a random multi-component outage targeting cyber and cyber-physical components, a lack of normality and organization compared to the original 74 network occurs. Overall, the 200 bus scenarios which contained physical disturbances decreased in modularity but experienced no changes in connectance or nestedness. A heavy disturbance including high relay and substation level outages increased modularity, the number of modules, and the interior to exterior ratio, but decreased connectance, and network interactions. Figure 34: Distributions of Hub, Non-Hub, and Peripheral node classes among cyber and physical devices in the ACTIV 200-bus system generated from the tripartite network. Figure from [60]. Network randomness caused by outages could indicate partial cyber network collapse and disarray for operators. Similarly, spikes in modularity, the number of modules, interior to exterior ratios, and nestedness could also indicate immediate warning signs in overall network health. A sharp decrease in the number of interactions could indicate overall CPS loss, but not necessarily entire module loss. Module loss is affected by the elimination of densely connected components including hub actors. The BN for the 200-bus was only able to identify non-hub, peripheral, and ultra-peripheral actors while similarly to the 9-bus the TN was able to identify hub actors in bus, relay, and substation- switch classes of devices (Figure 34). Buses are the most prominent hub components in the 200-bus model with at least 125 nodes and approximately 75 nodes labeled as non-hub actors. Substations contain the second most number of hub and non-hub actors. Combining hub identification with the TN visualization for the 200-bus leads to patterns in modularity when analyzing a high-level disturbance. 75 Figure 35: The plotting of a node elimination heatmap for the tripartite network for the ACTIV 200- bus system. Here each square represents the number of nodes remaining for each disturbance scenario for selected runs of node elimination. The color gradient to the right of the figure represents the range of nodes available. Figure from [60]. The construction of the node elimination heat map in Figure 35 showcases the patterns of node elimination. Disturbance scenarios 6, 7, 8, 9 all showed faster node extinction. Although disturbance 7 began with most nodes removed, scenario 9 (the full substation outage) essentially transitions the network into a BN. The cyber network represents around 30% of the nodes in the normal network but contributes greatly to the overall modularity. Recursive runs of node elimination provide a reliable option to find the overall portion of the network remaining and the level of network remnant based on consecutive failure by the highest node degree. 4.4.2.3.7. Conclusion In this work, the identification and interactions of the WSCC 9-bus and ACTIV 200-bus were characterized by metrics commonly used in herbivore, carnivore, and plant relationships in ecology. These metrics were utilized during the design of bipartite and tripartite graph networks. Tripartite networks applied in this study advance knowledge on indirect interactions between physical, cyber- physical, and cyber devices. Beyond quick visualization, they provide insight into how disturbances influence the interdependencies of critical network actors. This study contributes unconventional techniques to investigate network changes and adaptations under duress from different denial of 76 service outages. In summary, the information tripartite networks provide produce higher order insights into specific outage types and nodal interdependencies. Some limitations of this work include constructing specific scaled disturbance scenarios to emulate WSCC 9-bus disturbances from prior work. In the future multi-element multi-stage complex contingency scenarios can be studied to provide more insight regarding network strength and resilience. Additionally, other threat types such as Man-in-the-Middle (MitM) should be modeled for comparison to the work done in this study. Some future applications include expanding modeling techniques to other verified power grid case studies with the hope of standardization in the application of partite modeling to electrical infrastructure. CPS networks have become an underlying foundation and structure for various industries. Any network with CPS components at risk could benefit from a BN, TN analysis to test interdependence. For more information on the unseen higher-order interactions, multi-layer networks could be expanded into k-partite modeling. Overall, the techniques detailed in this study seek to enhance a fundamental understanding of cyber-physical interdependencies in power grid networks. 4.4.2.4. Bi-Op CPS: A Bio-Inspired Method for Cyber-Physical Power Systems Optimization* *This section contains content from [65]. 4.4.2.4.1. Introduction Modern technologies have made power systems more intelligent yet more vulnerable to attacks. The adoption of new technologies into preexisting grid components offers more opportunities for cyber- attacks, such as a Denial-of-Service (DoS) [67], to occur. Dealing with such threats requires treating cyber-physical systems (CPS) comprehensively to understand the role that system interdependencies play during disturbances. This prompts an investigation to develop methods that further define these interconnected systems and their components. Categorizing cyber-physical network interfaces and identifying underlying structures can assist in network integration and design for resilience. A goal for future power grid system designs is to improve network resilience and robustness to unexpected contingencies. Bio-inspired network design is one unconventional method that strives toward achieving these resilience goals. Beyond power grids, bio-inspired design has been applied to engineering makerspaces [74], water distribution networks [133], a system of systems design [134], building and the design of the built environment [135], [136], [137], and industrial resource networks [138]. Bio-inspired optimization has specifically been applied to redesign power grids to add physical line connections to maximize the system’s robustness [102]. Previous works have also utilized a specific graph type common in ecological network analysis which allows for species and device type comparisons [34], [60], [66]. Bipartite network (BN) and most recently Tripartite network (TN) graphs utilized modularity and nestedness metrics to understand the role of cyber-physical interdependencies in the resilience of complex CPS [60]. A modularity analysis supports investigations into network partitioning since modularity identifies actor groupings based on their interactions. Examples of bipartite include plant-pollinator networks, due to their resistance to disturbances and mutually beneficial interactions. Modularity, in conjunction with nestedness, helps to model the interactions between the cyber and physical components of a power system and reveals trends between a power system's resilience (based on contingency analyses) and its interconnectivity. 77 The motivation for this work is to use bio-inspired design methods to improve the resilience of cyber-physical power systems. 4.4.2.4.2. Methodology 4.4.2.4.2.1. Case Study Figure 36: A color coded diagram of the WSCC 9-bus case study [139]. Figure modified from [34]. Many different types of structural imbalances challenge power system security [67]. This work utilizes a well-known power system case study called the WSCC 9-bus system [139]. The physical system includes 3 generators, 9 buses, and 3 loads [72] separated into three individual substations (Figure 36 one-line diagram for the WSCC 9-bus physical system). The cyber system can be generated based on the 3 substations, comprised of relays, ethernet switches, human-machine interfaces, routers, and firewalls. The cyber system monitors and transfers real-time data and sends commands that control the physical components. Relays are the cyber-physical connections between the physical and cyber networks of the WSCC 9-bus. The case study is redesigned twice, once via traditional engineering methods and via non-traditional engineering methods. After each redesign, the same two DoS scenarios (defined below) are applied. The DoS is assumed to have occurred to both physical and cyber components. One component, whether physical or cyber, is in a state of failure giving an N-1 state for the system, two components are N-2 and so on. The risk is assessed and the most probable location of the next attack is guessed. Scenarios are follows: 1. DS1: DoS of physical components: generator 1 outage (located in Substation B in Figure 36) and bus 6 and bus 9 outages (located in substations B and C) 2. DS2: DoS of cyber components: relay 13 and relay 20 outage (connected to substations B and A) 78 4.4.2.4.2.2. Bipartite Networks and Metrics Previous bio-inspired design work using ENA explored graph visualization and matrix-based depictions to capture interactions between actors in a network [34], [60], [66]. The N x N structural matrix, where N is the number of actors or nodes inside the system boundaries, creates quantitative characteristics from the network. The bipartite network model in Figure 37 -left enables a modularity analysis to understand the interaction patterns in the network. Figure 37: Left - An example of a hypothetical cyber-physical power system bipartite network model of the cyber-physical interface, where “C” represents cyber components and “P” represents physical components. Right – a hypothetical plant-pollinator bipartite network model where generalists (left) interact with many partner species, while specialists (right) interact with few partners. In both models, interactions can be described as mutualistic, or mutually beneficial. Figure on right modified using [140]. Figure left modified from [34]. A bipartite interaction matrix is constructed for the WSCC 9-bus power grid and modularity and nestedness, two metrics adopted from the analysis of ecosystems, are calculated. Modularity (Qb, (4.16)) calculates the overall level of clustering due to interactions in the network, where E is the total number of interactions in the network and Eij are the matrix entries, 1 or 0 based on the presence or lack of an edge [74], and gi and hj are the module indices of the nodes i and j and ki and dj represent the degrees of i and j. Delta (𝛿) parses the module indices for pairings between actor groups. A value of one is assigned if actors are in the same module and a value of zero if they are in two different modules. This iterative process is carried out initially by dividing the network into two modules and calculating the modularity until the network has reached maximum modularity [76], [77]. The modularity of a bipartite network is limited by the total number of interactions possible versus actual interactions in a network, also known as connectance or density. 1 𝑘𝑗 𝑑𝑗 𝑄𝑏 = ∑ (𝐸𝑖𝑗 − ) 𝛿(𝑔𝑖 , ℎ𝑗 ) (4.16) 𝐸 𝐿 𝑖𝑗 Nestedness is another metric prominent in ecological plant-pollinator studies that quantifies the level of structural pathway hierarchy created by connections between network actors (see Figure 37-right) [78], [80], [81], [83]. Nestedness in human systems investigations include studies of the stability of bipartite networks to disturbances and analyzing failure rates of large-scale industrial networks and 79 even engineering makerspaces [75], [82], [138], [141]. Calculated here using Nestedness based on Overlap and Decreasing Fill (NODF), nestedness can help to understand how network components are coupled and relate connection patterns back to the resilience of mutualistic [84]. NODF ranges from 0 to 100, or from 0 to 1 if the values have been normalized. Higher values indicate a more hierarchical pattern of connections, similar to what is seen in mutualistic networks in nature [85]. NODF compares column values by looking for decreasing fill condition. When it is met by either the first or second pair of interactions being less than the other, the lower pair is represented by a zero in the calculations. 0 𝑛𝑖𝑗 𝑖𝑓 𝑐 ≤ 𝑘𝑗 𝑀𝑖𝑗 = { (4.17) min(𝑘𝑖 , 𝑘𝑗 ) 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 ∑𝑖𝑗 𝑀𝑖𝑗 𝑟𝑜𝑤 + ∑𝑖𝑗 𝑀𝑖𝑗 𝑐𝑜𝑙𝑢𝑚𝑛 𝑁𝑂𝐷𝐹 = (4.18) 𝑚(𝑚 − 1) 𝑛(𝑛 − 1) + 2 2 In (4.17), ki is the sum of the row or column i likewise kj, is the sum of the row or column j, nij is the total number of entries matching between the two viewed values, and c is the number of entries that are a value of 1 for all values in kj. In (4.18), NODF is the normalized nestedness of the overall system and produces the final NODF value from 0 to 1. Bipartite models and a modularity/nestedness analysis can provide quantitative insight into the cyber-physical interactions and the resultant reliability, serviceability, and sustainability of the network. Modularity and nestedness are related via network connectance, or density, with highly connected networks having the potential to have high or low nestedness and low modularity and weakly connected networks having low nestedness and high modularity [75]. The larger the network the clearer this relationship becomes. More generally networks with low nestedness tend to have high modularity and networks with high nestedness tend to have low modularity. 4.4.2.4.2.3. Bipartite Robustness Alongside the metrics introduced above, a new metric called bipartite robustness (BR) is introduced to compare the resilience of cyber-physical power system (CPPS) networks based on the design of the cyber-physical interface [60]. A Python algorithm was created in [60] to iteratively remove the most connected actors from the bipartite graph network, starting with those having the highest degree (i.e., most connections). (4.19) 𝑐𝑒𝑛𝑡𝑟𝑎𝑙𝑛𝑜𝑑𝑒 = arg max 𝑑𝑒𝑔𝑟𝑒𝑒 (v)𝑣 ∈𝐺 ⬚ (4.20) 𝐺 = 𝐺 − 𝑚𝑜𝑠𝑡 𝑐𝑒𝑛𝑡𝑟𝑎𝑙 𝑛𝑜𝑑𝑒 (4.21) (4.22) 80 𝑆𝑎𝑓𝑡𝑒𝑟 (4.23) 𝑇𝑎𝑡𝑡𝑎𝑐𝑘 = 𝑆𝑏𝑒𝑓𝑜𝑟𝑒 Sbefore and Safter represent the network size before and after an attack on the central node, or the actor with the largest number of edge connections. The most central node and its respective edges are iteratively eliminated from the bipartite graph. After each removal, the code calculates the fraction of remaining type B actors (cyber components in this case, rows in the matrix) that still have connections to any type A actors (columns). Connections remaining indicate the system's integrity and BR thus quantifies the resilience of the bipartite network, with higher values indicating greater robustness. As applied here, BR aids in determining the most impactful DoS scenarios. 4.4.2.4.2.4. Remote Relay Backups and Sensing A newer solution to mitigating threats and breaches on relay equipment specifically is altering the network to create local or remote backups. These relay backups are useful because they can provide alternative secure and redundant communication channels [142]. In terms of physical security, they provide additional connection points for fault minimization and duration while supplying information on locations and types of system failures. Relay protection should encompass sensitivity to small loads, selectivity in their communication and reporting structure, speed, and reliability to ensure proper system functionality [142]. Reliability can be further clarified by dependability and security: dependability is the degree of certainty in which a relay operates correctly under faults; security is the degree of certainty that the relay will not operate incorrectly for any fault within its assigned zone [143]. Focusing on remote relay backups in this study prompts redesigning the 9-bus power system with these redundant relay connections. Specifically, power experts when redesigning prefers a layered separation between primary and secondary information backup devices. This separation ensures that the backup system remains secure and unaffected in the event of a cyberattack on the primary system, allowing the backup system to take over operations seamlessly. They may also employ multi- level redundancy in their relay backup designs, as seen in Figure 38. This means that a failure in a primary component triggers an intermediate backup system, followed by a tertiary system if needed. For example, the protection relays and control centers are organized hierarchically to manage failures at both local and regional levels. 81 Figure 38: An example of different relay protection schemes, including individual relay zones (thin blue dashed circles), local and duplicate backup zones (thick light blue dashed oval), and remote relay backup zones (thick purple dashed circle). The red blocks with R# are the relays and the black lines connecting them are the lines/interactions. Vertical black lines are the physical buses (corresponding to those in Fig. 1). The disturbance is indicated by the lightning bold and “fault” label between relays 1 and 2. Remote relay backups are known to provide enhanced redundancy, and boost reliability, due to the large number of connections generated when they are added. They are also however constrained to a local circle of devices, as seen in Figure 38 where you see “individual relay zones”, “local and duplicate back up zones”, and “remote relay backup zones”. Redesigning the WSCC 9-bus case study using the traditional engineering solution of adding remote relay backups. The traditional redesign for example takes R2 (relay 2) and adds a remote protection scheme with relays 4-6 (R4-R6) associated with buses 2-4. This provides a clear comparison for the non-traditional solution proposed. 4.4.2.4.2.5. Optimization Bipartite robustness may be useful for understanding attack propagation and pathways in networks and for guiding designers toward resilience improvements. Payne et.al. established in [34], [60], [66] that bipartite representations of the WSCC 9-bus system consistently display Qb values closer to 1 and NODF values close to 0. This indicates a highly segregated network with minimal interactions between core groups or modules. While such a network of independent group clusters can be beneficial, it better suits distribution-style power systems such as microgrids or nano grids. This study investigates the benefit of increasing nestedness specifically in transmission power systems. The network is optimized to maximize nestedness, mimicking the highly nested structures found in mutualistic networks in nature. Prior work using this approach on industrial water networks found that designs with a nestedness close to 1 had more effective use of resources, minimizing raw material inputs and waste outputs, and responded the best to disturbances as compared to networks with lower nestedness values (closer to 0) [138]. Based on the ideas of remote relay backup implementation, this study adds constraints to the optimization, including row and column maximums for new line-to-line connections to optimize specific sections of the 9-bus case study. Performed in MATLAB utilizing the bipartite package [77], the optimizer takes in the bipartite matrix and iteratively adds connections to optimize nestedness. 82 Unlike the location-constrained connections in the traditional remote relay backup redesign, the optimization framework instead considers relays located at all local buses as viable locations for line additions. The optimization constraints ensure that each device (relay, generator, etc.) has realistic new connections that can be made: 7 for relays, 7 for buses, 2 for generators, and 3 for new load connections. This approach allows for refining the bipartite networks, ensuring that the network maintains high functionality while adhering to the power-specific structural constraints. The iterative method is useful for studying complex ecological or technological networks for connectivity patterns in the system's overall performance and robustness. 4.4.2.4.3. Results and Discussion Understanding the modularity and nestedness of the cyber-physical system concerning disturbance analysis results provides critical information about the design of cyber-physical interconnections and the grid's resilience. Table 13 shows the number of network interactions added beyond the original WSCC 9-bus model and the resulting nestedness (NODF), and bipartite robustness (BR). The modularity and nestedness of the original WSCC 9-Bus network are consistent with previous results, finding that the network is highly modular with no nested structures and has low bipartite robustness (below 30%) [60]. The remote relay scenario increases the number of connections between relays with 20 new interactions between the cyber and physical subsystems in the network. These new interactions slightly increase both the nestedness and BR compared to the original WSCC 9-Bus design. The remote relay protection redesign experiences the lowest drop in resilience, as defined by BR, after the cyber-only disturbance (DS2), suggesting that due to its design, DS2 had the smallest impact on the starting resilience of the network. The bio-inspired redesign adds more than double the interactions added by the remote relay redesign, at 50 total new interactions. The resultant network is significantly more nested (NODF 0.43), confirming the effectiveness of the optimization to maximize NODF. The bio-inspired redesign increases network resilience the most, as defined by BR. This suggests that the optimization algorithm selected to add relay connections that increased NODF and ones that provide a higher operational resilience. The bio-inspired redesign starts of the most resilient of the 3 network designs and remains the most resilient through the 2 disturbances tested. This is important because it is not unlikely that one disturbance/issue follows another, so the post-disturbance BR value is an important indicator of the network’s vulnerability to a secondary attack. Overall, the results here suggest that compared to the remote relay protection scheme, the bio-inspired redesign contributes more holistically to system health. The approach seems to place relay-to-relay interactions in the best possible locations for the overall system rather than focusing only on a local relay area, ultimately leading to a design with better overall network area coverage. Table 13: The three network designs are investigated under normal operational conditions and 2 disturbances (DS1 and DS2). The original WSCC 9-bus model, the traditional remote relay protection redesign, and the bio-inspired redesign are described by their before and after nestedness (NODF), bipartite robustness (BR), and interactions added beyond the original design (∆WSCC). The bolded/red values correspond to the maximum values of NODF and BR across all model versions. Model Version Scenario Interactions NODF BR Added (∆WSCC) WSCC 9-bus Normal N/A 0 0.27 Original Case Study 83 Model Version Scenario Interactions NODF BR Added (∆WSCC) DS1 N/A 0 0.17 DS2 N/A 0 0.20 Remote Relay Normal 20 0.04 0.34 Redesign DS1 15 0.06 0.23 DS2 19 0.03 0.32 Bio-inspired Normal 50 0.37 0.43 Redesign DS1 38 0.31 0.31 DS2 39 0.31 0.35 Figure 39: Bipartite network models of the cyber-physical interface of a) the original WSCC 9-Bus cyber-physical network, b) the remote relay traditional redesign, and c) the bio-inspired optimized redesign. The colored bar legend is the number of new line connections per device. Note: the two redesigns (b, c) go from 1-7 links while the original (a) is only 1-3 links. Actor abbreviations: B- bus, G-generator, L-load, R-relay. Figure from [65]. The bipartite graphs of the original WSCC 9-bus network and its redesigns are shown in Figure 39, highlighting the interaction changes made by each redesign. Notable in both redesigns (Figure 39, b and c) are the added generator (G1, G2, G3) and (L5, L6, L8) relay connections. The WSCC 9-Bus original (Figure39a) and remote relay protection (Figure39b) designs do not connect generators directly to any relay components because of transformers between generators and their closest relays. DoS attacks commonly follow graphical pathways when planning attack scenarios. Compared 84 to these models, the bio-inspired model directly connects all generators and loads with at least one relay component, providing redundancy and greater protection coverage because of the indirect interactions present within a highly nested network. Indirect interactions are any connections between components that must pass through another component first and exclude environmental or system boundary crossing interactions. The bipartite network plots of Figure 39 show that indirect interactions primarily occur from the overlapping of relay protection. Here “starting node” indicates a node in which direct and indirect interactions can be traced through the graph. The original WSCC 9-Bus model only has cyber components indirectly connected, for example relay 2 (R2) is indirectly connected to relay 3 (R3) via bus 7 (B7), however Figure 39b and c show that the indirect interactions become significantly more complex and occur also on the physical side in the redesigns, for example, the physical components R8 and L8 are indirectly connected via the cyber component R7. Table 14: The starting node, direct, and primary indirect interactions for the WSCC 9-bus original case study, remote relay protection redesign, and bio-inspired redesign. Model Starting Direct Indirect Interactions Node Interactions WSCC 9-Bus Original Case R12 B7 R3, R4 Study Remote Relay Redesign B2, B7 R1, R3, R4, R5 Bio-inspired Redesign B2, B3, B7, B9 R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12, R13, R15, R17, R19, R20 Table 14 highlights how the redesigns both increase the number of indirect interactions, especially in the bio-inspired redesign, which has 17 backup relay connections instead of 2 in the original design and 4 in the remote relay redesign. The bio-inspired redesign can be achieved at a relatively low cost, as the changes are primarily new cyber devices that further connect and store more data. A highly interconnected network may sometimes be seen unfavorably because it increases the interdependencies between the cyber and physical components, providing more routes for disturbances in one substation system to pass to the other. However, adding additional security layers within the cyber network such as firewalls, and router-substation checkpoints can alleviate the directness of the connection without changing the improved system architecture. 4.4.2.4.4. Conclusion This work has demonstrated the applicability of bio-inspired design to effectively optimize new connections for relay protection schemes in the WSCC 9-bus power system case study. By constructing traditional and bio-inspired relay protection schemes in a CPS network overall network resilience was increased even under DoS threats. Utilizing an optimization framework that maximizes bipartite network nestedness assisted in constructing systems with higher levels of bipartite robustness. It was also found that nestedness maximization led to a trade-off decrease in modularity creating a more balanced network better suited to the design of transmission power systems. Although successful in utilizing the 9-bus case study, future work will explore larger transmission-style case studies to establish a pattern for redesign. Additionally, microgrid and distribution style systems could produce diverse results from those presented here as a microgrid is a 85 power distribution-based system. Finally, increasing the number and types of disturbances could lead to a more concrete interpretation of the results by providing patterns of resilience loss among the cyber, physical, and CPS system networks. 4.4.2.5. EMERA MICROGRID Results Utilizing the methods above in other sections of this report, specifically C (4.2), NODF (4.4), and Qb (4.1) equations. We apply bipartite network metrics to the EMERA microgrid system, for which the cyber-physical graph is shown in Figure 40, for a normal operating state. Figure 40:Cyber-Physical Graph of Emera MG Model 86 Figure 41: Emera bipartite graph. Table 4.15 87 Table 16: Connectance, Nestedness, and Modularity metrics for the Emera microgrid system in normal operating state. C NODF Qb 0.1 0.0 0.9 Although these results are similar to that of the WSCC 9-bus test system, these results align better with the initial design and operation of the microgrid system. A microgrid being a distribution system which is meant to operate within separate sections and has low interdependence on the 10 modules within the system. This translates well with the results seen in Table 16 where NODF indicates a network with no layered interactions and no out of module connections. Here each of the modules are clearly defined in Figure 41. An exception to this would be any connection to the central power grid. However, for the purpose of this analysis the microgrid was looked at as an independent system. It is also worth noting that that there are no nodes present with out of module connections or nodes which could be labeled as ‘hub’ or more central actors. 4.4.3. Ecological Network Analysis (ENA) 4.4.3.1. Weighted and Undirected Ecological Network Analysis Ecological Network Analysis (ENA) quantifies graphical network architecture characteristics using the matrices. The benefit of this approach is the utilization of pre-existing software and graphical libraries for computing. Additionally, structural food web analysis employs directed binary values perfectly suitable for complex graphical structures and easily recompilable for transient simulations. The structural food web matrix [F] (Figure 42-right), also known as the adjacency matrix [A], contains zeros and ones, a transfer from producer i (prey) to consumer j (predators) is 𝐹𝑖𝑗 = 1 and no connection is a value of 0 [144]. Figure 42: A hypothetical food web directional graph (left) and its structural matrix (right). Edges/connections are drawn and quantified as being from prey to predator. Figure from [145]. The size of the network is described in terms of the number of actors (N, equal to the rows and columns of [F], 4 in the case of Figure 42. The number of links or edges in the network (L) is the sum of all entries in [F]. The prey-to-predator ratio (PR, (4.24)) is the average number of producers 88 connected to one consumer. When there is a significant difference between producers and consumers (PR close to zero) it can indicate a slower system recovery to disturbances. Specialized predator fraction (PS, (4.25)) is the average number of consumers who only interact with one producer. Large numbers of specialized predators or consumers indicates a highly vulnerable system with lots of “dead end” edges [146]. Generalization (G, (4.26)) is the total number of links divided by the number of predators/consumers and represents the average number of producers available for a consumer to interact with. Larger values may suggest a greater chance of recovery from disruptions due to higher producer availability. 𝑐𝑜𝑢𝑛𝑡 nonzero rows [𝐅] 𝑃𝑅 = (4.24) 𝑐𝑜𝑢𝑛𝑡 nonzero columns [𝐅] 1 𝑓𝑜𝑟 ∑𝑁 𝑗=1 𝐹𝑖𝑗 = 1 ∑𝑁 𝑖=1 ( ) 0 𝑓𝑜𝑟 ∑𝑁 𝑗=1 𝐹𝑖𝑗 = 0 (4.25) 𝑃𝑆 = 𝑐𝑜𝑢𝑛𝑡 nonzero columns [𝐅] 𝐿 𝐺= (4.26) 𝑁𝑝𝑟𝑒𝑑𝑎𝑡𝑜𝑟𝑠 {𝜆𝑚𝑎𝑥 = max (ℝ [𝑭−1 − 𝜆𝐈])} (4.27) The magnitude of the largest real eigenvalue of the transpose of [F], also known as the adjacency matrix [A], is the cyclicity (𝜆𝑚𝑎𝑥 , (4.27) [111]. This describes the presence and strength of structural cycling and corresponds to the increased complexity of the cycles within a network. Higher cyclicity values also correspond to higher amounts of indirect cycling that have been found to have additional benefits for sustainability [86]. A value of zero indicates no structural cycling is present, one indicates the presence of simple cycles, and greater than one indicates strong structural cycling. Equations 1-4 evaluate the stability and cycling properties of a power system when modeled as an ecological network and enable a before and after comparison of the adjustment of the network. 4.4.3.1.1. START Utilizing the START framework defined earlier in this report, the topologies associated with the WSCC 9-bus network structure changes from the defined disturbances 1, 2, and 3. These changes reflect how cyclic and balanced the START connections represented from the Autoencoder. Table 17: Network analysis metrics for the START framework captured for changes in topology and scenario. Topology Scenario Pr Ps G Cyclicity Normal 1.00 0.75 1.25 0.00 Physical DS1 1.00 0.82 1.18 0.00 89 Topology Scenario Pr Ps G Cyclicity DS2 1.00 0.75 1.25 0.00 DS3 1.00 0.82 1.18 0.00 Normal 1.20 0.11 3.20 2.78 DS1 1.20 0.11 3.20 2.78 Cyber DS2 1.21 0.12 3.09 2.78 DS3 1.21 0.12 3.09 2.68 Normal 1.15 0.09 3.15 2.78 DS1 1.13 0.11 3.11 2.78 Cyber-Physical DS2 1.20 0.09 3.09 2.68 DS3 1.18 0.11 3.02 2.68 Table 17 reveals that the physical topology is stable but inflexible and contains low connectance and no structural cycling. The cyber topology is highly connected, with low specialization and strong cycling. The cyber-physical topology is balanced between the cyber redundancy and the physical rigidity. Pr within the physical network is constant and linear compared to the cyber with a high flexibility, and the cyber-physical in between. Ps demonstrated that the physical network has more specialized consumers, and is more fragile, while the cyber network is very generalist which could contribute to resilience. The cyber-physical network shows a low value for the normal scenario, which then rises during the disturbances indicating that it becomes more fragile. Similarly, the cyber network has a high generalization (G), indicating the possible ability to reconfigure under duress. The cyber-physical network however loses some flexibility under these disturbances. Lastly, the physical network has no cycling, indicating a very rigid network. The cyber network has more complexity and the cyber and cyber-physical networks both see a slight decrease in cyclicity under the disturbances. 4.4.3.1.2. EMERA When modeling the Emera microgrid network we considered the cyber-physical system as the base topology. We performed the metrics referenced above in the “ENA” section after constructing matrices and network topologies. Table 18: Network analysis metrics for the Emera microgrid. Topology Scenario Pr Ps G Cyclicity Cyber- Normal 1.00 0.87 1.98 4.35 Physical 90 Table 18 suggests that PR indicates a low network redundancy, implying in terms of ecology that each predator has on average one prey. This suggests limited adaptability if a prey or in this case the main power supply fails. However, since Emera is a microgrid it could be run off of PV panels and battery energy storage for some time, depending on the amount of said energy stored. In this case, more than one source of primary energy would benefit Emera (PV, battery energy storage, wind energy, etc.). The high number of specialized predators (PS) also indicates extreme specialization which makes the system fragile to disturbance which affect the primary power supply. Generalization (G) appears contradictory with PR and PS, since it implies that each predator is connected to two prey on average, meaning that some generalist consumers are connected to many preys increasing G. This mismatch suggests an uneven distribution of links, which accurately represents the lightly connected structure of the microgrid. Cyclicity is very high for this network, however considering nature on average is over 9 [135]. This indicates complex structural cycling, which accurately reflects the shared energy and interconnections between different nano-grid boxes and energy sources. Theoretically this level of recycling should support resilience. Overall, Emera as a system presents a strong structural cyclicity, but high nodal vulnerability. This implies that if lower-level devices or actors fail, then critical nodes could then fail as a direct result. However, this configuration may also recover well from some types of disturbances, this is a good candidate for power systems and design optimization. 4.4.3.2. Directed and Weighted Ecological Network Analysis 4.4.3.2.1. Insight Into Cyber Information Flow Interdependencies Using Bio-Inspired Topologies* *This section contains content from [66]. 4.4.3.2.1.1. Introduction Modern technologies are revolutionizing power systems, improving their control and making them more interconnected but also leaving them increasingly susceptible to cyber-attacks. The rapid adoption of emerging technologies is creating a fertile ground for cyber threats, such as Denial-of- Service (DoS), Man-in-the-Middle (MiTM) [67], and malware attacks [89]. A prime example of this growing vulnerability is the report concerning the cyber attack to the Ukrainian power grid in December 2015 [68]. They detail the repeated targeting of their energy infrastructure monitoring systems, showcasing the unexpected potential of cyber disruptions to influence physical components and ultimately lead to costly blackouts. Addressing these significant challenges necessitates a holistic approach to cyber-physical systems by first focusing on the intricate interdependencies at play during security breaches and disturbances. and disturbances. Traditional methods of handling power system emergencies rely on grid operators and hazard operation plans. Operators are trained to deal with emergencies to prevent blackouts [147] yet their operation departments often have minimal cooperation and collaboration with cyber departments [148]. A structural gap exists between power systems’ Operational Technology (OT) and Information Technology (IT). Current strategies for handling emergencies in cyber-physical systems (CPS) are evolving but still in their infancy, highlighting the need for integrated and well-conceived 91 plans that bridge this OT/IT divide [149]. Such plans are essential for developing more resilient and reliable power systems and preventing previous major US incidents. Biological ecosystems on the other hand have demonstrated remarkable resilience in the face of disturbances, a quality captured by ecologists studying network architecture [45] and applied to human networks by engineering researchers. Ecological Network Analysis (ENA), which features quantitative metrics relating network characteristics to their functioning, has been applied to human networks such as supply chains [150], systems of systems [134], water distribution networks [104], and industrial resource networks [138]. These works however have primarily concentrated on physical networks, including power systems [73], [102], [151], [152], aiming to emulate ecosystem resilience. These studies have used several ENA metrics, including Ecological Robustness (RECO), Average Mutual Information (AMI), Cyclicity (λmax), and Degree of System Order (DoSO), to evaluate and design physical power systems network structures. Following the successful characterization of cyber-physical power systems utilizing DeepWalk and bipartite network modeling methods [34], the cyber component connections of power systems are investigated to understand the strengths and vulnerabilities of the cyber network layer separately. Cyber network topologies here are modeled in a similar fashion to ecological food webs and evaluated using ENA [58] to assess the resilience, risks, and vulnerabilities related to the overall topology of cyber-physical systems. Prior work has elaborated on the development of preliminary topologies for a cyber network [151], examining 2 distinct scenarios (in a 3-substation and an 8-substation cyber-physical system) and emphasizing the potential of an ENA based approach. That work also stated a need to further enhance cyber topology modeling by adding more content to the overall system, including firewalls, substations, and relays, to enable advanced disturbance patterns to be observed and provide insight into the inner workings of network resilience, robustness, and security. This work advances these previous studies by computing disturbances specifically applicable to communication connections, described in [34]. Packet delivery information and changes in topological structure are analyzed, providing new insight into the possible openings available to adversaries and their impact on the overall network’s ability to function under duress. The main contributions of this paper are as follows: 1. A bio-inspired model for assessing the strengths and weaknesses of communication system interactions. 2. Evaluation of the WSCC 9-bus power system case study using ENA and related network methods. 3. Highlighting the most resilient network structures and the most robust and reliable network designs and data routing approaches using ENA. 4.4.3.2.1.2. Background and Problem Formulation A Denial of Service (DoS) attack is a particularly critical threat that targets communication networks by flooding the network with an overwhelming volume of traffic using protocols. A DoS attack overloads the processing of messages and effectively disrupts the flow of necessary operational data. This disruption can lead to a cascade of negative outcomes, from operational delays to a complete shutdown of system monitoring and control functions. The consequences of such an attack can be far-reaching, potentially resulting in power outages, compromised system security, and even safety hazards. 92 This study uses an augmented Western System Coordinating Council (WSCC) 9-bus System [72] as the primary case study. There are 3 substations in total, composed of generators (G), buses (B), and loads (L). The focus here however extends to the cyber layer of the WSCC 9-bus system. A reconstructed cyber topology is created, inspired by a previous topology [34] where all the backup routers are removed. Devices in this topology have the following functions: routers (r) act as intermediaries between networks, Human Machine Interfaces (HMI) provide system operators with real-time monitoring and control, Firewalls (FW) guard against unauthorized access and cyber intrusion, and Relays (R) serve cyber-physical components for system protection and data acquisition. The cyber network is created as a hierarchical system, as is shown in Figure 43, which mirrors the real-world power system networks. Each substation is composed of Relays (R), Firewalls (FW), routers (r), Ethernet Switches (SW), and HMI. As is shown in Figure 43, each is colored differently with its own control room, which aims to run local operations coherently with the control center and monitor the stability of the power system. A main control center oversees the control rooms, acting as the brain of the system, which is responsible for data analysis, synchronization of different substations, and coordination of power generation. Figure 43: WSCC 9-bus System Cyber Topology. Figure from [34]. Several assumptions are made to analyze the information flows and the robustness of the cyber system. Firstly, the system here is analyzed based on a steady-state model. Secondly, cyber communication packets are modeled as discrete numerical values. For example, in Disturbance 1, the relay packet information begins as 1-per-relay in the system and is additive as it passes through other components/nodes. The cyber packets refer to digital data packets that carry critical information such as voltage magnitude, state of the breaker and the other electrical commands sent back from the operators. The disturbance scenarios below highlight the diverse range of adversary targets. Within these scenarios, several different elements have been placed at risk in order to 93 explore the system’s response to singular or multiple actor (cyber element) type disturbances. All disturbance scenarios are modeled as a Denial of Service (DoS) attack to the following components: • Disturbance 1: Relay 13 and Relay 20 • Disturbance 2: Communication between Relay 3 to Substation A’s Ethernet Switch 1, Substation A’s Ethernet Switch 0, and router 0 4.4.3.2.1.2.1. Sending Data The objective of the cyber network’s communication is to send data from the relays to the control center. Since both local and remote controls are controlled by the relays in this system, the data of the entire physical network is transmitted to both the respective local Human-Machine Interface (HMI) and the control center HMI. As we can see from the diagram in Figure 44, the packets are calculated based on information flow. Figure 44: Information Flow Diagram for One Way Sending Data. Figure from [66]. 4.4.3.2.1.2.2. Delivering Commands Figure 45 demonstrates the information flow when delivering commands. The primary goal of the information flow is to facilitate the efficient transmission of commands, dispatched through localized and control center HMIs. Local and remote controls both issue the same types of commands, supporting the consistent and coherent operation of the network. However, a key distinction lies in the prioritization of these commands. While local commands, issued from the local HMI, are crucial for on-site management and immediate responses, commands from the control center, which oversees wider network operations and strategic decisions, take precedence during conflicts. This ensures that local operators can make real-time adjustments and respond to immediate needs in the field, while overall network strategy and security are the responsibility of the central control system. Figure 45: Information Flow Diagram of One-Way Delivering Command. Figure from [66]. 4.4.3.2.1.3. Methodology The analysis here includes Ecological Network Analysis (ENA) metrics, and traditional topology metrics including Average Node Degree (AND), Average Clustering Coefficient (ACC), and Average Betweenness Centrality (ABC) as calculated in [153], [154]. 94 4.4.3.2.1.3.1. Ecological Network Analysis Prior work used ENA, which uses graph visualizations and matrix-based depictions to calculate quantitative network characteristics, to investigate a cyber-physical power network [70]. ENA uses two matrices, a structural food web [F] an N x N matrix where N is the number of actors or nodes inside the system boundaries, and an (N + 3) x (N + 3) flow magnitude-based matrix [T]. Figures 46 and 43 show a directional graph depiction of a cyber network for a power system and the flow matrix [T] for that same network, respectively, the latter of which includes system inputs, outputs, and dissipation beyond the selected system boundary [44], [46], [58], [155]. Figure 46: A high-level directional graph for a cyber network of a power system. The cloud represents the system boundary. Figure from [66]. Figure 47: The flow matrix [T] for the cyber network depicted in Figure 4.25. Figure from [66]. Ecological Fitness (𝑅𝑒𝑐𝑜 , (4.28)) is rooted in information and graph theory as well as statistics, based on the quantification of surprise and uncertainty. Reco and its related metrics identify unused energies within a network with respect to the highest possible effective performance [58]. Degree of System Order (DoSO, (4.29)) is the independent metric within 𝑅𝑒𝑐𝑜 and is used by ecologists to quantify the organization, structure, and complexity of ecological food webs. The level of order and complexity within ecological food webs can have important implications for their stability, resilience, and ability to respond to disturbances [58]. 𝐴𝑀𝐼 (4.28) 𝐷𝑜𝑆𝑂 = 𝐻 95 𝑅𝑒𝑐𝑜 = −(𝐷𝑜𝑆𝑂) ln(𝐷𝑜𝑆𝑂) (4.29) Ecologists, when plotting 𝑅𝑒𝑐𝑜 vs. DoSO for long-surviving food webs discovered what they now call the ecological Window of Vitality (WoV, Figure 48) [45], [156]. Networks that form at the extreme values of DoSO (close to zero and one) are hypothesized as being those not fit to survive in nature, while peak ecological fitness is observed within the ecological WoV, a range of DoSO values approximately 0.21-0.59 (note: the boundary is fuzzy). Average Mutual Information (AMI) and the Shannon Index (H) together calculate DoSO. AMI, (4.33) gauges the cumulative uncertainty concerning a flow’s origin and subsequent direction, while H (4.31) represents the maximum potential for organizational development within a flow network [34], [151]. Total System Throughput (TSTp), (4.30), denotes system size, measured by the total energy units passing through the system (from inputs to outputs or the sum of the entire matrix [T]). 𝑁+2 𝑁+2 𝑇𝑆𝑇𝑝 = ∑ ∑ 𝑇𝑖𝑗 (4.30) 𝑖=0 𝑗=0 𝑁 𝑁 𝑇𝑖𝑗 𝑇𝑖𝑗 𝐻 = −∑∑ 𝑙𝑜𝑔2 ( ) (4.31) 𝑇𝑆𝑇𝑝 𝑇𝑆𝑇𝑝 𝑖=1 𝑗=1 𝐻′ 𝐸= (4.32) 𝐻𝑚𝑎𝑥 𝑁 𝑁 𝑇𝑖𝑗 𝑇𝑖𝑗 × 𝑇𝑆𝑇𝑝 𝐴𝑀𝐼 = ∑ ∑ (( ) log 2 ( )) (4.33) 𝑇𝑆𝑇𝑝 𝑇𝑖 × 𝑇𝑗 𝑖=𝑖 𝑗=𝑖 Evenness, also known as Species Evenness (E, (4.32)), is a vital aspect of biodiversity and is mathematically assessed through various indices and metrics, with the well-known Shannon Evenness Index, also referred to as Pielou’s Evenness Index [157], [158]. 𝐻𝑚𝑎𝑥 in (4.32) denotes the maximum achievable value of the Shannon Index when all species have an equal abundance. An E value near 1 signifies a community exhibiting evenly distributed species without dominance. Inversely, a value substantially below 1 implies that certain species dominate, leading to an uneven network. Greater E has been found to correlate with increased stability and resilience in ecosystems, due to no single species exerting excessive influence and improved adaptability to alterations or disruptions. Low E on the other hand may render ecosystems susceptible to disturbances since any impact on a dominant species can significantly affect the entire community [157], [159], [160], [161]. 4.4.3.2.1.4. Results and Discussion The results provide new insight regarding a cyber network’s resilience and robustness against possible adversarial attack patterns. 4.4.3.2.1.4.1. Ecological System Metrics The results indicate that the Relay-to-Control Center topology is less resilient than the Control Center-to-Relay topology. Figure 48 shows that all the Relay-to-Control Center topologies lie outside of the ecological WoV. This suggests that these system designs lack ecological similarity related to their resilience, instead having a higher level of pathway efficiency in their network design 96 (DoSO closer to 1). This may correspond to the reporting mechanism enacted from the Relay-to- Control Center and suggests that creating more redundancy in the system, possibly via backup routers or relay-to-relay communications, would improve the network’s response to disturbances. Notably, within this topology, all the disturbance scenarios have a lower DoSO than the undisturbed scenario, bringing them closer to the ecological WoV. This indicates that the disturbance is removing specific elements that cause lower redundancy, thereby creating a topology that is no longer a linear path. Disturbance 2 falls just shy of the ecological WoV, indicating that the cyber component loss of Relay 3 to Substation A’s SW1 and Substation A’s SW0, and router 0 may actually strengthen topology due to all devices having equal weights. In reality the loss of a substation as a physical device is a major concern, but these results show that it should not be in terms of system resilience. Figure 48: Normal and disturbed cases plotted on the ecological fitness curve, with the ecological Window of Vitality highlighted in green. “Normal” indicates the fully operational/undisturbed case study, “DS” indicates disturbance scenario 1 or 2. Figure from [66]. Similarly, the Control Center-to-Relay topology has the highest DoSO of the network in its non- disturbed state. Disturbance 2 moves the case study closest to the ecological WoV. Cyber components in both case studies with flows passing through them or concluding at them are at risk by adversaries. These are the ideal areas for added redundancy to disperse responsibilities and improve DoSO. Table 19: Shannon Index (H) and Evenness (E) for the normal, DS1, and DS2 scenarios for both topologies. Topology Scenario H E Ecological System - 4.38 1.27 Average Relay to Control Normal 4.76 1.52 Center 97 Topology Scenario H E - DS1 4.75 1.52 - DS2 4.31 1.38 Control Center to Normal 4.86 1.57 Relay - DS1 4.85 1.55 - DS2 4.00 1.28 H and E in Table 19 can also help to identify the level of diversity in the system. While food webs have an average H of 4.38, the Control Center-to-Relay topology is slightly lower at 4.00. This lower value indicates that the network lacks species richness in DS2. This is most likely due to the uneven distribution of devices as a result of the disturbance scenario. At first glance, E further supports this conclusion. Although the ecological average for E is 1.27, any value higher than ecology is highly unlikely. When computed for all case studies, E, within the context of this study may prove to be unreliable for inter-cyber modeling. However, H still assists us with the conclusion that the traditional topology structure in place for cyber systems in the WSCC 9-bus topology lacks redundancy and support from nearby devices. This can be seen in both topologies for the normal scenario, or where H exceeds the ecology average. However, in both DS2 scenarios H falls below the ecological average. This change indicates a lack of balance and integrated network structure. 4.4.3.2.1.4.2. Topology Metrics Traditional topology metrics (average node degree, AND, average clustering coefficient, ACC, and average betweenness centrality, ABC) were also applied to the cyber case studies to understand the network node dependencies. Table 20: Topology Evaluation Results of Normal, DS1, and DS2. Topology Scenario AND ACC ABC Relay to Control Normal 2.19 0.02 0.70 Center - DS1 2.00 0.03 0.63 - DS2 1.65 0.02 0.05 Control Center to Normal 2.10 0.02 0.70 Relay - DS1 2.00 0.03 0.04 - DS2 1.22 0.02 0.04 The results from Table 20 indicate that the more densely connected nodes are found in both case studies’ normal scenarios. The ABC decreases significantly in Disturbance 2 for both topologies. This suggests that the network structure has lost key nodes through which critical information 98 passes. The AND similarly decreases for all the disturbances. The largest loss is seen for Disturbance 2 (Control Center-to-Relay), affirming the criticality of a system lacking redundancy. At edge nodes in a graph, there is a possibility of higher disturbance in lower-level network devices like relays. Lastly, ACC matches well with the ecological results in the previous paragraphs where the network (WoV) gains strength from the loss of an end of topology node in DS1. 4.4.3.2.1.4.3. Performance Goals Disturbance scenarios can lead to a diverse range of impacts on cyber networks. Specifically, in the case of Disturbance 1, relays 13 and 20 confront a Denial-of-Service (DoS) attack. This attack not only disrupts the functionality of the targeted relays but also causes a ripple effect on associated components. The local HMI and control center HMI are adversely affected due to their inability to communicate with specific relays. On the other hand, in the case of Disturbance 2 where the communication between Relay 3 to Substation A’s SW1, Substation’s SW 0, and router 0 are facing a DoS attack, the disruption affects more network of components. As seen in Figure 43, router 0 stands as the pivotal connection hub, interlinking all routers, and is responsible for the aggregation of both data and commands. As a consequence, Disturbance 2 influences all the connected actors in the cyber network. Performance Percentage (PP, (4.34)) is defined as the ratio of output packets in different cases and the total output packets in normal conditions to better quantify the performance. The affected components and performance evaluation are listed in Table 21. # 𝑜𝑓 𝑀𝑖𝑠𝑠𝑖𝑛𝑔 𝑝𝑎𝑐𝑘𝑒𝑡𝑠 𝑃𝑃 = (1 − ) 𝑥 100% (4.34) Table 21: Performance evaluation (components impacted and PP) for the two disturbance scenarios tested. Disturbance Scenarios Impact PP Disturbance 1 Relay to Control Center, Control Center to Relay 2.19 0.02 Disturbance 2 Relay to Control Center, Control Center to Relay 2.00 0.03 The PP in disturbance 1 is 90% while it is 33% in disturbance 2, suggesting disturbance 2 scenario caused more packet loss. Comparing this with the earlier results, it coincides well with topology metrics but is inverse from the WoV predictions. This indicates that system performance is lacking substantially, but the overall topology design itself after the disturbance is more resilient. Therefore, it can be concluded that creating a more interconnected cyber communication topology is possible via the integration of backup substations and routers that can continue to deliver data even when the primary device is no longer operable 4.4.3.2.1.5. Conclusion In this work a cyber communication topology is explored for the WSCC 9-bus system, employing ecological analysis methods that have been related to resilience and sustainability, in addition to traditional topological metrics. These scenarios revealed the lack of redundancy in the normal topology for both sending and receiving communications. The results conclude that a secure information flow network would optimally include several redundant devices including one at the main control center in the network. Overall, our findings are summarized as: 99 • Ecological Network Analysis for the WSCC 9-bus system is effective in identifying topologies more susceptible to disturbances in the inter-cyber interdependencies. • Topology metrics and Ecological resilience metrics support the need for a more redundant topology where devices have support from many connected components. • All results support the need for a higher number of devices dedicated to the support of primary existing devices in the system specifically at the control and substation areas. The topology developed in this work can be applied to other power system case studies for further analysis of overall system resiliency among a higher device diversity network. Identifying patterns in additional power grid case studies will help to quantify a Widow of Vitality range suitable for cyber components in power grid systems. Possible challenges include retaining the same disturbance scenarios with the growth of network size in new case studies. Specifically, the addition of multiple two-way topologies of information flow will be explored utilizing the same metrics. The fundamental goal of this work revolves around enhancing the robustness of the power grid. Grounding the approach in real-world feasibility and ensuring that it remains accessible and user- friendly for decision-makers is critical. 4.4.3.2.2. Bio-Inspired Metrics for Real-Time Detection of Denial-of-Service Power System Attacks* *This section contains content from [65]. 4.4.3.2.2.1. Introduction Technology-driven integration has created an exponential increase in the number of cyber-physical systems and the number of targets for cyber-based adversary exploitation. Cyber disturbances often propagate to physical systems and cause many issues. Man-in-the-middle and false data injection attacks have been found to instigate cascading failures that can cause entire substation and multi- device outages [162]. Research has largely focused on attack replication, post-attack analysis, risk analysis, and preventative attack resources to understand and mitigate negative impacts [34], [66], [67], [163]. In [162], the Common Open Research Emulator (CORE) was used as a network emulator to help in forensic cyber analysis via a set of monitoring metrics that can alert the presence of adversaries. Additional real-time system-level health metrics are still very much needed to address the increasing frequency, potential severity, and broadening targets of cyber-attacks worldwide. Modeling the cyber-physical interdependencies of a cyber-physical system (CPS) using static and temporal graphs provides a systems perspective that can help fill this gap. Static graphs provide insight into the structural nature of the network as snapshots in time. Temporal graphs capture dynamic network properties. Both graphs can help quantitatively establish patterns based on common interactions. Graph models of power network topologies and physical system behaviors can then use graph and information theory to study the impacts of attacks [164]. Graph-based models are quick, simple to construct, and endlessly customizable. Graphs of CPSs result in layers that can highlight power flows on one level, cyber flows on another, and the cyber-physical interactions between them (Figure 49) (Parandehgheibi, Modiano et al. , Soltan, Yannakakis et al. 2016, Wang, Lin et al. 2016). Flow-based edge information in each layer captures device level, class level, and any larger systems level interactions. Currently, finite state modeling is a popular method to analyze dynamic behavior, but poor repeatability creates challenges 100 in post-scenario analysis [68], [165]. There is a need for quantitative metrics that can reliably capture changes in temporal information while preserving the structure of the network. Figure 49: An example of the WSCC 9-bus power system cyber topology. Ecological Network Analysis (ENA) is a technique employed by ecologists to evaluate food web models of biological ecosystems to quantitatively connect network structure to function. One particularly interesting avenue for engineers has been the resultant system-level understanding of ecosystem resilience and robustness in the face of unexpected disruptions [44], [45], [155]. Engineering investigations of these biological system characteristics have found applicability to diverse human systems, including but not limited to supply chains [150], systems of systems [134], and water distribution networks [104]. Past work specific to power grids has explored the design of the physical power grid, finding that the characteristics of ecosystems are valuable for the adaptation of current power systems and the design of future power systems to improve N-X contingencies [71], [73], [102], [152]. Current work specific to cyber-physical power systems has begun evaluating them from a static perspective, which contributes information on vulnerable devices, device groups or types, and the risks of attacks on devices [34], [60], [66], [119]. This article builds upon this past research, applying the ENA metric Degree of System Order (DoSO) to evaluate a Denial-of-Service (DoS) attack propagating over time from an unknown location within the Western System Coordinating Council’s (WSCC) 9-bus power system case study [70]. The metric shows promise for establishing a holistic system-health analysis for understanding temporal cyber interdependencies in CPSs. 4.4.3.2.2.2. Methodology 4.4.3.2.2.2.1. WSCC 9-Bus Power System Case Study and Disturbances Denial-of-Service (DoS) attacks are relatively common in CPSs. They use traffic protocols to drown a network in both/either excess or incorrect operational data [166], with the goal of overloading system processing units to negatively impact physical and/or cyber devices. Most DoS attacks propagate from the cyber network causing spikes in system communications that are often too fast for operators to evaluate the situation accurately. It is also difficult to understand where the DoS attack originated. This study uses the Western System Coordinating Council (WSCC) 9-Bus System [72] case study (Figure 50). Substations in the system act as central hubs for the cyber 101 communication network to communicate with lower-level devices, such as relays, and transmitting devices, such as substations, to human-machine interfaces located in the control centers. Devices such as buses, relays, and firewalls are embedded within the substations in this model, thus supporting a focus on data capture at the primary transport points (i.e., substations). Figure 50: The Western System Coordinating Council (WSCC) 9-Bus power system case study. Figure modified from [60]. The control center is the location of human influence in the system: it is the main communication juncture that oversees power synchronization and coordinates system actions. The substations are centralized locations that represent sections of the CPS system. Typically, substations are modeled as collection points within the cyber system. Instructions for system actions are then carried in data information signals or packets and can include physical system reports e.g., power generation status and commands influencing the state of power devices. The case study information contains packet round trip times (RTT) and packet information at all three substations over 1700 milliseconds. RTT is the time it takes to send a packet of cyber information and receive confirmation that it was delivered to the designated device. Each device in the 9-bus system seen in Figure 50 contains a maximum packet capacity of 9000, which are packets available for receiving during communications. Additionally, due to the nature of it being an imperfect network, up to 100 packets could be dropped on random RTT rows due to various normal communication issues. The three substations in the WSCC 9-Bus System interact with the control center, providing an optimal opportunity for an adversary to perform a DoS. Normal and DoS emulation setups are both analyzed. 4.4.3.2.2.2.2. Ecological Network Analysis Ecological Network Analysis (ENA) was developed by ecologists to study food webs using directional graphs. As such, the approach is flexible and can be applied to almost any weighted or unweighted network directional graph describing components that exchange and transform things 102 like materials and energy. ENA then uses matrix-based depictions of the directional graph to calculate quantitative metrics describing various network characteristics [45], [104]. These matrix depictions consider node-level interactions as well as the magnitude of these interactions. Figure 51 shows a flow-based directional matrix [T] (right) from the hypothetical ecological food web (left). Figure 51: A hypothetical flow matrix [T] (right) of a small food web network model (left), red dashed lines represent the selected system boundaries. The matrix (right) highlights the following entries: “In” are system inputs, “Out” are useful system exports, and “Diss” are dissipations or lost usefulness that leaves the system. Flow magnitude-based (weighted graphs) ecosystem analysis approaches have been particularly successfully utilized to design and analyze engineering systems to adopt ecologically similar system- level characteristics [44], [46], [58], [155]. Degree of System Order (DoSO, (4.38)) is one such metric that shows potential in prior works to quantitatively assess a network’s structure to predict its performance during a disturbance. The metric is used by ecologists to quantify the organization, structure, and complexity of ecological networks and finds that food webs reside in a specific range of values, from 0.21-0.59, suggesting an ecologically superior system performance occurs when the system falls in that range [167]. DoSO for a system can have important implications for its stability, resilience, and ability to respond to disturbances [58]. DoSO is calculated as the ratio of Average Mutual Information (AMI, (4.37)) and the Shannon’s Index (H, (4.36)) and thus represents the complexity of the pathways within a network as measured from 0 to 1. AMI represents cumulative uncertainty regarding a flow's entry into the system and subsequent direction through the system. Here, Tij represents the magnitude of a flow from any node i to any node j within or crossing the system boundaries. Total System Throughput (TSTp) represents the probability of the events occurring at nodes i and j and is the total sum of all flows passing through the network (4.35). Shannon’s Index (H) represents the maximum potential for systems-level organization and diversity within a flow network [151]. Shannon originally labeled this term as the “surprisal” that occurs within a network due to no prior knowledge of an event [168]. As AMI approaches H, the pathways in the system become more constrained. TSTp is the known event in the network that can thereby impose a quantifying organization-based constraint. 103 𝑁+2 𝑁+2 𝑇𝑆𝑇𝑝 = ∑ ∑ 𝑇𝑖𝑗 (4.35) 𝑖=0 𝑗=0 𝑁 𝑁 𝑇𝑖𝑗 𝑇𝑖𝑗 𝐻 = −∑∑ 𝑙𝑜𝑔2 ( ) (4.36) 𝑇𝑆𝑇𝑝 𝑇𝑆𝑇𝑝 𝑖=1 𝑗=1 𝑁 𝑁 𝑇𝑖𝑗 𝑇𝑖𝑗 × 𝑇𝑆𝑇𝑝 𝐴𝑀𝐼 = ∑ ∑ (( ) log 2 ( )) (4.37) 𝑇𝑆𝑇𝑝 𝑇𝑖 × 𝑇𝑗 𝑖=𝑖 𝑗=𝑖 𝐴𝑀𝐼 𝐷𝑜𝑆𝑂 = (4.38) 𝐻 These metrics together produce a quantitative measure for the CPS of the orderliness of the substation to control center interactions based on the number of packets transferred. Although DoSO is not time dependent, this study performs a DoSO calculation at every time step within the RTT data, taking snapshots at every moment in time. These snapshots in time provide insight into how long it takes to send or receive packets between components in the CPS. 4.4.3.2.2.3. Results and Discussion Analyzing the 3400 rows of normal and disturbance CPS data via ENA resulted in the creation of 6800 flow matrices (one physical and one cyber for each scenario) to investigate the ability of ENA to correctly identify DoS attacks over time in a power system. Total system throughput had a maximum difference between normal and disturbance scenarios of 21,000 for all substations and 24,000 for the control center, in sending packets. The origination of the disturbance was determined to be at t = 232 milliseconds from these values. This was confirmed to be the initial insertion point of a DoS propagation into the power system network. At this t, the RTT values for the network also increase from 0.03 milliseconds to the dataset maximum of 3.5 milliseconds. Considering that most RTT times linger between 0.01 and 0.030 milliseconds for this dataset, a time of 3.5 milliseconds indicates a temporarily non-responsive device in the network. 104 Figure 52: TSTp (packets) over time (milliseconds, ms) for normal and disturbance scenarios. Blue and green colored areas in the disturbance plot (right) show times when the TSTp drifted from the maximum and minimum values captured in the normal scenario (left). 105 TSTp values in this analysis highlight a clear distinction between the normal operation of the system Figure 52-top and the DoS event in Figure 52-bottom. Colored areas in Figure 52-bottom highlight the area in which the disturbance scenario caused a rise above normal operation values. Although time-based traffic values are known to fluctuate based on things like internet connection and hardwire reliability (hence the slight differences in values/patterns between Figure 52 top and bottom), any deviations captured above the maximum or below the minimum TSTp for the substations and control centers in normal scenario were a part of the DoS attack. Utilizing DoSO over time can help to find any patterns in network flow and communication. Figure 53 shows that patterns in communication spikes from the three Substations highlight the larger fluctuations and support the identification made from Figure 53 that the DoS occurs at t = 244 ms and is from the Substation level rather than the Control Center. The Control Center can be seen in Figure 53 to stay relatively stable throughout the disturbance (red and blue lines) seeing only a small change at t = 325 ms. This indicates that a DoS attack originating from a substation will first propagate to other substations and then to the control center due to the hierarchal topology. Current CPS designs ensure that the path to a power system Control Center includes various routers and firewalls to protect the human-machine interfaces from collapse, enabling operator function even in a disturbance. Figure 53: DoSO over time for normal and disturbance scenarios. The colored section represents the point of occurrence of the disturbance. Although DoSO is seen here as sensitive to TSTp changes in a network, it may struggle to represent a DoS that floods the system under or equal to a device’s maximum packet capacity. At the time of the disturbance (t = 244 ms), a deeper investigation shows that the network was flooded to the maximum packet capacity of 9000 per device. Since the change is not outside of the normal operating limits of the system the DoSO at this moment fluctuates from 0.315 to 0.300, representing a minimal change in the ecological network metric. But what appears to represent a critical change for this power system case study. As a result, the Substations and the Control Center send high traffic levels within 80 ms of the initial flood (Figure 53). The results here suggest that ENA, and DoSO specifically, in combination with time-based data can alert operators to change from normal, providing a potentially valuable and interesting set of metrics for control center operators. 106 Ultimately, this work supports the hypothesis that ENA metrics can help capture holistic system changes caused by specific disturbances, including the propagation of cyber-attacks over time. Since the data used in this study was manufactured, we hypothesize that obtaining real data or mimicking patterns from previous DoS attacks would show more extreme changes to DoSO for this system. 4.4.3.2.2.4. Conclusion This work searched for systems level methods to monitor and evaluate Substation and Control Center interactions in a cyber-physical power system. New knowledge for cyber-physical power system design configurations is presented by modeling packet traffic flow through the system under various operating conditions. Ecological Network Analysis (ENA) was then shown as able to detect a DoS attack in a cyber-physical power system. Utilizing round trip time data corresponding with substation packet information, Degree of System Order (DoSO) specifically was found to be useful as a system level metric for CPS. The representation of the cyber system level from an ecological perspective provides new knowledge on system health and communication complexity. DoSO was found to have unique ranges for each substation depending on whether the information was sent by the control center or by the substations. Additionally, network analysis over time was able to pinpoint the origination of the cyber-attack at Substation 2 and displayed the new range of system level changes to all substations affected. The ENA approach is seen to be effective in characterizing the state of the system in both normal and disturbance scenarios, especially where the origin of the disturbance may be unknown to operators. The size of the case study is a limitation of this work. In addition to processing larger case studies, future efforts will seek to capture more data at the relay and backup devices to get more detailed packet information and additional insight into large-scale power distribution systems. 4.5. Dynamic Mode Decomposition – Dynamic Mode Decomposition (DMD) is a dimensionality reduction algorithm developed by Peter Schmid and Joern Sesterhenn [169]. It is a data-driven method used for analyzing and extracting spatiotemporal coherent structures from time-varying datasets. Using DMD, it can easily decompose complex, high-dimensional datasets into a set of coherent spatial and temporal modes, capturing the underlying dynamics and extracting important features. For a set of vectors {x1, x2, x3, …, xm} where each, xk ∈ Rn , is assumed that the data is generated by a process that can be approximated by a linear dynamic as 𝑥𝑘+1 ≈ 𝐴𝑥𝑘 (4.51) If the system that generates the data is linear, then the modes as frequencies computed by DMD will correspond to the normal modes of the system. The DMD algorithm is shown below. 1. Arrange the data into a matrices as 𝑋𝑚−1 = [𝑥1 𝑥2 . . . 𝑥𝑚−1 ] (4.52) 107 𝑋𝑚 = [𝑥2 𝑥3 . . . 𝑥𝑚 ] (4.53) 𝑋𝑚 ≈ 𝐴𝑋𝑚−1 (4.54) Where † 𝑨 = 𝑋𝑚 𝑋𝑚−1 (4.55) † is the Moore-Penrose pseudoinverse. 2. Compute the singular value decomposition (SVD) of Xm-1. † (4.56) 𝑨 = 𝑋𝑚 𝑋𝑚−1 (4.57) 𝑋𝑚−1 ≈ 𝑈Σ𝑉 𝐻 3. Compute A, a dimension reduced projection of the full matrix A in (5). 𝑨 = 𝑈 𝐻 𝑨𝑈 = 𝑈 𝐻 𝑋𝑚 𝑉Σ −1 (4.58) 4. Compute the eigen decomposition of A 𝑨𝑾 = 𝑾⋀ (4.59) where (4.60) 𝑊 = [𝑤1 𝑤2 . . . 𝑤𝑟 ] 𝜆1 0 ⋯ 0 0 𝜆2 ⋯ 0 ⋀= (4.61) ⋮ ⋮ ⋱ ⋮ 0 0 ⋯ 𝜆𝑟 108 5. Reconstruct the eigenvectors of A by (4.62) Φ = 𝑋𝑚 𝑉Σ −1 𝑊 Where (4.63) Φ = [Φ1 Φ2 . . . Φ𝑟 ] A is a matrix in discrete time. The relationship between discrete and continuous matrices is Adiscrete=eAcontinousTs, where Ts is the sampling time. The eigenvalues can then be transformed as ln (𝜆𝑘 ) Υ𝑘 = ∀𝑘 = 1, … , 𝑟 (4.64) 𝑇𝑠 In continuous time, the solution of the system computed by DMD can be expressed as 𝑟 (4.65) 𝑥(𝑡) = ∑ 𝜙𝑘 𝑒 Υ𝑘𝑡 𝑏𝑘 = Φ𝑒 Γ𝑡 𝒃 𝑘=1 Where bk(∀𝑘 = 1, … , 𝑟) are the initial conditions of each mode. 4.5.1.1. Power System Measurements Analysis With DMD Power system measurement such as voltages and currents are represented as (4.66) 𝑝(𝑡) = 𝐴𝑚 cos (ωt + 𝜙𝑚 ) Where 𝐴𝑚 , 𝜙𝑚 , and ω are the amplitude, phase, and frequency of the signal. These signals can be vectorized as p(t) = [𝐴𝑚 , 𝐴𝑚 , … , 𝐴𝑚 ]𝑇 . The measurements p(t) taken at time 𝑡𝑘 will be noted as 𝑝1 (𝑡𝑘 ) 𝑝 (𝑡 ) (4.67) 𝑝(𝑡) = [ 2 𝑘 ] ⋮ 𝑝𝑛 (𝑡𝑘 ) Power system measurements are sampled and stored at regular intervals, 𝑇𝑠 . Therefore, a collection of vectors 𝒚𝑘 can be organized as ⋮ ⋮ ⋮ (4.68) 𝑃𝑡−𝑎𝑙𝑙 = [𝑦1 𝑦2 … 𝑦𝑚 ] ⋮ ⋮ ⋮ 109 Where m is the number of samples. Note that the form of (4.68) is the same as (4.52), and (4.53). The synchronous generators mechanical motion results in the sinusoidal signature of the power system. These signals are the result of a marginally stable system with a pair of purely imaginary conjugate poles. Using the DMD method, to identify the power system for a dataset such as (4.68). 4.5.2. Stability Analysis with Dynamic Mode Decomposition Using the HARMONIE dataset and the DMD method to identify regions and samples of data for stability is presented below. The use case takes the voltage magnitude from a randomly selected Bus 4 in the HARMONIE 9 bus system to evaluate the stability of the system during an event. Because the HARMONIE system is a balanced three phase system only a single phase of the Bus 4 voltage is shown in Figure 54. Figure 54- DMD identification of unstable windows of data. A windowing method was used by the DMD algorithm. The output of the was a pair of eigenvalues. The window of data could move as new data was available replacing the old data values so that the window size was constant. For this use case the window was set to a size of 100. Figure 55 displays the results for which windows of data were stable and which ones were unstable in the python console. Figure 55-DMD Stable and Unstable Window of Data. 110 Figure 56-Stable Windows of data Understanding Stability of Eigenvalues In the context of DMD, eigenvalues represent the temporal behavior of modes of the system. They indicate how modes grow or decay over time. The location of eigenvalues in the complex plane helps to identify stability: If the real part of an eigenvalue is negative, it indicates a decaying mode, meaning it is stable. If the real part is positive, it indicates a growing mode, which is unstable. If the real part is zero, the mode is neutral and oscillatory, which might still be a concern for power systems depending on the magnitude. For a system to be stable in discrete time the eigenvalues must lie inside the unit circle. In Figure 56, some of the windows of eigenvalues are plot on top of each other. 4.5.3. Forecasting With Dynamic Mode Decomposition While DMD is not a machine learning technique, the data driven nature of DMD allows for a similar approach to the way data is split between training and testing sets [170]. Matrix A from equation (4.51) is constructed using the training set of data. DMD is applied to this data set which is then compared to the testing set. As an example, in Figure 57 generator one of the IEEE 39 systems 111 mechanical torque is being modulated every 200 ms with a random perturbation within ± 1 % of the nominal. The blue line data set is used for training the DMD algorithm and the yellow is used for Figure 57-DMD Forecasting Generator 1 Frequency. testing. After the DMD algorithm has determined the eigenvectors and modes of the training set of equation (4.65). Each mode evolves independently by its eigenvalues. Therefore, forecasting k steps is Φ𝑒 Γ𝑘 𝒃 for continuous time. From Figure 57 we can see DMD has a good performance of forecasting compared to the testing set of data. This method could be used to predict the future state of the system and then compare that with current future states of the system to detect events. 4.6. Long Short-Term Memory Autoencoder Neural Networks Detection of abnormal events, whether from malicious or failure origins, is a fundamental task in ensuring the security and resilience of the grid. For cyber-physical systems such as the electric grid, such detection must process both cyber and physical data streams to accurately characterize and detect cyber-physical system anomalies [171]. Yet challenges exist around input space dimensionality, generalizability, multimodality, and etc. that accompany diverse data sets with varying temporal and spatial characteristics. Autoencoders (AEs) are one of the most fundamental data fusion techniques that can learn complex non-linear patterns in the data [172]. They are artificial neural networks (ANN) whose structure is symmetrical consisting of an encoder and a decoder, shown in Figure 58, and fall in the category of unsupervised learning focusing on learning the optimal encoding-decoding scheme from data. 112 Further details on their implementation can be found here [171]. Ultimately, a cyber-physical LSTM AE was developed that is uniquely suitable for processing time-series data from a CPS. Figure 58: Autoencoder encoding-decoding scheme. The LSTM-based AE model is trained offline only on the normal operation normalized cyber- physical data, trying to understand its inherent non-linear patterns and temporal dynamics and minimize the reconstruction error (RE) between the encoder’s input and decoder’s output. Through this process, it learns how to perform cyber-physical data fusion and how to distill the salient temporal patterns into a low-dimensional information-rich representation. The key idea of the cyber- physical threat detection lies on the exploitation of the RE metric, which quantifies the dissimilarity between the original and reconstructed data. Specifically, the higher the RE, the more pronounced the deviation from the normal data patterns; therefore, signifying a potential threat/anomaly in the electric grid. Thus, only after the model is optimized and fully trained, we analyze the reconstruction errors of the normal data points and find a threshold that indicates when a data point is normal or abnormal. Overall, the LSTM AE enables classification and detection of normal operation, cyber events, physical events, and combined cyber-physical events. Results of the LSTM AE with the previously presented WSCC 9-bus (“HARMONIE) emulation environment are shared in [171]; next, we will provide results of the LSTM AE for the Emera microgrid environment. 4.6.1. LSTM AE Classification of Emera Microgrid Disturbances The LSTM Autoencoder was trained only on normal cyber-physical data from the Emera Microgrid with 18% compression level of the input (i.e., the latent space is ~18% smaller than the input space). The pre-processing of features was broken into physical features and cyber features and aggregated using a windowing approach. For physical features, all physical features recording integer or floating-point values were included in their raw form using the last value in the given window. This provides a ‘snapshot’ of the measured physical values for that window in time. One improvement to this approach would be down selecting the number of physical field/features included based on either expert knowledge or correlation analysis. For the cyber features, all packets within a window were aggregated and a set of derived features were calculated. The set of derived features came from prior work in a different project. They included ‘single window’ fields such as the number of unique IP’s and average packet size and then ‘rolling window’ fields which take into account multiple adjacent windows, such as the standard deviation of packet sizes. There are a larger number of 113 physical fields compared to cyber fields, and balancing those fields could also be an improvement for future work. For data labeling, the timestamps from the emulation scripts were used for the attack initiation and ending. This will provide reasonably accurate labeling for the events. One known issue with this type of labeling is that there is likely to be latency associated with the emulation process, so the event being detectable in the data may be delayed compared to the timestamp in the emulation script. More sophisticated labeling may be addressed in future work. For the MITM ARP poisoning attack described in Section 3.3.4.3, the following overall data mean and standard variance is shown in Figure 59. Figure 60 shows training loss and Figure 61 shows validation loss. Figure 59: Overall data mean and standard variance. Figure 60: Training Loss 114 Figure 61: Validation Loss From Figure 61, we can see that the validation loss converges and, thus, stop the training after 50 epochs. Next, hyperparameters optimization was performed via grid search for optimizer, dropout rate, learning rate and weight decay. (Batch size = 1, for less noisy results). The results can be summarized as: • Optimizer: Adadelta • Dropout rate: 0.1 • Learning rate: 0.001 • Weight decay: 0.001 Subsequently, the thresholding approach is established by exploring the normal data reconstruction errors distribution. In Figure 62, we can observe that most of the reconstruction errors are between 0.05 and 0.15. However, an interesting observation is that there are some (a few) reconstruction errors that exist between 0.20 and 0.27. Figure 62: Normal data reconstruction error distribution. Next, we study the MITM Attack RE Distribution, as shown in Figure 63. 115 Figure 63: MITM attack RE distribution. Two experiments are conducted: - Experiment 1: Threshold at the tail of the overall distribution (0.27) o Correct Normal Predictions: 100% [3086/3086] o Correct Abnormal Predictions: 79% [2434/3086] - Experiment 2: Threshold at the tail of the first Gaussian (0.15) o Correct Normal Predictions: 86% [2664/3086] o Correct Abnormal Predictions: 100% [3086/3086] o This threshold is obviously better (0 FNs, more FPs) The resulting reconstructions are show in Figure 64. 116 Figure 64: Reconstruction results from varying thresholds for MITM attack. From Figure 64, we can see that for the abnormal data the reconstruction is higher. The reconstructed line operates as a function approximator/regressor. This could suggest that with 147 features and ~5000 samples, we need more data improving the learning of this number of features. The experiment results can be summarized as: - Conclusion 1: It seems that in the normal data there are some outliers that overlap with the abnormal data distribution (which actually verifies Figure 62). This might be due to experiment issues and/or labeling issues. However, it is very interesting that with the REs distribution we are still able to understand and select a threshold where the majority of the REs fall under and achieve accurate classification results (thus, a good threshold). - Conclusion 2: Therefore, even in cases where we might have data issues, the thresholding approach of the LSTM AEs can provide us with some “guarantees”. For the DOS ATTACK described in Section 3.3.4.2, the same trained model is used to validate the AE’s attack agnostic nature. The overall data mean and standard variance (with increased overlap) is shown in Figure 65 and the abnormal RE distributions are shown in Figure 66. 117 Figure 65: Overall data mean and standard variance. Figure 66: Abnormal RE distribution. Figure 66 proves the overlap and the importance of the thresholds. For the DoS scenario, the RE distribution is between 0.2 and 0.3 (as indicated by the secondary threshold option in Figure 62. Again, we perform two different experiments with varying thresholds. - Experiment 1: Threshold at the tail of the overall distribution (0.27) o Correct Normal Predictions: 100% [3086/3086] o Correct Abnormal Predictions: 8% [65/900] - Experiment 2: Threshold at the tail of the first Gaussian (0.15) o Correct Normal Predictions: 86% [2664/3086] o Correct Abnormal Predictions: 100% [900/900] o This threshold is obviously better (0 FNs, more FPs) 118 The resulting reconstructions are show in Figure 67. Figure 67: Reconstruction results from varying thresholds for DoS attack. Thus, the LSTM AE is able to detect and classify each of the attacks as abnormal and as a cyber attack. An overall conclusion from this initial study with the Emera microgrid data can be that the threshold-driven LSTM AE is robust to data outliers. Future steps include a deep dive into the Emera dataset and perform more sophisticated data analysis, including feature correlation, across additional attack scenarios and deeper dive into labeling attack data. Important to note this particular emulation model had a simplistic communication network and did not include an energy management system, which will be added in the future to assess further system complexity. Additionally, in this initial analysis we had more physical data features than cyber data features; this may require down-selection of physical feature or further study of the features. However, even with this imbalance, we can effectively detect the purely cyber attacks (impacting only cyber features). 4.7. Power Resilience Inspection of Synchronous Motor Model 4.7.1. Introduction of PRISM The synchronous motor (SM) model expresses the swing equations of a power grid into an oscillatory dynamical system with forcing terms, known as the second order Kuramoto model. Our power resilience inspection of synchronous motor (PRISM) model builds off the SM model in such a way as to allow for cyber events as well as certain methods of analysis. The form of the dynamical system is given by 119 where and where I is the moment of inertia, Ω is the base frequency (60 Hz), KD is the damping constant, and 𝑃𝑖𝑗𝑀𝐴𝑋 is the maximum power that can be transmitted over the power line from i to j. 𝑃𝑠𝑜𝑢𝑟𝑐𝑒 is the power generated (negative for power consumed) at that node and is estimated by computing the sum of the transmitted power, energy dissipated, and accumulated kinetic energy, see [173] for further details. Under the SM model, the dynamics govern the evolution of the voltage phase angles of the generators and loads. The parameters are computed from the inertia and damping coefficients, the natural frequencies, coupling coefficients, phase shifts between oscillators, and the standard frequency, see [173]. We used the software written in MATLAB by the authors of [174]. We rewrote this software for Python using the PYPOWER library [175], which is itself a Python port of MATPOWER [176]. Figure 68: SIMULATION OF THE PHASE AND FREQUENCY OF THE WSCC 9-BUS SYSTEM BY THE PRISM MODEL WITH THE MAIN GENERATOR STARTING OUT OF PHASE 4.7.2. Modeling Cyber Events and Control with PRISM One of the important utilities of PRISM is that it can analyze many post-cyber event scenarios applied to the grid, so long as they can be described within the common data file. This includes power line outages, load shredding, changes to the reference power of a generator, etc. This makes the PRISM model useful for n-1 type analysis. Since PRISM is evaluated quickly, it can be integrated well in with other tools for analysis. 120 Figure 69: THE WSCC 9-BUS SYSTEM In figure 70, we are given the WSCC 9-bus system, where three seconds after the simulation by PRISM starts, the power line between buses 6 and 9 goes down. In figure 71, the frequencies and phases of the voltage at each bus are graphed. Immediately after the three second mark, we see the effect of power line 6-9 going down on the generators. Generator 1 (in blue) must pick up the slack for the load at bus 6, as generator 3 (in green) can no longer support it. The increase of power from generator 1 and the fact that generator 3 is now only powering bus 8 causes generator 2 (in orange) to also have a decrease in the load that it is responsible for. Figure 70: THE WSCC 9-BUS SYSTEM WITH THE 6-9 POWER LINE DOWN AS INDICATED BY THE LARGE RED ‘X’ 121 Figure 71: FREQUENCY AND PHASE FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM WITH THE 6-9 POWER LINE REMOVED AT 3 SECONDS The WSCC 9-bus system is essentially a cycle graph with three loads and three generators alternating. In figures 72 through 77 we see the effect of losing each single power line in the system. The power line that is most dangerous to the system to lose was the line between the largest load and the largest generator. In contrast, the power line that caused the least disturbance to lose was the power line between the smallest load and the smallest generator, see figure 78. Figure 72: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 4-5 LINE DOWN 122 Figure 73: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 4-6 LINE DOWN Figure 74: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 5-7 LINE DOWN Figure 75: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 7-8 LINE DOWN 123 Figure 76: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 8-9 LINE DOWN Figure 77: FREQUENCY FOR PRISM RUN OF THE WSCC 9-BUS SYSTEM AFTER 6-9 LINE DOWN Figure 78: CONTRASTING THE FREQUENCIES OF THE WSCC 9-BUS SYSTEM AFTER 4-5 POWER LINE GOES DOWN (LEFT) AND 6-9 LINE GOES DOWN (RIGHT) The PRISM model can simulate changes in demand of load or throttle of the generator, in order to express controller inputs and consumer behaviors. Throttle control is accomplished by adding a 124 control term, weighted so that its units are proportional to the force it takes to alter the generator. However, since PRISM is only modeling the swing equations, many such manipulations to the power grid can make the model less accurate. Such “events” and “control” rules are added in as functions defined by the user. These functions may depend on the various bus values, or time, etc. Figure 79: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, AN EVENT OCCURS WHERE ONE OF THE LOADS INCREASES LINEARLY BEFORE HOLDING CONSTANT In figure 79 we see how two controllers, one at generator 1 and the other at generator 2, that both are trying to avoid their generators from having near zero phases may not disturb the system too much by themselves. However, these two controllers together actually break the system due to their lack of consideration for each other. This example demonstrates the need for oversight, or at least protocols that understand the system in its entirety. In figure 80, we see a controller that tries to resist a growing load, but it does too little too late and causes the system to malfunction. Notice that PRISM dynamics will continue to run even though the load is 180 degrees from the rest of the grid. Figure 80: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, TWO GREEDY CONTROLERS TASKED TO ASSURE THEIR GENERATOR’S FREQUENCY DOESN’T GO TOO LOW. EITHER CONTROLLER IS SAFE FOR THE SYSTEM (LEFT AND MIDDLE), BUT TOGETHER THEY VIOLENTLY BREAK THE SYSTEM (RIGHT). 125 Figure 81: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, EVENT WHERE A LOAD IS INCREASING AS IN FIGURE 81 (LEFT). ADDED IN SUPER POSITION IS A CONTROLLER THAT DOES ACTIVATES TO STABILIZE THE SYSTEM BUT IS NOT A DROOP CONTROLLER AND DOESN’T TAKE INTO ACCOUNT THE MOMENTUM OF THE SYSTEM Figure 82: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, SYSTEM RESPONDS TO A SHORT SINUSOIDAL THROTTLE ADDED TO GENERATOR 3 4.7.3. Phase Shift Stability and Sensitivity Analysis The PRISM model has the capability to manipulate initial conditions and forces within the swing equations of the dynamical system. Using the Nelder-Mead optimization method, PRISM can test the sensitivity of a transmission power system to perturbations of phase of buses. In particular we solve for where 𝜃𝑗 is at a stable starting point for j is not i. This allows us to examine how much the power system is robust to discrete disturbances. We use this to test a system’s margin of stability for each 126 generator over their initial phase state, by choosing a tolerance of disturbance to the frequency. In figures 83-85, we see how much phase shift is allowed in the initial phase state, before the frequency of one of the buses surpasses hertz. Notice that a disturbance to generator 1 (largest generator) effects all the other buses quite significantly, figure 83, while on the other hand, a disturbance to generator 3 (smallest generator) effects the other buses to a minimal extent, figure 85. CASE 9 GEN EFFECT GEN 1 GEN 2 GEN 3 LOAD 5 26.07% 17.69% 12.22% LOAD 6 23.96% 12.57% 15.07% LOAD 8 10.56% 24.69% 19.87% Figure 83: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF CASE 9 GEN 1 GEN 2 GEN 3 GEN RESPONSE 6.94965396 6.83054275 6.61142876 CASE 9 GEN EFFECT GEN 1 LOAD 5 26.07% LOAD 6 23.96% LOAD 8 10.56% Figure 84: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF 127 CASE 9 GEN 1 GEN RESPONSE 6.94965396 Figure 85: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 1 CASE 9 GEN EFFECT GEN 2 LOAD 5 17.69% LOAD 6 12.57% LOAD 8 24.69% 128 CASE 9 GEN 2 GEN RESPONSE 6.83054275 Figure 86: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF SAFETY IS TESTED ON THE INITIAL PHASE OF GENERATOR 2 CASE 9 GEN EFFECT GEN 3 LOAD 5 12.22% LOAD 6 15.07% LOAD 8 19.87% 129 CASE 9 GEN 3 GEN RESPONSE 6.61142876 Figure 87: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF 130 Figure 88: IEEE 39-Bus Oneline Diagram. Table 22: Generator sensitivity analysis for IEEE 39-bus system without any topology changes. CASE 39 GEN 30 GEN 31 GEN 32 GEN 33 GEN 34 GEN 35 GEN 36 GEN 37 GEN 38 GEN 39 LOAD 1 3.9% 23.19% 18.23% 3.97% 4.81% 4.57% 5.22% 9.87% 8.58% 74.07% LOAD 3 12.25% 9.55% 8.7% 4.46% 3.31% 5.3% 4.69% 17.8% 9.6% 16.93% LOAD 4 4.65% 21.98% 21.43% 5.07% 3.24% 5.51% 4.38% 6.36% 6.68% 20.98% LOAD 7 3.25% 36.24% 26.47% 5.03% 4.0% 5.47% 5.18% 9.15% 10.21% 29.51% LOAD 8 3.09% 32.05% 22.39% 4.33% 3.51% 4.11% 4.6% 8.09% 9.23% 33.11% LOAD 9 1.64% 7.95% 6.77% 2.81% 3.42% 3.2% 3.68% 5.99% 7.19% 53.9% LOAD 12 0.21% 1.08% 1.42% 0.47% 0.47% 0.51% 0.52% 0.53% 0.99% 1.2% LOAD 15 2.95% 8.78% 9.44% 11.51% 7.85% 14.36% 13.1% 8.16% 6.59% 11.1% LOAD 16 2.05% 4.88% 5.24% 12.05% 8.9% 15.83% 13.04% 6.24% 5.96% 7.23% LOAD 18 4.87% 7.61% 6.64% 5.53% 4.81% 8.0% 8.35% 15.02% 9.82% 11.34% LOAD 20 1.3% 6.16% 4.19% 37.41% 59.82% 9.3% 12.3% 3.22% 7.74% 10.77% LOAD 21 1.58% 5.15% 4.3% 10.02% 8.3% 35.27% 27.28% 4.82% 5.75% 7.98% LOAD 23 1.39% 5.94% 4.39% 7.84% 9.2% 41.54% 46.44% 3.59% 6.77% 10.01% LOAD 24 1.91% 4.76% 4.89% 10.56% 8.48% 19.15% 15.35% 5.72% 5.8% 7.11% LOAD 25 13.38% 8.92% 7.01% 3.29% 3.1% 3.8% 4.19% 56.08% 16.24% 17.51% LOAD 26 3.48% 5.81% 4.34% 3.79% 2.94% 4.36% 4.29% 14.72% 35.05% 8.48% LOAD 27 4.42% 5.94% 5.05% 5.16% 4.03% 7.56% 6.94% 14.86% 26.20% 9.44% LOAD 28 2.26% 6.05% 4.95% 3.57% 3.07% 3.29% 3.27% 7.97% 89.16% 8.06% LOAD 29 2.16% 7.12% 5.17% 3.17% 3.53% 3.32% 3.75% 7.74% 121.1% 9.44% 131 Figure 89: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF Table 23: SIMULATION OF THE WSCC 9-BUS SYSTEM BY PRISM, WHERE THE MARGIN OF CASE 39 GEN 30 GEN 31 GEN 32 GEN 33 GEN 34 GEN 35 GEN 36 GEN 37 GEN 38 GEN 39 RESP. 12.7 9.65 10.07 10.84 10.61 10.98 10.18 10.53 8.18 9.02 Using PRISM we can analyze what happens to the sensitivity from changes to the system. If we add a powerline from bus 29 to 22 then we see a small improvement in the sensitivity from generator 38, however, if we add a powerline from bus 29 to 1 then we see a much larger improvement in the sensitivity from generator 38, see Table 24. Table 24: IEEE 39 BUS SYSTEM WITH POWERLINE ADDED FROM BUS 1 TO 29 CASE39 GEN30 GEN31 GEN32 GEN33 GEN34 GEN35 GEN36 GEN37 GEN38 GEN39 LOAD 3 -0.1% -0.1% 0.% -0.1% 0.% 0.% 0.% -0.2% 1.1% -2.8% LOAD 4 -0.4% 0.5% 0.2% -0.1% 0.% 0.% 0.% -0.1% 1.7% -3.1% LOAD 7 -0.5% 0.8% 1.1% -0.3% 0.% 0.4% 0.% -1.% 2.6% -2.3% LOAD 8 -0.6% 0.8% 1.% -0.2% 0.% 0.% 0.% -0.9% 2.2% -3.2% LOAD 12 0.% 0.1% 0.1% 0.% 0.% 0.% 0.% 0.% 0.% -0.2% LOAD 15 0.% 0.2% 0.% 0.% 0.% 0.% 0.% 0.1% -0.4% -1.5% LOAD 16 0.1% -0.1% 0.% 0.% 0.% 0.% 0.% 0.1% -0.3% -0.5% LOAD 18 0.1% -0.1% 0.% 0.% 0.% 0.% 0.% 0.% 0.1% -1.% LOAD 20 0.% 0.% 0.1% 0.% 0.% 0.% 0.% 0.% 0.% -0.9% 132 LOAD 21 0.% 0.% 0.% 0.% 0.% 0.% 0.% 0.% -0.3% -0.6% LOAD 23 0.% 0.% 0.% 0.% -0.1% 0.% 0.% 0.% -0.1% -0.9% LOAD 24 0.% 0.% 0.% 0.% 0.% 0.% 0.% 0.1% -0.2% -0.6% LOAD 25 -0.2% -0.4% -0.3% 0.% -0.1% 0.% 0.% -0.2% 0.9% -2.7% LOAD 26 0.% 0.8% 0.7% 0.3% 0.1% 0.3% 0.1% -0.1% -3.% 3.1% LOAD 27 -0.1% 0.4% 0.8% 0.2% -0.1% 0.1% -0.1% -0.6% -2.6% 1.8% LOAD 28 0.3% 2.6% 3.3% 0.4% 0.5% 0.1% 0.% 1.% -8.6% 3.7% LOAD 29 0.4% 2.8% 3.6% 0.4% 0.2% 0.1% 0.3% 1.% -16.2% 2.4% 4.7.4. Performance As PRISM just models the dynamics of the swing equations, it brings speed of simulation and analysis. Our PRISM model takes a common data file of a power grid and gives a dynamical system of the grid based on the swing equations to compute the phase and frequency of the buses over time. The main value of PRISM is that it is evaluated at extreme speeds, typically faster than real life, even for power grids up to 1,000 nodes large. The size of the PRISM model parameters grows quadratically, where a 4k grid was 2GB to store in RAM. PRISM is written entirely in Python, see the table below. Table 25: PRISM computational runtime and complexity System Peak RAM Parameter Elapsed File Elapsed Bus/Generator File Size Rate Size/Buses Rate/Buses Count Squared Squared 9-Bus 3-Gen 56.71 MB 8 KB 0.0042 0.09881 KB 51.9 microseconds seconds per Buses2 per Buses2 39-Bus 10-Gen 56.93 MB 139 KB 0.0051 0.09139 KB 3.35 microseconds seconds per Buses2 per Buses2 118-Bus 19-Gen 58.42 MB 963 KB 0.0108 0.0692 KB 0.776 seconds per Buses2 microseconds per Buses2 300-Bus 69-Gen 65.83 MB 3.8 MB 0.0443 0.0432 KB 0.492 seconds per Buses2 microseconds per Buses2 4000-Bus 360-Gen 1895.4 MB 895.4 MB 9.36 0.0573 KB 0.585 seconds per Buses2 microseconds per Buses2 133 5. CYBER-PHYSICAL UNCERTAINTY QUANTIFICATION In this section, an uncertainty quantification (UQ) methodology was developed specific to cyber- physical systems and tested upon a variety of the InterGraph-CPS techniques. 5.1. Developing Initial UQ Methodology Plan A direct outcome of the team engagement and project activities is SwRI’s increased understanding of each of the four cyber-physical interdependency modeling approaches currently being pursued by the Sandia and Texas A&M University (TAMU) researchers along with the CPS emulation environment that links SCEPTRETM and Real-Time Digital Simulation (RTDS) of power systems. These approaches will comprise the InterGraph-CPS architecture – K-shell analysis, ecological network analysis (ENA), dynamic mode decomposition (DMD), and synchronous motor (SM) modeling. This integration is shown in Figure 90. Figure 90: SCEPTRE-RTDS CPS emulation environment and four InterGraph-CPS modeling approaches to capture cyber-physical interdependencies. Initially, SwRI began drafting Verification, Validation, and Uncertainty Quantification (VVUQ) documents for each of the InterGraph-CPS modeling approaches. However, through iteration, it was determined that a UQ plan for the SCEPTER-RTDS emulation environment was more suitable at this stage of the program to further guide development of InterGraph-CPS. For example, if the key outputs are understood and the key sensitivities of cyber-physical interactions are quantified, InterGraph-CPS can be guided towards approaches that adequately capture the same phenomena. Thus, the goal of the UQ plan is to quantify the sensitivities of both cyber and physical nodal outputs 134 in the 9-bus grid architecture (Figure 91) under different disturbance scenarios as simulated in the CPS emulation environments. Figure 91: 9-bus system detailing the 3 generators, 9 busses, 21 relays, and 3 load shedding devices. First, the LDRD team analyzed four different scenarios – a baseline (S0), a denial of service (DoS) (S1), an outage on Line 6-9 and generator 1 (G-1) plus a DoS (S2), and then the S2 emulation plus a mitigated DoS (S3). Each scenario was executed 3 different times only, differing by the instantiation times of the RTDS server and cyber-attack due to manual starts. Those analyses were completed in July 2023. Time-dependent discrepancy analysis, variability, and recovery in each nodal output solution is the target result to be generated, with decisions about key features to capture and study being made as a result. Next, for the initial UQ plan, G-1 performance reduction was studied and designed to be coupled with different DoS on load shedding stations 5, 6, or both. The team additionally thought to impose a communication delay (1s, 10s, and 30s) with a 10% coefficient of variation (CoV) on the delay parameter. The goal was to perform local sensitivity analysis via a finite difference scheme to determine the nodes and quantity of interest (QoI) that are being pushed to their limits, and to isolate the G-1 performance reduction and DoS case to take forward. Through discussion with additional LDRD team members, the G-1 performance reduction scheme was thought to be too coarse to be meaningful, largely because the sensitivity to the communication delay was not well-understood. Further, the communication delay would likely encounter a threshold due to the time-phased packet protocol used in the emulation. In lieu of doing analyses to discover the protocol threshold, it was determined to parameterize the packet loss percentage on Bus 5 and Bus 6. Therefore, the design of experiment of the next set of analyses to be executed is repeating S2 and S3, however with packet loss percentages incrementally being changed between 10% to 100% in 135 the emulation. These changes constitute the basis of the updated UQ plan. One expectation that the system can be stressed such that “time to system crash” can be tracked and studied as a QoI when load shedding functions are compromised due to cyber-physical attack. 5.2. Testing of InterGraph-CPS Approach Within Emulation To kick off this task, the design of experiment for packet loss percentages in the 9-bus system was executed by the SCEPTRE analysis team. One gap identified by the collective team is the post- processing of the SCEPTRE emulation data for both the 9-bus system and the eventual Emera microgrid simulations. SwRI will use their experience to assist in developing a Python framework structure for team members to plug in future time-series interdependency analyses. Python is preferred, because the data collection and aggregation from the SCEPTRE emulation results are performed by a Sandia-developed Python code. 5.2.1. UQ Studies around Ecological Network Analysis (ENA) Researchers at TAMU constructed a candidate cyber topology for the WSCC 9-bus system in which a single control center communicates with three substations where the 21 physical relays are present (Figure 92). Figure 92: WSCC 9-bus cyber topology with three sub-stations and one control center communicating with the 21 physical relays. Several network health metrics are of interest to the LDRD team that signal network vitality – Degree of System Order (DoSO), average mutual information (AMI), ascendency (ASC), conditional entropy (Hc-), total system overhead (TSO), evenness (E), Shannon index (H), development capacity (DC), 136 and ecological fitness function (Reco). As Reco is a composite metric derived from the other metrics, including DoSO, the immediate interest is looking at the sensitivities of R eco and DoSO to specific flow values in TAMU’s ENA model for the WSCC 9-bus system. Figure 93 depicts DoSO vs. Reco for six different off-nominal packet flow scenarios relevant to the WSCC 9-bus system. Note that the baseline normal operations scenario is not depicted, which has a network health value combination of Reco = 0.35 and DoSO = 0.49, placing it within the window of vitality. Figure 93: Degree of System Order (DoSO) vs Ecological Fitness (Reco) for six candidate 9-bus packet flow scenarios developed by TAMU. The relays and routers in the 9-bus system all send information in the form of packets, with the ENA model containing 56 non-zero values for different packet count flow between cyber-cyber and cyber-physical elements. Figure 94 shows the cyber packet flow matrix for normal operations and highlights specific groupings of non-zero values. In the UQ study, variability was added to these 56 flow parameters to ascertain which individual, or groups of networked assets contribute the most to variability in both Reco and DoSO. Figure 94: ENA cyber packet flow matrix for normal operations 137 5.2.1.1. Defining and Propagating Uncertainties Two methods of defining uncertainties around the 56 flow values were investigated. First, a change to the nominal value was chosen randomly from a uniform distribution bounded by -1 and 1, denoted as U (-1,1). This was chosen so that the flow values of 1 would not be negative when modified by the additive U (-1,1). The other scheme was adding a 10% coefficient of variation (CoV) to each flow value such that the change to the nominal value was chosen randomly from a normal distribution with zero mean and 10% of the nominal value, denoted as N (0, 10% CoV). This was done to reflect that nodes with higher flow values may vary more while still ensuring a low probability of being less than zero. Both sampling schemes produced similar results in terms of uncertainty propagation, however, they differ in subsequent sensitivity studies which will be discussed. Figure 95 depicts the distributions of computed DoSO and Reco values for the nominal system after propagating 10,000 Monte Carlo samples of the 21-packet flow through the model to the network health metrics. Figure 96 contextualizes this variation relative to the window of vitality, in which the normal operating scenario exhibits robustness to variability by staying within the window of vitality despite variation in the system parameters. Note that, in the nominal system, uncertainty doesn’t seem important in assessing DoSO. However, for off-nominal system cases like TAMU’s “Control to Relay, DS2” case, the considered uncertainty will give us a finite probability of being in or out of the window of vitality. Figure 95: Distributions of DoSO (left) and R-eco (right) after uncertainty propagation with 10% CoV applied to each packet flow value in the nominal operating condition. 138 Figure 96: Propagated uncertainty around DoSo and Reco relative to the window of vitality for the nominal operating condition. 5.2.1.2. Variance-Based Sensitivity Analysis For the normal operating scenario, a variance-based global sensitivity analysis (GSA) around DoSO was conducted to isolate which of the 56 varied packet flows contributed the most to variance in DoSO for both the U(-1,1) and 10% CoV studies. Variance-based sensitivity analyses are an important tool for UQ studies to determine where greater fidelity in the model hierarchy is needed, or data is explicitly needed. For example, if the flow packets between relays are determined to be important in this coarse representation of the network, this can help the LDRD team guide data collection in both future emulation scenarios and downstream microgrid studies with the Emera system as well as increase fidelity around those nodal representations in the model hierarchy. Further, an open question is still which data streams on cyber nodes are important to provide to the InterGraph-CPS modeling hierarchy. While the uncertainty propagation analyses using the U(-1,1) and 10% CoV sampling schemes produced similar results in terms of variation predicted in Reco and DoSO, the GSA analyses produced quite different results in identifying the most important flow relationships. These differences primarily highlight the need for characterizing the statistical distributions and variations of packet flow variation that may occur in different parts of the cyber network. Figure 97 shows the first order sensitivity indices for each of the 56 flow parameters when the U(- 1,1) scheme is used. These results highlight that the two-way flow between switches at each substation (SubA1_SW↔SubA2_SW, SubB1_SW ↔SubB2_SW, SubC1_SW↔SubC2_SW) are contributing nearly 10% each, accounting for approximately 60% of the total variation in DoSO. Decisions based on this analysis, under the U(-1,1) variation assumption, would conclude to carefully monitor and control communication between switches. Figure 98 shows the first order sensitivity indices for each of the 56 flow parameters when the 10% CoV sampling scheme is used. These results highlight that DoSO is most sensitive to the one-way communication from the control center Ethernet switch (CC_CW) to router 0 (r0) as well as the control center human machine interface (HMI) to the CC_CW. Each is contributing approximately 20-22% each to the total variation in DoSO, accounting for approximately 42% of the total variation 139 in DoSO. Decisions based on this analysis, under the 10% CoV variation assumption, would conclude to carefully monitor the CC_SW →r0 and the HMI →CC_SW flow paths. Figure 97: First order sensitivity indices for 56 flow parameters to DoSO using the U(-1,1) variation scheme. Figure 98: First order sensitivity indices for 56 flow parameters using the 10 % CoV variation scheme. 5.2.2. UQ Studies around Dynamic Mode Decomposition (DMD) SwRI and SNL have additionally discussed applying UQ methods around the DMD modeling approach used for event detection and impact prediction. In a DMD analysis, modal decomposition is performed on scenario output results over a window of prescribed number of time-steps (e.g. 100 time-steps). The 32 outputs considered thus far are the voltages and frequencies of 8 Phasor Measurement Units (PMUs) through time over approximately 150 seconds (i.e. 9000 timesteps at 60 Hz) of a cyber-physical disruption scenario without mitigation as depicted in Figure 99. 140 Figure 99: PMU frequencies (top row) and voltages (bottom 3 rows) for the 8 PMUs in the 9-bus system for a cyber-physical disruption scenario without mitigation. The normalized eigenvalues and eigenvectors of the outputs computed over a given window indicate whether the dynamic system is stable or unstable if the eigenvalues fall inside or outside of a unit circle, respectively. Figure 100 depicts two eigenvalues λ on the real number line Re{λ} unit circle where the eigenvalue of a stable system falls within the unit circle (green point) and the eigenvalue of an unstable system falls outside of the unit circle in the unstable region (red point). Further, even if the system is stable, there are limit states that are to be upheld for system safety (i.e. voltage is never less than 95% of the max voltage). Figure 100: Assessment of system stability using eigenvector values relative to the unit circle. 141 A probabilistic approach was conceptualized in which the eigenvalues of a standard scenario (i.e., no cyber-physical attack) could be used to compare against new data. This conceptual approach is depicted in the progression from Figure 101 to Figure 102. In Figure 101, 10 Mode 1 and Mode 2 eigenvalues from a trusted, stable scenario (red) are shown against a new scenario (yellow). In this case, the yellow Mode 1 and Mode 2 eigenvalues are stable, however, they are unlike the eigenvalues observed before. Therefore, we would like to evaluate the probability that the current scenario is part of the set of trusted operations. Figure 102 demonstrates a 2-dimensional probabilistic model trained on the existing data. The probabilistic model is a kernel density estimate (KDE) in which, say a Gaussian kernel density is placed around each data point with a fixed bandwidth. Then the probability values in 2D space around each data point are summed and normalized to create a 2D probability estimate across the entire space. Therefore, KDEs are sensitive to both the bandwidth parameter and the number of data points considered. In Figure 102, the bandwidth is fixed at 0.2 and the resulting KDE is comprised of clusters of high probabilities in between areas with near-zero probability. In Figure 103, however, the bandwidth is increased to 0.5 making the KDE more diffuse across space. Further, there may be a non-zero probability outside the unit circle which would suggest the utility of a hierarchical model – first assessing stability with the unit circle, then assessing probability with a KDE. Figure 101: Mode 1 and Mode 2 eigenvalues on real and imaginary axes from an existing scenario (red) and a new scenario (yellow). Figure 102: 2D probability contours from KDEs with a bandwidth of 0.2 for Mode 1 and Mode 2 eigenvalues from an existing scenario (red) compared to new scenario (yellow). 142 Figure 103: 2D probability contours from KDEs with a bandwidth of 0.5 for Mode 1 and Mode 2 eigenvalues from an existing scenario (red) compared to new scenario (yellow). For the cyber-physical attack scenario of the 9-bus system, first the data for each output is divided into timestep windows with a fixed window size. The timestep window is fixed at 1000 timesteps for all 32 outputs shown in Figure 105. Then, the Mode 1 and Mode 2 eigenvectors and values across that time window are determined using the DMD methodology. As an example, Figure 104 shows the frequency output of PMU1 and the resulting real and imaginary components of the Mode 1 and Mode 2 eigenvectors at the 9 checkpoints across the ~9000 timesteps and in relation to the unit circle. Figure 104 shows that the imaginary component of Mode 2 in the later time steps pushes the system out of the stable unit circle. Figure 104: (Left) PMU1 frequency output on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues on the unit circle scale on the left axis. The red shaded area depicts the -1 to 1 range on the left-axis to represent the un Figure 105 shows the DMD analysis for the 32 voltages and frequency outputs of the 9-bus system during the cyber-physical attack scenario. First, the frequencies and first two voltages of PMU3 and PMU4 (3rd and 4th columns in Figure 105) show that these outputs are stable across the entire 143 scenario. Therefore, they were not sensitive to the cyber-physical attack. Likewise, the frequencies of PMU5 and PMU6 were also not sensitive as they are stable across the scenario. In contrast, all the other outputs display some instability across the time history, specifically through the imaginary component of Mode 2. In the frequency output of PMU1, PMU2, PMU7 and PMU8, the instability is later in the time history after the cyber-physical attack. However, for the voltage output, the instability is flagged in the imaginary component of Mode 2 before the cyber-attack. Further investigation and understanding of this scenario are needed to ascertain if this phenomenon is real or simply numerical, as the implication would be that instability of voltage could be an early indicator of cyber-physical attack. Figure 105: 32 voltages and frequency outputs for the 8 PMUs on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues on the unit circle scale on the left axis. The red shaded area depicts the -1 to 1 range on the left-axis While useful for early-warning algorithms and stability analysis, the eigenvalues for this scenario are not varied enough to warrant the previously outlined 2D KDE probability model. However, recall that the previous DMD analysis was performed on output time histories subdivided with a fixed time window of 1000 timesteps. In Figure 106, the timestep window was reduced by an order of magnitude to 100 timesteps, meaning the Mode 1 and Mode 2 eigenvalues computed were increased by an order of magnitude to ~90. First, it is observed that the change in time window length did not change that the later timesteps of PMU1 are unstable from the imaginary component of Mode 2. However, the earlier timesteps before the attack event also now show instability in the real component of Mode 2. This sensitivity of the stability analysis to the time step window size warrants further investigation. In addition, treating all the 32 outputs as independent does not reflect reality. Additional work will begin grouping based on PMU and load groups to both reduce sensitivity to small numerical variations on the eigenvalue decompositions and subsequent stability analysis. Once appropriately grouped, the eigenvalues computed may contain the anticipated variation to train the conceptual 2D KDE probability model. Otherwise, the assessment of probability of being a trusted operational scenario will be simplified to a binary analysis of being exact within a small numerical tolerance. 144 Figure 106: (Left) PMU1 frequency output on the right-axis and real (R) and imaginary (I) components of the Mode 1 and Mode 2 eigenvalues from a time window of 100 timesteps on the unit circle represent the unit circle range identified on the right axes. (Right) T Another WSCC 9-bus disturbance scenario was analyzed with DMD in which a disturbance occurs at 300s. The 9 phases and frequencies across the 600s time history are shown below in Figure 107. The DMD approach was applied to each node to decompose its time history into eigenvalues and eigenvectors with which to reconstruct the system time history for forecasting purposes. Figure 107: Phases and frequencies of each node in the WSCC 9-bus system during a disturbance scenario A coarse sensitivity study was performed on the comparative stability assessment and the resulting eigenvalue and eigenvectors when using the first 300 seconds of data (i.e. data not containing the disruption), the first 350 seconds of data (i.e., including the disruption and partial recovery) and data from 300 to 350 seconds (i.e. data during the disruption and partial recovery). It was observed that 145 each time span considered for training had a different optimal number of modes used to capture the total variance, with 11 modes retained being the maximum (Figure 108, top row). For the system prediction architecture, it may be useful to homogenize the number of modes retained across all busses and training time histories considered, and the eigenvalues for 11 modes across all cases are shown (Figure 108, bottom row). Figure 108: (Top row) Optimal numbers of retained modes for Bus 1 across different training data time spans along with (bottom row) the eigenvalues if 11 modes are retained across all time spans Finally, an example of forecasting with the DMD model trained on 0 to 350 seconds, retaining 11 modes, and predicting 30 seconds into the future is shown in Figure 109. It is observed that there is an immediate discrepancy between the time-series collected data and the prediction through 400 seconds during recovery. This indicates that the trained model is not sufficient to forecast and would indicate that the model needs to be retrained on new data. This is also expected when disturbance scenarios and system behaviors that have never been observed before cause discrepancies in predictions and the in-time observed data. 146 Figure 109: An example of DMD forecasting and resulting prediction discrepancy. It is at this point in the InterGraph CPS framework that an update of the system topology (i.e. ENA) or the time-series decomposition (i.e. DMD) needs to occur and constitutes the scenario in which an integrated cyber-physical framework will be most useful. For conceptualizing the integration of probabilistic methods on system forecasting with DMD and fault analysis with other InterGraph-CPS methods such as ENA, the decision-making pathways that could be explored further are depicted in Figure 110. At the first level, the DMD model training has variable system input that leads to uncertainty in the eigenvalues and eigenvectors that are used to assess the system stability. As a result, the probability of system stability (pstable) can be assessed and is a finite value between 0 and 1. The probability of the system being unstable is therefore 1-pstable, and is likewise a finite value between 0 and 1. Both predictive pathways of stability and instability can be exercised in decision-making framework to evaluate the validity of the continued use of the DMD model for forecasting and/or make real-time assessments of time, consequence, and postulate corrective moves with graph-based methods such as ENA. Figure 110: DMD training and decision-making pathways regarding forecasting, stability, and the dynamically changing system. 147 Proposed UQ studies in the future to be pursued with DMD analysis using the remaining SCEPTRE emulation results are the following: 1. Appropriate grouping of 9-bus outputs and verification of the number of time-steps sufficient for event detection and stability analysis to possibly justify reducing the number of time-steps considered in the window. As a primary utility of this analysis is for event detection, reducing the time window required may improve detection time. 2. Conditional on a stable system, a forward propagation analysis with DMD to assess the probability of maintaining at least 95% of the max voltage is of interest. 3. A Bayesian model calibration analysis may be of interest for a system that is well-described and captured by a DMD model. For example, if a DMD model predicts that in the next time-step or series of time-steps, a series of outputs are a certain value, and they do not fall within a tolerance of those values, an inverse analysis could be performed. The inverse analysis would ascertain the distribution of eigenvalues that would lead to the observed output combinations, and if those eigenvalues are highly likely to be outside of the unit circle, then a disturbance has been detected, the system is unstable, and an updated DMD model can make predictions about what will happen within the next critical series of time-steps. 148 6. INTEGRATED CYBER-PHYSICAL INTERDEPENDENCY INIGHTS USING THE START FRAMEWORK Each of the InterGraph-CPS techniques within the START framework, summarized in Figure 111, provides unique perspectives into the cyber-physical interdependencies within a system during normal operation and disturbances. Figure 111: Summary of the START framework. To explore the integrated benefits and insights gained from leveraging the diverse InterGraph-CPS techniques, we can explore the integration of a CPS graph/structural technique, ENA, and a CPS temporal data technique, autoencoder or AE. Sections 4.4.3.1.2 and 4.6.1 detail the results of each technique for the Emera Microgrid use-case and attack scenarios (MITM and DoS) described in Section 3.3.4.3. The results for the individual techniques can be summarized as follows: ENA and Emera Microgrid ENA provides insight into Emera microgrid structure during normal operation: low network redundancy with very specialized connections (each nanogrid box with each of the generation and loads) but with a single back-up power option as well as interconnections between nanogrids. This implies complex structural cycling due to this shared energy and interconnections and overall strong cyclicity yet high nodal vulnerability. These ENA results indicate that the Emera microgrid has resilient structural design with nodal failure points but many back-up interconnections for power delivery. LSTM AE and Emera Microgrid 149 LSTM AE enables classification of events, detecting abnormalities and the type of event (cyber, physical or combined cyber-physical). The AE is robust to data outliers and we can use the normal data distributions to select uniquely suitable thresholds for detection. Specifically, we can tune the threshold based on preferences on false positives vs. false negatives. Overall, AE is giving us understanding if the cyber nodes or physical nodes or both cyber-physical nodes are being impacted. It also provides us thresholds and understanding of cyber and physical features and their relationships during normal operation and attack. Thus, for joint graph/structural and temporal analysis of the Emera microgrid disturbances using InterGraph-CPS techniques ENA and LSTM AE, we learn that: • ENA helps us understand which links are fragile and the strong cyclicity of the system due to the microgrid distributed nature. This can imply denser cyber-physical data flows due to bidirectional linkages, especially since the system is smaller and very distributed. • AE helps us understand that in normal operation that the data can have outliers even though the system is cyclical • AE reinforces that the system is fragile, as the attacks are detected quickly and impact many of the cyber data features. Overall, the insights gained from the ENA and AE analysis enables understanding of the Emera Microgrid normal operation cyber-physical characteristics and relationships as well as predict impact of failures or disturbances when attacks occur. This impact can then be validated by the real-time anomaly detection performed by AE and which cyber-physical features were impacted. In the next section, discussion of this integration and UQ approach is provided. 6.1. Overall START Integration Takeaways To effectively analyze and increase understanding of cyber-physical systems, the START framework was developed with a diverse set of graph and temporal techniques. This included graph/structural techniques such as ecological network analysis (ENA), bipartite/tripartite analysis, Node2Vec, DeepWalk, and other graph-based analysis techniques and temporal analysis techniques such as dynamic mode decomposition (DMD), synchronous motor models (PRISM), long short-term memory autoencoder (LSTM AE) neural networks. In this paper, we focused on a subset of these START techniques, specifically ENA, DeepWalk, Node2Vec, and LSTM AE to explore the integration between their results and analyze insights gained. This report presented results for each of these START techniques for two different use-cases: WSCC 9-Bus and Emera Microgrid cyber-physical emulation environments where a variety of cyber- physical disturbances were implemented. For each use-case, we analyzed the individual results and discussed the integrated insights gained into detection the cyber-physical anomaly and predicting consequences. Specifically, we found that the LSTM AE classification and detection of an anomaly could be cross-correlated to increase confidence in detection while both ENA and DeepWalk/Node2Vec provided insight on structural fragility and changes and predict which nodes could be attacked next and their risk levels. These multi-faceted perspectives on the same event analysis provide unique understanding on how the cyber-physical system is being impacted, how individual cyber-physical nodes are linked and impacted, and how impact could propagate through the CPS. This is demonstrated with presented results where: 150 1. ENA provides overall characterization of the CPS and its strength/fragility, 2. AE detects and classifies CPS anomalies, 3. ENA confirms the AE detection with changes in strength/fragility metrics, 4. DeepWalk and Node2Vec predicts what nodes could be impacted next and their risk levels, 5. AE detection can confirm predicted node compromise, 6. ENA and DeepWalk/Node2Vec structural and risk analysis updates, 7. and START analysis continues. Overall, this continuous START analysis provides both structural and temporal analysis to not only understand changes in the CPS interdependencies but also the physical system dynamics. Additionally, START enabled comparison of unique resilience and data flow characteristics between transmission power systems and microgrid power systems. In future work, we will fully integrate START techniques such as DMD to assess the fragility of dynamic physical system and predict dynamic consequences and PRISM which provides dynamic system characterization. Larger power systems with diverse configuration and autonomous control (e.g., energy management systems) will be explored in future START studies, in addition to application to different critical infrastructure domains. This comprehensive framework, with structural and temporal (both cyber-physical and purely physical) analysis provides situational awareness across the CPS and flexible granularity for different planning, operation, and response activities. 6.2. Integrated UQ Analysis for Cyber-Physical Systems Here we present a conceptualization of an integrated model-based framework for tracking, identification, restructuring, assessment, and recovery of cyber-physical systems under attack using several of the data-driven modeling approaches explored in this program. Respectively, LSTM AE, ENA, and DMD represent specific instances of a real-time ML-based tracking model (LSTM AE), a graph-based structural model (ENA), and a time-dependent system stability model (DMD). Consider a notional cyber-physical system composed of both cyber and physical assets as shown Figure 1Error! Reference source not found.12. Here, there is an underlying CPS structure at each time instant that results in tracked CPS system output data at each time instant. There may be a relationship between observed outputs in the system, however, for the purposes of the graphic in Figure 112 they are depicted as independent. Figure 112: Notional relationship between true system structure and observed outputs in a cyber- physical system 151 The LSTM AE model, having been trained on the normal operational structure and output cycles of the system, is used for real-time system tracking where it is reconstructing the tracked output fields at ti and documenting reconstruction error. If the assumption that the system is behaving as expected is valid (i.e. the true structure at ti is the structure that the LSTM AE model was trained on), then the reconstruction error of each output data stream or its features at ti will be small. Figure 113 depicts this scenario in which the structure and behavior of the system is well-known and well- understood. Figure 113. LSTM AE reconstruction producing small errors because the true structure and assumed structure are equivalent Conversely, consider a cyber-physical attack scenario in which the true cyber-physical structure departs from the normal operating data on which the LSTM AE model was trained. Depicted in Figure 1144, there is a quantifiable reconstruction error which triggers a sequence of actions: 1. Identify the subset of likely nodes affected using the LSTM AE model to update the known system structure 2. Analyze the candidate CPS structures with ENA to identify subsequent modes of fragility and subsequent cyber-physical attack 3. Make probabilistic forecasting predictions with dynamic modal decomposition (DMD). Figure 114. LSTM AE reconstruction producing triggering errors because the true structure and assumed structure are not equivalent Once the system has been identified as being in an off-nominal state, a system stability and probabilistic forecasting model of the system dynamics can be trained. The goal is to analyze system stability and time and consequence of a system crash. Here, we use DMD as an example approach to provide these two functions where over a small window of time post-event, Δtt, the eigenvectors and values of the tracked outputs are computed and used to compute system stability (Figure 115). 152 Figure 115.CPS UQ integration with DMD, LSTM AE, and ENA approaches As shown in Figure 115, the system stability analysis can be probabilistic itself. If stable, the DMD model can be used to forecast the system state over the next time instances. Meanwhile, the LSTM AE model is continuing to monitor the system to identify when the reconstruction error changes. This event would indicate that the DMD model needs to be retrained. If unstable, DMD can estimate the time and consequence of the system crash, however, the structural assessments by ENA may illuminate pathways of corrective moves to be made toward system recovery or restabilizing. This conceptual sequence of modeling steps and interactions can be formalized in future work where the integration of the LSTM AE, ENA, and DMD approaches can be implemented. 153 7. CONCLUSIONS AND NEXT STEPS To effectively analyze and increase understanding of cyber-physical systems, the InterGraph-CPS framework was developed with a diverse set of graph and temporal techniques. This included graph/structural techniques such as ecological network analysis, bipartite/tripartite analysis, Node2Vec, DeepWalk, and other graph-based analysis techniques and temporal analysis techniques such as dynamic mode decomposition, synchronous motor models (PRISM), long short-term memory autoencoder neural networks. Each of these techniques were tested with different real-time cyber-physical emulation environments to validate individual and integrated cyber-physical interdependency insight and detection of anomalies. This included the development of a new emulation capability based on a real microgrid system where both the communication and control network and microgrid power system design was based on the real-world system. A variety of cyber-physical disturbance events were implemented in these emulation environments and the InterGraph-CPS technique results were used to understand the system impact, cyber-physical interdependency impact, and predicted impact pathways. Lastly, a cyber-physical UQ analysis approach was developed to assess each of the interdependency results to help decision-makers (e.g., cyber defenders and system operators) understand detection results and take mitigation action, if needed. Overall, InterGraph-CPS demonstrated successful integrated cyber-physical interdependency analysis within relevant emulation environments. For next steps, the team would like to explore automating the InterGraph-CPS technique integrated analysis and testing with real field data or field sites with key industry partners. 154 REFERENCES [1] “Information Technology – Operational Technology Convergence Guidebook,” Electric Power Research Institute, 2018. [2] G. Cheng, Y. Lin, A. Abur, A. Gómez-Expósito, and W. Wu, “A Survey of Power System State Estimation Using Multiple Data Sources: PMUs, SCADA, AMI, and Beyond,” IEEE Trans. Smart Grid, vol. 15, no. 1, pp. 1129–1151, 2024, doi: 10.1109/TSG.2023.3286401. [3] A. Reyna et al., “Towards the Design of Grid Cyber-Physical Integrated Security Operations Center Visualizations,” in 2024 IEEE Kansas Power and Energy Conference (KPEC), 2024, pp. 1–6. doi: 10.1109/KPEC61529.2024.10676242. [4] B. J. Pierre, D. Krofcheck, K. Munoz-Ramos, and B. Arguello, “A Framework to Model and Analyze Electric Grid Cascading Failures to Identify Critical Nodes,” in 2022 17th International Conference on Probabilistic Methods Applied to Power Systems (PMAPS), 2022, pp. 1–6. doi: 10.1109/PMAPS53380.2022.9810579. [5] B. Falahati and Y. Fu, “A study on interdependencies of cyber-power networks in smart grid applications,” in 2012 IEEE PES Innovative Smart Grid Technologies (ISGT), 2012, pp. 1–8. doi: 10.1109/ISGT.2012.6175593. [6] S. Wang, X. Gu, S. Luan, and M. Zhao, “Resilience analysis of interdependent critical infrastructure systems considering deep learning and network theory,” Int. J. Crit. Infrastruct. Prot., vol. 35, p. 100459, 2021, doi: https://doi.org/10.1016/j.ijcip.2021.100459. [7] F. Petit and D. Verner, “Critical Infrastructure Interdependencies Assessment,” World Secur. Rep., vol. 2016, Nov. 2016, [Online]. Available: https://www.osti.gov/biblio/1400396 [8] X. Liu, B. Chen, C. Chen, and D. Jin, “Electric power grid resilience with interdependencies between power and communication networks – a review,” IET Smart Grid, vol. 3, no. 2, pp. 182–193, 2020, doi: https://doi.org/10.1049/iet-stg.2019.0202. [9] J. D. Taft and A. S. Becker-Dippmann, “The Emerging Interdependence of the Electric Power Grid & Information and Communication Technology,” Pacific Northwest National Laboratory (PNNL), Richland, WA (United States), Aug. 2015. doi: 10.2172/1221500. [10] “Power Grid and Communication Interdependencies: Key Challenges for Reliable, Resilient Operations,” U.S. Department of Energy Office of Electricity, Sep. 2023. [Online]. Available: https://www.energy.gov/sites/default/files/2023- 10/Electric_Power_Telecommunications_Interdependencies_508.pdf [11] “Sandia Minimega Homepage.” Accessed: Sep. 16, 2025. [Online]. Available: https://www.sandia.gov/minimega/ [12] “SCEPTRE.” Sandia National Laboratories, Aug. 2016. [Online]. Available: https://www.osti.gov/biblio/1376989 [13] “SCEPTRE Documentation Github Repository.” Accessed: Sep. 17, 2025. [Online]. Available: https://sandialabs.github.io/sceptre-docs/ [14] “Sandia Emulytics Homepage.” Accessed: Sep. 16, 2025. [Online]. Available: https://www.sandia.gov/emulytics/ [15] S. Hossain-McKenzie et al., “Adaptive, Cyber-Physical Special Protection Schemes to Defend the Electric Grid Against Predictable and Unpredictable Disturbances,” in 2021 Resilience Week (RWS), 2021, pp. 1–9. doi: 10.1109/RWS52686.2021.9611801. [16] S. Hossain-McKenzie et al., “Harmonized Automatic Relay Mitigation of Nefarious Intentional Events (HARMONIE) - Special Protection Scheme (SPS),” Sandia National Laboratories (SNL- NM), Albuquerque, NM (United States), Sep. 2022. doi: 10.2172/1890265. 155 [17] M. Fisher, “CRADA enables resilient microgrid research between Sandia, Emera Technologies,” Sandia Lab News, Nov. 21, 2018. [Online]. Available: https://www.sandia.gov/labnews/2018/11/21/crada-microgrid/ [18] R. Darbali-Zamora, A. R. R. Dow, F. Palacios, J. D. Flicker, and D. Bauer, “Development of a State-Space Average Nanogrid Model for DC Microgrid Power Management Applications,” in 2023 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), 2023, pp. 1– 5. doi: 10.1109/ISGT51731.2023.10066354. [19] M. J. Reno et al., “Adaptive Protection and Control for High Penetration PV and Grid Resilience (Final Technical Report),” Sandia National Laboratories (SNL-CA), Livermore, CA (United States); Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), Apr. 2024. doi: 10.2172/2382709. [20] “hping3 Github Repository.” Accessed: Sep. 16, 2025. [Online]. Available: https://salsa.debian.org/debian/hping3 [21] “Kali Linux Homepage.” Accessed: Sep. 16, 2025. [Online]. Available: https://www.kali.org/ [22] “Scapy Homepage.” Accessed: Sep. 16, 2025. [Online]. Available: https://scapy.net/ [23] “iperf3 Homepage.” Accessed: Sep. 16, 2025. [Online]. Available: https://software.es.net/iperf/ [24] A. Pinar et al., “Science and Engineering of Cybersecurity by Uncertainty quantification and Rigorous Experimentation (SECURE) (Final Report),” Sandia National Laboratories (SNL- NM), Albuquerque, NM (United States); Sandia National Laboratories (SNL-CA), Livermore, CA (United States), Sep. 2021. doi: 10.2172/1821322. [25] T. pandas development team, pandas-dev/pandas: Pandas. (Aug. 2025). Zenodo. doi: 10.5281/zenodo.16918803. [26] N. Jacobs, S. Hossain-McKenzie, A. Summers, C. B. Jones, B. Wright, and A. Chavez, “Cyber- Physical Observability for the Electric Grid,” in 2020 IEEE Texas Power and Energy Conference (TPEC), 2020, pp. 1–6. doi: 10.1109/TPEC48276.2020.9042494. [27] N. Jacobs et al., “Leveraging graph clustering techniques for cyber-physical system analysis to enhance disturbance characterisation,” IET Cyber-Phys. Syst. Theory Appl., vol. 9, no. 4, pp. 392– 406, 2024, doi: https://doi.org/10.1049/cps2.12087. [28] S. Carmi, S. Havlin, S. Kirkpatrick, Y. Shavitt, and E. Shir, “A model of Internet topology using k-shell decomposition,” Proc. Natl. Acad. Sci., vol. 104, no. 27, pp. 11150–11154, 2007, doi: 10.1073/pnas.0701175104. [29] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 855–864. [30] J. Ha and S. Park, “NCMD: Node2vec-Based Neural Collaborative Filtering for Predicting MiRNA-Disease Association,” IEEEACM Trans Comput Biol Bioinforma., vol. 20, no. 2, pp. 1257–1268, Jul. 2022, doi: 10.1109/TCBB.2022.3191972. [31] H. Zhou, G. Sun, S. Fu, L. Wang, J. Hu, and Y. Gao, “Internet Financial Fraud Detection Based on a Distributed Big Data Approach With Node2vec,” IEEE Access, vol. 9, pp. 43378–43386, 2021, doi: 10.1109/ACCESS.2021.3062467. [32] R. Baembitov, M. Kezunovic, and Z. Obradovic, “Graph Embeddings for Outage Prediction,” in 2021 North American Power Symposium (NAPS), 2021, pp. 1–6. doi: 10.1109/NAPS52732.2021.9654696. [33] Q. Xiao, J. Liu, Q. Wang, Z. Jiang, X. Wang, and Y. Yao, “Towards Network Anomaly Detection Using Graph Embedding,” Comput. Sci. – ICCS 2020, vol. 12140, pp. 156–169, 2020. [34] S. Sun, E. Payne, K. Davis, and N. Jacobs, “Bio-inspired and AI DeepWalk Based Approach to Understand Cyber-Physical Interdependencies of Power Grid Infrastructure,” North American Power Symposium, vol. 55. 2023. 156 [35] M. A. Haque, S. Shetty, C. A. Kamhoua, and K. Gold, “Attack Graph Embedded Machine Learning Platform For Cyber Situational Awareness,” in MILCOM 2022 - 2022 IEEE Military Communications Conference (MILCOM), 2022, pp. 464–469. doi: 10.1109/MILCOM55135.2022.10017837. [36] B. Perozzi, R. Al-Rfou, and S. Skiena, “DeepWalk: online learning of social representations,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, in KDD ’14. ACM, Aug. 2014, pp. 701–710. doi: 10.1145/2623330.2623732. [37] National Institute of Standards and Technology, “Risk Management Guide for Information Technology Systems,” National Institute of Standards and Technology, NIST Special Publication 800–30, 2002. [Online]. Available: https://csrc.nist.gov/pubs/sp/800/30/final [38] National Vulnerability Database, “Common Vulnerability Scoring System v2 Calculator.” [Online]. Available: https://nvd.nist.gov/vuln-metrics/cvss/v2-calculator [39] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space.” 2013. [Online]. Available: https://arxiv.org/abs/1301.3781 [40] L. B. V. de Amorim, G. D. C. Cavalcanti, and R. M. O. Cruz, “The choice of scaling technique matters for classification performance,” Appl. Soft Comput., vol. 133, p. 109924, 2023, doi: https://doi.org/10.1016/j.asoc.2022.109924. [41] J. Reap, “Holistic biomimicry: a biologically inspired approach to environmentally benign engineering.” 2009. [Online]. Available: http://hdl.handle.net/1853/31771 [42] R. E. Ulanowicz, “The dual nature of ecosystem dynamics,” Ecological Modelling, vol. 220, no. 16. pp. 1886–1892, 2009. doi: 10.1016/j.ecolmodel.2009.04.015. [43] A. Layton, “Food Webs: Realizing Biological Inspiration for Sustainable Industrial Resource Networks,” Thesis, 2014. [Online]. Available: http://hdl.handle.net/1853/54307 [44] R. E. Ulanowicz, “Information theory in ecology,” Computers & chemistry, vol. 25, no. 4. pp. 393– 399, 2001. [45] R. E. Ulanowicz, S. J. Goerner, B. Lietaer, and R. Gomez, “Quantifying sustainability: Resilience, efficiency and the return of information theory,” Ecological Complexity, vol. 6, no. 1. pp. 27–36, 2009. doi: 10.1016/j.ecocom.2008.10.005. [46] R. E. Ulanowicz, “Growth and development: ecosystems phenomenology.” Springer Science & Business Media, 2012. [47] N. M. Waser and J. Ollerton, “Plant-pollinator interactions: from specialization to generalization.” University of Chicago Press, 2006. [48] C. Campbell, S. Yang, R. Albert, and K. Shea, “A network model for plant–pollinator community assembly,” Proceedings of the National Academy of Sciences, vol. 108, no. 1. pp. 197–202, 2011. [49] J. Bascompte and P. Jordano, “Mutualistic networks.” Princeton University Press, 2013. [50] W. Dáttilo et al., “Unravelling Darwin’s entangled bank: architecture and robustness of mutualistic networks with multiple interaction types,” Proceedings of the Royal Society B: Biological Sciences, vol. 283, no. 1843. p. 20161564, 2016. [51] S. Saavedra, F. Reed-Tsochas, and B. Uzzi, “A simple model of bipartite cooperation for ecological and organizational networks,” Nature, vol. 457, no. 7228, pp. 463–466, Jan. 2009, doi: 10.1038/nature07532. [52] H. Toju, P. R. Guimaraes, J. M. Olesen, and J. N. Thompson, “Assembly of complex plant– fungus networks,” Nature communications, vol. 5, no. 1. p. 5273, 2014. [53] J. L. Bronstein, “Mutualism.” Oxford University Press, 2015. [54] R. J. Williams and N. D. Martinez, “Simple rules yield complex food webs,” Nature, vol. 404, no. 6774, pp. 180–183, Mar. 2000, doi: 10.1038/35004572. 157 [55] A. Solé-Ribalta, C. J. Tessone, M. S. Mariani, and J. Borge-Holthoefer, “Revealing in-block nestedness: Detection and benchmarking,” Physical Review E, vol. 97, no. 6. p. 062302, 2018. [56] S. J. Whipple, “Analysis of ecosystem structure and function: Extended path and flow analysis of a steady-state oyster reef model,” Ecological Modelling, vol. 114, no. 2–3. pp. 251–274, 1999. doi: 10.1016/S0304-3800(98)00154-9. [57] B. D. Fath, S. E. Jørgensen, B. C. Patten, and M. Straškraba, “Ecosystem growth and development,” BioSystems, vol. 77, no. 1–3. pp. 213–228, 2004. doi: 10.1016/j.biosystems.2004.06.001. [58] R. E. Ulanowicz, “Quantitative methods for ecological network analysis,” Comput. Biol. Chem., vol. 28, no. 5, pp. 321–339, 2004, doi: https://doi.org/10.1016/j.compbiolchem.2004.09.001. [59] J. Wu and R. Jin, “Exploring the impact of symbiotic measures on the evolution of structure and function of the iron and steel industrial symbiosis network,” Journal of Cleaner Production, vol. 273. 2020. doi: 10.1016/j.jclepro.2020.122990. [60] E. M. Payne, S. S. Hossain-Mckenzie, N. Jacobs, K. R. Davis, and A. Layton, “Analyzing Cyber- Physical Modularity and Interdependence Using Bio-Inspired Graph Modeling,” IEEE Access, vol. 12, 2024, doi: 10.1109/ACCESS.2024.3450368. [61] E. Payne, S. Sun, S. Hossain‐McKenzie, N. Jacobs, K. Davis, and A. Layton, “Utilizing Bio- Inspired Metrics for Real-Time Detection of Denial-of-Service Attacks,” Conference on Systems Engineering Research. Springer Nature, 2025. [62] S. Allesina, A. Bodini, and C. Bondavalli, “Ecological subsystems via graph theory: the role of strongly connected components,” Oikos, vol. 110, no. 1. pp. 164–176, 2005. [63] C. F. Dormann, J. Fründ, N. Blüthgen, and B. Gruber, “Indices, graphs and null models: analyzing bipartite ecological networks,” Open Ecol. J., vol. 2, no. 1, 2009. [64] W. Cai, J. Snyder, A. Hastings, and R. M. D’Souza, “Mutualistic networks emerging from adaptive niche-based interactions,” Nature Communications, vol. 11, no. 1. p. 5470, 2020. doi: 10.1038/s41467-020-19154-5. [65] E. Payne, S. Hossain‐McKenzie, N. Jacobs, K. Davis, and A. Layton, “BiOpCPS: A Bio-inspired Method for Resilient CyberPhysical Power System Interface Optimization,” Conference on Systems Engineering Research, vol. Augmented Intelligence in Systems Engineering and Engineered Systems. Springer Nature, 2025. [66] E. Payne, S. Sun, A. Layton, K. Davis, S. Hossain-McKenzie, and N. Jacobs, “Insight Into Cyber Information Flow Interdependencies Using Bio-inspired Topologies,” in 2024 58th Annual Conference on Information Sciences and Systems (CISS), 2024, pp. 1–6. doi: 10.1109/CISS59072.2024.10480172. [67] F. Li, X. Yan, Y. Xie, Z. Sang, and X. Yuan, “A Review of Cyber-Attack Methods in Cyber- Physical Power System,” in 2019 IEEE 8th International Conference on Advanced Power System Automation and Protection (APAP), 2019, pp. 1335–1339. doi: 10.1109/APAP47170.2019.9225126. [68] R. M. Lee, M. J. Assante, and T. Conway, “Analysis of the Cyber Attack on the Ukrainian Power Grid,” Electricity Information Sharing and Analysis Center, Mar. 2016. [69] H. Huang, V. Panyam, M. R. Narimani, A. Layton, and K. R. Davis, “Mixed-Integer Optimization for Bio-Inspired Robust Power Network Design,” in 2020 52nd North American Power Symposium (NAPS), 2021, pp. 1–6. doi: 10.1109/NAPS50074.2021.9449755. [70] S. Hossain-McKenzie et al., “Towards the Characterization of Cyber-Physical System Interdependencies in the Electric Grid,” IEEE Power Energy Conf. Ill. PECI, 2023. [71] H. Huang, K. R. Davis, and H. V. Poor, “An Extended Model for Ecological Robustness to Capture Power System Resilience,” arXiv preprint arXiv:2303.04281. 2023. 158 [72] A. S. Al-Hinai, “Voltage collapse prediction for interconnected power systems.” West Virginia University, 2000. [73] A. Chatterjee, H. Huang, K. R. Davis, and A. Layton, “A Multigraph Modeling Approach to Enable Ecological Network Analysis of Cyber Physical Power Networks,” in 2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), 2021, pp. 239–244. doi: 10.1109/SmartGridComm51999.2021.9631989. [74] S. Blair, H. Banks, J. Linsey, and A. Layton, “Bipartite Network Analysis Utilizing Survey Data to Determine Student and Tool Interactions in a Makerspace,” Virtual Conference: ASEE Conferences, Jul. 2021. doi: 10.18260/1-2--36750. [75] S. Blair, G. Hairston, H. Banks, C. Kaat, J. Linsey, and A. Layton, “Bio-inspired human network diagnostics: Ecological modularity and nestedness as quantitative indicators of human engineered network function,” Syst. Eng., vol. 27, no. 5, pp. 886–898, 2024, doi: https://doi.org/10.1002/sys.21756. [76] M. E. Newman, “Modularity and community structure in networks,” Proc. Natl. Acad. Sci., vol. 103, no. 23, pp. 8577–8582, 2006. [77] C. O. Flores, T. Poisot, S. Valverde, and J. S. Weitz, “BiMat: a MATLAB package to facilitate the analysis of bipartite networks,” Methods in Ecology and Evolution, vol. 7, no. 1. pp. 127–132, 2016. [78] A. James, J. W. Pitchford, and M. J. Plank, “Disentangling nestedness from models of ecological complexity,” Nature, vol. 487, no. 7406, pp. 227–230, 2012. [79] P. Arena, M. Bucolo, A. Buscarino, L. Fortuna, and M. Frasca, “Reviewing Bioinspired Technologies for Future Trends: A Complex Systems Point of View,” vol. 9. 2021. doi: 10.3389/fphy.2021.750090. [80] F. Senghore, E. Campos-Nanez, P. Fomin, and J. S. Wasek, “Using Social Network Analysis to Investigate the Potential of Innovation Networks: Lessons Learned from NASA’s International Space Apps Challenge,” Procedia Computer Science, vol. 28. pp. 380–388, 2014. doi: 10.1016/j.procs.2014.03.047. [81] A. Clifton and G. D. Webster, “An introduction to social network analysis for personality and social psychologists,” Social Psychological and Personality Science, vol. 8, no. 4. pp. 442–453, 2017. [82] S. Bustos, C. Gomez, R. Hausmann, and C. A. Hidalgo, “The dynamics of nestedness predicts the evolution of industrial ecosystems,” PloS One, vol. 7, no. 11, p. e49393, 2012. [83] W. Ulrich, M. Almeida‐Neto, and N. J. Gotelli, “A consumer’s guide to nestedness analysis,” Oikos, vol. 118, no. 1. pp. 3–17, 2009. [84] E. A. Martin, B. Feit, F. Requier, H. Friberg, and M. Jonsson, “Assessing the resilience of biodiversity-driven functions in agroecosystems under environmental change,” in Advances in Ecological Research, vol. 60, Elsevier, 2019, pp. 59–123. [85] M. Almeida‐Neto, P. Guimaraes, P. R. Guimaraes Jr, R. D. Loyola, and W. Ulrich, “A consistent metric for nestedness analysis in ecological systems: reconciling concept and measurement,” Oikos, vol. 117, no. 8. pp. 1227–1239, 2008. [86] A. Layton, B. Bras, and M. Weissburg, “Ecological Principles and Metrics for Improving Material Cycling Structures in Manufacturing Networks,” J. Manuf. Sci. Eng., vol. 138, no. 101002, Jun. 2016, doi: 10.1115/1.4033689. [87] P. H. Warren, “Spatial and Temporal Variation in the Structure of a Freshwater Food Web,” Oikos, vol. 55, no. 3. pp. 299–299, 1989. doi: 10.2307/3565588. [88] M. Korkali, J. G. Veneman, B. F. Tivnan, J. P. Bagrow, and P. D. H. Hines, “Reducing Cascading Failure Risk by Increasing Infrastructure Network Interdependence,” Sci. Rep., vol. 7, no. 1, p. 44499, Mar. 2017, doi: 10.1038/srep44499. 159 [89] S. Katsikeas, P. Johnson, M. Ekstedt, and R. Lagerström, “Research communities in cyber security: A comprehensive literature review,” Comput. Sci. Rev., vol. 42, p. 100431, 2021, doi: https://doi.org/10.1016/j.cosrev.2021.100431. [90] J. E. Sullivan and D. Kamensky, “How cyber-attacks in Ukraine show the vulnerability of the US power grid,” The Electricity Journal, vol. 30, no. 3. pp. 30–35, 2017. [91] C.-C. Sun, A. Hahn, and C.-C. Liu, “Cyber security of a power grid: State-of-the-art,” International Journal of Electrical Power & Energy Systems, vol. 99. pp. 45–56, 2018. doi: 10.1016/j.ijepes.2017.12.020. [92] M. L. L. B. Muthuppalaniappan and K. Stevenson, “Healthcare cyber-attacks and the COVID- 19 pandemic: an urgent threat to global health,” International Journal for Quality in Health Care, vol. 33, no. 1. p. mzaa117, 2021. doi: 10.1093/intqhc/mzaa117. [93] M. A. Mohamed, T. Chen, W. Su, and T. Jin, “Proactive Resilience of Power Systems Against Natural Disasters: A Literature Review,” IEEE Access, vol. 7. pp. 163778–163795, 2019. doi: 10.1109/ACCESS.2019.2952362. [94] E. Hossain, S. Roy, N. Mohammad, N. Nawar, and D. R. Dipta, “Metrics and enhancement strategies for grid resilience and reliability during natural disasters,” Applied Energy, vol. 290. p. 116709, 2021. doi: 10.1016/j.apenergy.2021.116709. [95] C. Fachkha, E. Bou-Harb, A. Keliris, N. D. Memon, and M. Ahamad, “Internet-scale Probing of CPS: Inference, Characterization and Orchestration Analysis,” Network and Distributed System Security Symposium. 2017. [96] E. Bou-Harb, W. Lucia, N. Forti, S. Weerakkody, N. Ghani, and B. Sinopoli, “Cyber Meets Control: A Novel Federated Approach for Resilient CPS Leveraging Real Cyber Threat Intelligence,” IEEE Communications Magazine, vol. 55, no. 5. pp. 198–204, 2017. doi: 10.1109/MCOM.2017.1600292CM. [97] S. M. Dibaji, M. Pirani, D. B. Flamholz, A. M. Annaswamy, K. H. Johansson, and A. Chakrabortty, “A systems and control perspective of CPS security,” Annual Reviews in Control, vol. 47. pp. 394–411, 2019. doi: https://doi.org/10.1016/j.arcontrol.2019.04.011. [98] L. Cao, X. Jiang, Y. Zhao, S. Wang, D. You, and X. Xu, “A Survey of Network Attacks on Cyber-Physical Systems,” IEEE Access, vol. 8. pp. 44219–44227, 2020. doi: 10.1109/ACCESS.2020.2977423. [99] M. Amin, F. F. M. El-Sousy, G. A. A. Aziz, K. Gaber, and O. A. Mohammed, “CPS Attacks Mitigation Approaches on Power Electronic Systems With Security Challenges for Smart Grid Applications: A Review,” IEEE Access, vol. 9. pp. 38571–38601, 2021. doi: 10.1109/ACCESS.2021.3063229. [100] M. S. Mahmoud, M. M. Hamdan, and U. A. Baroudi, “Modeling and control of Cyber- Physical Systems subject to cyber attacks: A survey of recent advances and challenges,” Neurocomputing, vol. 338. pp. 101–115, 2019. doi: 10.1016/j.neucom.2019.01.099. [101] S. Watts, C. F. Dormann, A. M. Martín González, and J. Ollerton, “The influence of floral traits on specialization and modularity of plant–pollinator networks in a biodiversity hotspot in the Peruvian Andes,” Annals of Botany, vol. 118, no. 3. pp. 415–429, 2016. [102] V. Panyam, H. Huang, K. Davis, and A. Layton, “Bio-inspired design for robust power grid networks,” Appl. Energy, vol. 251, p. 113349, 2019, doi: https://doi.org/10.1016/j.apenergy.2019.113349. [103] H. Huang, Z. Mao, A. Layton, and K. R. Davis, “An Ecological Robustness Oriented Optimal Power Flow for Power Systems’ Survivability,” IEEE Trans. Power Syst., vol. 38, no. 1, pp. 447–462, 2023, doi: 10.1109/TPWRS.2022.3168226. 160 [104] T. Dave and A. Layton, “Designing ecologically-inspired robustness into a water distribution network,” J. Clean. Prod., vol. 254, p. 120057, 2020, doi: https://doi.org/10.1016/j.jclepro.2020.120057. [105] H. Shiroyama, M. Yarime, M. Matsuo, H. Schroeder, R. Scholz, and A. E. Ulrich, “Governance for sustainability: knowledge integration and multi-actor dimensions in risk management,” Sustainability Science, vol. 7. pp. 45–55, 2012. [106] D. E. Alexander, “Resilience and disaster risk reduction: An etymological journey,” Natural Hazards and Earth System Sciences, vol. 13, no. 11. pp. 2707–2716, 2013. doi: 10.5194/NHESS-13- 2707-2013. [107] D. Baird and R. E. Ulanowicz, “The Seasonal Dynamics of the Chesapeake Bay Ecosystem,” Ecological Monographs, vol. 59, no. 4. pp. 329–364, 1989. [108] R. R. Christian and C. R. Thomas, “Network analysis of nitrogen inputs and cycling in the Neuse River estuary, North Carolina, USA,” Estuaries, vol. 26, no. 3. pp. 815–828, 2003. doi: 10.1007/BF02711992. [109] S. Allesina and R. E. Ulanowicz, “Cycling in ecological networks: Finn’s index revisited,” Comput. Biol. Chem., vol. 28, no. 3, pp. 227–233, 2004, doi: https://doi.org/10.1016/j.compbiolchem.2004.04.002. [110] B. D. Fath, “Structural food web regimes,” Ecol. Model., vol. 208, no. 2, pp. 391–394, 2007, doi: https://doi.org/10.1016/j.ecolmodel.2007.06.013. [111] B. D. Fath and G. Halnes, “Cyclic energy pathways in ecological food webs,” Ecol. Model., vol. 208, no. 1, pp. 17–24, 2007, doi: https://doi.org/10.1016/j.ecolmodel.2007.04.020. [112] B. D. Fath, U. M. Scharler, R. E. Ulanowicz, and B. Hannon, “Ecological network analysis: network construction,” Ecol. Model., vol. 208, no. 1, pp. 49–55, 2007, doi: https://doi.org/10.1016/j.ecolmodel.2007.04.029. [113] D. K. Gattie, N. N. Kellam, and H. J. Turk, “Informing ecological engineering through ecological network analysis, ecological modelling, and concepts of systems and engineering ecology,” Ecological Modelling, vol. 208, no. 1. pp. 25–40, 2007. doi: 10.1016/j.ecolmodel.2007.04.027. [114] S. Allesina and S. Tang, “Stability criteria for complex ecosystems,” Nature, vol. 483, no. 7388. pp. 205–208, 2012. [115] D. Tang, L. Wang, W. Gu, W. Yuan, and D. Tang, “Modelling of Bio-inspired Manufacturing System,” in Proceedings of the 6th CIRP-Sponsored International Conference on Digital Enterprise Technology, G. Q. Huang, K. L. Mak, and P. G. Maropoulos, Eds., Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 1165–1174. [116] V. Panyam, H. Huang, K. Davis, and A. Layton, “An ecosystem perspective for the design of sustainable power systems,” Procedia CIRP, vol. 80. pp. 269–274, 2019. doi: 10.1016/j.procir.2018.12.005. [117] H. Huang, Z. Mao, V. Panyam, A. Layton, and K. R. Davis, “Ecological Robustness- Oriented Grid Network Design for Resilience Against Multiple Hazard,” IEEE Transactions on Power Systems, vol. 39, no. 1. pp. 416–428, 2023. doi: 10.1109/TPWRS.2023.3262501. [118] M. De Domenico et al., “Mathematical Formulation of Multilayer Networks,” Physical Review X, vol. 3, no. 4. p. 041022, 2013. doi: 10.1103/PhysRevX.3.041022. [119] S. Sun et al., “A graph embedding‐based approach for automatic cyber‐physical power system risk assessment to prevent and mitigate threats at scale,” IET Cyber‐Physical Systems: Theory & Applications. 2024. 161 [120] S. Macfadyen et al., “Do differences in food web structure between organic and conventional farms affect the ecosystem service of pest control?,” Ecology letters, vol. 12, no. 3. pp. 229–238, 2009. [121] C. J. Melián, J. Bascompte, P. Jordano, and V. Krivan, “Diversity in a complex ecological network with two interaction types,” Oikos, vol. 118, no. 1. pp. 122–130, 2009. [122] M. J. O. Pocock, D. M. Evans, and J. Memmott, “The robustness and restoration of a network of ecological networks,” Science, vol. 335, no. 6071. pp. 973–977, 2012. [123] T. D. Hackett, A. M. C. Sauve, N. Davies, D. Montoya, J. M. Tylianakis, and J. Memmott, “Reshaping our understanding of species’ roles in landscape‐scale networks,” Ecology Letters, vol. 22, no. 9. pp. 1367–1377, 2019. [124] N. Shinohara, K. Uchida, and T. Yoshida, “Contrasting effects of land‐use changes on herbivory and pollination networks,” Ecology and Evolution, vol. 9, no. 23. pp. 13585–13595, 2019. [125] R. M. May, “Will a Large Complex System be Stable?,” Nature, vol. 238, no. 5364, pp. 413– 414, Aug. 1972, doi: 10.1038/238413a0. [126] A. Mougi and M. Kondoh, “Diversity of interaction types and ecological community stability,” Science, vol. 337, no. 6092. pp. 349–351, 2012. [127] A. M. C. Sauve, C. Fontaine, and E. Thébault, “Structure–stability relationships in networks combining mutualistic and antagonistic interactions,” Oikos, vol. 123, no. 3. pp. 378–384, 2014. [128] M. Lurgi, D. Montoya, and J. M. Montoya, “The effects of space and diversity of interaction types on the stability of complex ecological networks,” Theoretical ecology, vol. 9. pp. 3–13, 2016. [129] C. McWilliams, M. Lurgi, J. M. Montoya, A. Sauve, and D. Montoya, “The stability of multitrophic communities under habitat loss,” Nature Communications, vol. 10, no. 1. p. 2322, 2019. [130] K. R. S. Hale, F. S. Valdovinos, and N. D. Martinez, “Mutualism increases diversity, stability, and function of multiplex networks that integrate pollinators into food webs,” Nature Communications, vol. 11, no. 1. p. 2182, 2020. doi: 10.1038/s41467-020-15688-w. [131] D.-G. Virginia and K. Sonia, “The structure and robustness of tripartite ecological networks,” bioRxiv. p. 2021.10.05.463170, 2021. doi: 10.1101/2021.10.05.463170. [132] T. Dave and A. Layton, “Extending the Use of Bio-Inspiration for Water Distribution Networks to Urban Settings,” in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, 2020, p. V008T08A001. [133] T. Dave and A. Layton, “Bio-inspired design for resilient water distribution networks,” Procedia CIRP, vol. 80, pp. 275–280, 2019, doi: https://doi.org/10.1016/j.procir.2019.01.020. [134] A. Chatterjee, R. Malak, and A. Layton, “Exploring System of Systems Resilience vs. Affordability Trade-Space Using a Bio-Inspired Metric,” Journal of Computing and Information Science in Engineering, vol. 21. 2021. doi: 10.1115/1.4050288. [135] S. Warrington and A. Layton, “Ecosystem guidance for the incorporation of renewable utilities in a multi-use campus network,” PLoS ONE, vol. 17, no. 5 May. 2022. doi: 10.1371/journal.pone.0267431. [136] E. Payne, H. Wagner, and A. Layton, “Resilience and Sustainability in Certified Green Buildings: Applying Ecosystem Concepts to Aid in More Dynamic Green Communities,” in IDETC-CIE2023, Volume 5: 28th Design for Manufacturing and the Life Cycle Conference (DFMLC), Aug. 2023. doi: 10.1115/DETC2023-114798. [137] E. Payne and A. Layton, “Integrating Machine Learning Into the Design of Green Building Systems,” in IDETC-CIE2024, Volume 2B: 44th Computers and Information in Engineering Conference (CIE), Aug. 2024. doi: 10.1115/DETC2024-143653. 162 [138] A. Chatterjee, C. Brehm, and A. Layton, “Evaluating benefits of ecologically-inspired nested architectures for industrial symbiosis,” Resources, Conservation and Recycling, vol. 167. p. 105423, 2021. doi: 10.1016/j.resconrec.2021.105423. [139] A. H. Amer, “Voltage collapse prediction for interconnected power systems,” Morgantown west Virinia. 2000. [140] S. Griffin, “Habitat loss and fragmentation disrupt plant-pollinator networks.” 2020. [Online]. Available: https://conservationcorridor.org/digests/2020/05/plant-pollinator- networks/ [141] C. Brehm and A. Layton, “Nestedness of eco‐industrial networks: Exploring linkage distribution to promote sustainable industrial growth,” Journal of Industrial Ecology, vol. 25, no. 1. pp. 205–218, 2021. doi: 10.1111/jiec.13057. [142] P. M. Anderson, C. F. Henville, R. Rifaat, B. Johnson, and S. Meliopoulos, “Power system protection.” John Wiley & Sons, 2022. [143] S. H. Horowitz, A. G. Phadke, and C. F. Henville, “Power system relaying.” John Wiley & Sons, 2022. [144] K. E. Havens, “Predator-Prey Relationships in Natural Community Food Webs,” Oikos, vol. 68, no. 1. 1993. doi: 10.2307/3545316. [145] E. Payne and A. Layton, “grENA: Ecological network analysis to assess LEED green buildings’ sustainability,” J. Ind. Ecol., Jul. 2025, doi: 10.1111/jiec.70083. [146] A. Layton, B. Bras, and M. Weissburg, “Designing Sustainable Manufacturing Networks: The Role of Exclusive Species in Achieving Ecosystem-Type Cycling,” in IDETC-CIE2017, Volume 4: 22nd Design for Manufacturing and the Life Cycle Conference; 11th International Conference on Micro- and Nanosystems: ASME, Aug. 2017. doi: 10.1115/DETC2017-68334. [147] J. Giri, “Real-time grid management: Keeping the lights on!,” IEEE Power and Energy Magazine, vol. 21, no. 3. pp. 51–60, 2023. [148] “Cyber-Informed Transmission Planning: Roadmap for Integrating Cyber Security into Transmission Planning Activities.” North American Electric Reliability Corporation, 2023. [Online]. Available: https://www.nerc.com/comm/RSTC_Reliability_Guidelines/ERO_Enterprise_Whitepaper_C yber_Planning_2023.pdf [149] “Multiyear Plan for Energy Sector Cybersecurity.” U.S. Department of Energy, Mar. 2018. [Online]. Available: https://www.energy.gov/ceser/articles/doe-multiyear-plan-energy- cybersecurity [150] A. Chatterjee and A. Layton, “Mimicking nature for resilient resource and infrastructure network design,” Reliab. Eng. Syst. Saf., vol. 204, 2020, doi: https://doi.org/10.1016/j.ress.2020.107142. [151] H. Huang, A. Chatterjee, A. Layton, and K. Davis, “An Investigation into Ecological Network Analysis for Cyber-Physical Power Systems,” in 2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), 2021, pp. 252– 257. doi: 10.1109/SmartGridComm51999.2021.9632310. [152] H. Huang, Z. Mao, V. Panyam, A. Layton, and K. R. Davis, “An Ecological Robustness- Oriented Approach for Power System Network Expansion,” ArXiv, vol. abs/2107.06178, 2021. [153] B. Bollobás, “Modern graph theory,” vol. 184. Springer Science & Business Media, 1998. [154] E. W. Zegura, K. L. Calvert, and M. J. Donahoo, “A quantitative comparison of graph-based models for Internet topology,” IEEE/ACM Transactions on networking, vol. 5, no. 6. pp. 770–783, 2002. [155] R. Margalef, “Information theory in ecology.” 1973. 163 [156] B. Fath, “Quantifying economic and ecological sustainability,” Ocean & Coastal Management, vol. 108. 2014. doi: 10.1016/j.ocecoaman.2014.06.020. [157] I. Nijs and J. Roy, “How important are species richness, species evenness and interspecific differences to productivity? A mathematical model,” Oikos, vol. 88, no. 1. pp. 57–66, 2000. [158] A. Chao and T.-J. Shen, “Nonparametric estimation of Shannon’s index of diversity when there are unseen species in sample,” Environmental and ecological statistics, vol. 10, no. 4. pp. 429– 443, 2003. [159] T. M. DeJong, “A comparison of three diversity indices based on their components of richness and evenness,” Oikos. pp. 222–227, 1975. [160] J. A. Camargo, “On measuring species evenness and other associated parameters of community structure,” Oikos. pp. 538–542, 1995. [161] C. P. H. Mulder, E. Bazeley‐White, P. G. Dimitrakopoulos, A. Hector, M. Scherer‐Lorenzen, and B. Schmid, “Species evenness and productivity in experimental plant communities,” Oikos, vol. 107, no. 1. pp. 50–63, 2004. [162] P. Wlazlo et al., “Man‐in‐the‐middle attacks and defence in a power system cyber‐physical testbed,” IET Cyber‐Physical Systems: Theory & Applications, vol. 6, no. 3. pp. 164–177, 2021. [163] V. S. Rajkumar, M. Tealane, A. Ştefanov, A. Presekal, and P. Palensky, “Cyber Attacks on Power System Automation and Protection and Impact Analysis,” in 2020 IEEE PES Innovative Smart Grid Technologies Europe (ISGT-Europe), 2020, pp. 247–254. doi: 10.1109/ISGT- Europe47291.2020.9248840. [164] L. Shi, Q. Dai, and Y. Ni, “Cyber–physical interactions in power systems: A review of models, methods, and applications,” Electric power systems research, vol. 163. pp. 396–412, 2018. [165] W. Li, L. Xie, Z. Deng, and Z. Wang, “False sequential logic attack on SCADA system and its physical impact analysis,” Computers & Security, vol. 58. pp. 149–159, 2016. [166] C. L. Schuba, I. V. Krsul, M. G. Kuhn, E. H. Spafford, A. Sundaram, and D. Zamboni, “Analysis of a denial of service attack on TCP,” in Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No.97CB36097), 1997, pp. 208–223. doi: 10.1109/SECPRI.1997.601338. [167] R. E. Ulanowicz, R. D. Holt, and M. Barfield, “Limits on ecosystem trophic complexity: insights from ecological network analysis,” Ecology Letters, vol. 17, no. 2. pp. 127–136, 2014. doi: 10.1111/ele.12216. [168] R. B. Ash, “Information theory.” Courier Corporation, 2012. [169] P. Schmid and J. Sesterhenn, “Dynamic mode decomposition of numerical and experimental data,” Bulletin of the American Physical Society, Sixty-First Annual Meeting of the APS Division of Fluid Dynamics., Jul. 2008. Accessed: Aug. 01, 2023. [Online]. Available: http://meetings.aps.org/link/BAPS.2008.DFD.MR.7 [170] M. C. Muñoz, M. A. Peñalba, and A. E. S. González, “Analysis of aggregated load consumption forecasting in short, medium and long term horizons using Dynamic Mode Decomposition,” Energy Rep., vol. 12, pp. 1000–1013, 2024, doi: https://doi.org/10.1016/j.egyr.2024.06.040. [171] G. Fragkos, L. Blakely, S. Hossain-McKenzie, A. Summers, and C. Goes, “Cyber-Physical Data Fusion & Threat Detection with LSTM-Based Autoencoders in the Grid,” in 2024 IEEE Kansas Power and Energy Conference (KPEC), 2024, pp. 1–6. doi: 10.1109/KPEC61529.2024.10676133. [172] S. Chen and W. Guo, “Auto-Encoders in Deep Learning—A Review with New Perspectives,” Mathematics, vol. 11, no. 8, 2023, doi: 10.3390/math11081777. 164 [173] G. Filatrella, A. H. Nielsen, and N. F. Pedersen, “Analysis of a power grid using a Kuramoto-like model,” Eur. Phys. J. B, vol. 61, no. 4, pp. 485–491, Feb. 2008, doi: 10.1140/epjb/e2008-00098-8. [174] T. Nishikawa and A. E. Motter, “Comparative analysis of existing models for power-grid synchronization,” New J. Phys., vol. 17, 2015, [Online]. Available: https://api.semanticscholar.org/CorpusID:119176758 [175] “PYPOWER Github Repository.” Accessed: Sep. 15, 2025. [Online]. Available: https://github.com/rwl/PYPOWER [176] R. D. Zimmerman, C. E. Murillo-Sanchez, and R. J. Thomas, “MATPOWER: Steady-State Operations, Planning and Analysis Tools for Power Systems Research and Education,” IEEE Trans. Power Syst., vol. 26, no. 1, pp. 12–19, Feb. 2011. 165 This page left blank 166 DISTRIBUTION Email—Internal Name Org. Sandia Email Address Brian Pierre 8813 bjpierr@sandia.gov Derek Hart 5621 derhart@sandia.gov Technical Library 1911 sanddocs@sandia.gov 167 This page left blank 168 Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.