Evolutionary and Coevolutionary Multi-Agent Design Choices and Dynamics
Erik Hemberg, Eric Liu, Lucille Fuller, Stephen Moskal, Una-May O’Reilly · 2025
Read Evolutionary and Coevolutionary Multi-Agent Design Choices and Dynamics by Erik Hemberg, Eric Liu, Lucille Fuller, Stephen Moskal, Una-May O’Reilly (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05534.txt.
Introduction
Erik Hemberg, Eric Liu, Lucille Fuller, Stephen Moskal, and Una-May O'Reilly study how teams of cyber agents learn to compete in a 2025 study. Agent controllers can be written in different forms, and evolution can improve them in different ways. One algorithm tested here uses a language model to help mutate controllers.
A cybersecurity scenario pits trained teams against a fixed opponent and against a second team that evolves alongside them. Grammatical evolution expresses each controller in a code-like form from a grammar. Comparing combinations of representation and algorithm shows clear trade-offs in team performance. The results guide designers of adaptive defense and attack agents.
Catalog record
| CREATOR | Erik Hemberg, Eric Liu, Lucille Fuller, Stephen Moskal, Una-May O’Reilly |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05534
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evolutionary and Coevolutionary Multi-Agent Design Choices and Dynamics Erik Hemberg, Eric Liu, Lucille Fuller, Stephen Moskal, Una-May O’Reilly Abstract . We investigate two representation alternatives for the controllers of teams of cyber agents. We combine these controller representations with different evolutionary algorithms, one of which introduces a novel LLM-supported mutation operator. Using a cyber security scenario, we evaluate agent learning when one side is trained to compete against a side that does not evolve and when two sides coevolve with each other. This allows us to quantify the relative merits and tradeoffs of representation and algorithm combinations in terms of team performance. Our versions of grammatical evolution algorithms using grammars that allow a controller to be expressed in code-like logic can achieve the best team performance. The scenario also allows us to compare the performance impact and dynamics of coevolution versus evolution under different combinations. Across the algorithms and representations, we observe that coevolution reduces the performance highs and lows of both sides while it induces fluctuations on both sides. In contrast, when only one-side is optimized, performance peaks are higher and is more sustained than when both sides are optimized with coevolution. 1. Introduction With larger scale and closer-to-reality scenarios, complex challenges aim to push the limits of autonomous agent training. The CybORG gym, an engine that models cyber networks, threats, and security response, has supported several such challenges ( Standen et al. 2021 ) . A recent example is the Cyber Cage Challenge 4 (CCC4). It expects cyber agents to be designed by means of machine learning. It presents a fictitious scenario (with stochastic variants) of two competing nations. One nation (red) seeks to disrupt the server which handles the other’s (blue’s) mission. The server is within a complex cyber environment with four networks, five security zones defined over the networks, and various firewall defenses. In the scenario there are multiple "blue" agents comprising a team that are positioned in different zones. Blue must defend the environment by taking actions against a multi-phase attack controlled by red. The CCC4 scenario poses the fundamental problem of learning a sequential decision controller for an agent or team of agents. The agent’s controller, with only partial observability must decide two things, one simulation step at a time: the host on which it will target its next action, and what that action will be. Additionally, the multiple agents must work in coordination while being positioned in different zones. Their actions realistically model the defender’s preventative, monitoring and repairing options. CybORG provides easy performance comparison among controllers and teams with a point-based system of rewards for different accomplishments. While CCC4 requires multi-agent training, this training is confined to blue teams. Because we are interested in training autonomous red and blue cyber agents, we extend CCC4 in two ways. Red teams can also be trained and, second, more information about the scenario is made accessible through new observation functions. This leads to the question of how to best train red and blue teams when they compete against each other. To start, we consider controller design in terms of representation and algorithm choices. These are critical to agent training and agent performance. A designer must decide what part of the solution, in this case the agent’s controller, to hard code versus what part they ask the Evolutionary Algorithm (EA) to optimize for them. They can opt for largely predetermined solution logic that sets up a smaller and simpler search space for optimization. Or they can hard code only basic solution logic and provide the EA with powerful logic elements that must be composed into a controller. This implies the EA must optimize in a more complex and larger search space. Choosing between these two extremes or finding a design choice between them is an open question in the challenge setting we have created. To address this, we explore agent controller representations and their variants which have different search space size and expressivity tradeoffs. Our research question is: What is the best combination of algorithm and representation that achieves the best performing red and blue teams? Our interest in competing cyber agents that take offensive and defensive sides motivates a related interest in adversarial dynamics. Defenses in cyber systems are mostly optimized to repel specific attackers whose activities are well characterized, see, for example, the CCC4 Challenge itself. We seek optimizations that assume attackers will evolve and adapt counter measures to defensive measures. Though two-sided competition and optimization are more challenging in complexity, modeling them is important to better understand the dynamics ( Popovici et al. 2012 ; Krawiec and Heywood 2016 ) . Our second research question centers on this model of adversarial dynamics. How do team performance levels change over time as the two sides adapt while competing? Does peak performance of either side diminish when both sides co-optimize versus when they are optimized independently? Can we expect to see more or less rewards going to each team? To answer this second set of research questions, our baseline is when one side (red or blue) evolves and optimizes with combinations of representations and evolutionary algorithms, while the other (blue or red respectively) does not. We then compare baseline performance with the performance of our combinations of representations and coevolutionary algorithms. Our contributions follow: Around design choices, one of our two representations is an action selection matrix, indexed by state and action. For each state, the likelihood of each action needs to be optimized. This probability can be optimized by a genetic algorithm (GA) ( Goldberg 1989 ) or evolutionary strategy (ES) ( Back 1996 ) . Alternatively, we use grammars that are optimized by grammatical evolution (GE) ( O’Neill and Ryan 2001 ) . We explore progressively more powerful grammars which sets up progressively larger search spaces. Each of these algorithms, GA, ES and GE, is also integrated into their own coevolutionary algorithm variant. • We compare designs with the same controller being referenced by all members of a team versus designs where each team member references its own controller when the grammar variants are used. We find the use of one controller per team can sometimes perform as well as multiple controllers per team. • We introduce a version of GE that incorporates a large language model as part of its mutation operator. We compare this algorithm to GE in both one-side learning (with 1 GE algorithms) and two-sided learning (2 GE algorithms connected by competition) and conduct a small study into the LLM supported operator. We observe that the LLM operator is very sensitive to the LLM-model and prompt used. • Given the action selection matrix representation, we compare the performance under continuous or discrete parameter optimization. We find that the discrete parameter optimization is a better choice since it works for both cases and is a better choice for red. • We compare the grammar variants using the same algorithm, asking how their expressiveness impacts team performance. We find that the grammar expressiveness can influence the performance and that a larger grammar can perform well with the correct terminals. Around adversarial dynamics we run four comparisons. In each we compare the training dynamics of a population’s mean performing and best performing