Playing Hex and Counter Wargames using Reinforcement Learning and Recurrent Neural Networks
Guilherme Palma, Pedro A. Santos, João Dias · 2025
Read Playing Hex and Counter Wargames using Reinforcement Learning and Recurrent Neural Networks by Guilherme Palma, Pedro A. Santos, João Dias (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13918.txt.
Introduction
Guilherme Palma, Pedro Santos, and João Dias build a system that learns to play hex-and-counter wargames, in work from 2025. Such games simulate real wars on hex maps, with stacked units, varied ground, and many choices per turn. Classic board game AI methods strain under that scale.
Their system joins a memory-equipped neural network with a self-play learning method in the AlphaZero line. New ways to encode the game state and the legal actions fit the wargame setting. With modest training, the system played typical scenarios well and carried its skill across ground types and setups. Tests of larger maps probe how far it scales. The code is open for others to build on.
Catalog record
| CREATOR | Guilherme Palma, Pedro A. Santos, João Dias |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13918
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Playing Hex and Counter Wargames using Reinforcement Learning and Recurrent Neural Networks Guilherme Palma, Pedro A. Santos, João Dias Abstract Hex and Counter Wargames are adversarial two-player simulations of real military conflicts requiring complex strategic decision-making. Unlike classical board games, these games feature intricate terrain/unit interactions, unit stacking, large maps of varying sizes, and simultaneous move and combat decisions involving hundreds of units. This paper introduces a novel system designed to address the strategic complexity of Hex and Counter Wargames by integrating cutting-edge advancements in Recurrent Neural Networks with AlphaZero, a reliable modern Reinforcement Learning algorithm. The system utilizes a new Neural Network architecture developed from existing research, incorporating innovative state and action representations tailored to these specific game environments. With minimal training, our solution has shown promising results in typical scenarios, demonstrating the ability to generalize across different terrain and tactical situations. Additionally, we explore the system’s potential to scale to larger map sizes. The developed system is openly accessible, facilitating continued research and exploration within this challenging domain. † † paperid: 123 † † orcid: 0000-0002-1369-0085 † † orcid: 0000-0002-1653-1821 1 Introduction The pursuit of perfect or super-human proficiency in board games has long been a benchmark for assessing the capabilities of contemporary knowledge and technologies in artificial intelligence. Historically, significant milestones have been achieved in games such as Chess, Othello, and Backgammon, where AI models have either matched or surpassed the best human players [ Rosenbloom(1982) , Silver and Anthon(2022) , Silver et al.(2018)Silver, Hubert, Schrittwieser, Antonoglou, Lai, Guez, Lanctot, Sifre, Kumaran, Graepel, Lillicrap, Simonyan, and Hassabis ] . Additionally, considerable advancements have also been noted in strategic negotiation games like Diplomacy [ Bakhtin et al.(2022)Bakhtin, Brown, Dinan, Farina, Flaherty, Fried, Goff, Gray, Hu, Jacob, Komeili, Konath, Kwon, Lerer, Lewis, Miller, Mitts, Renduchintala, Roller, Rowe, Shi, Spisak, Wei, Wu, Zhang, and Zijlstra ] . Despite these successes, there are still numerous even more complex games that still elude good AI-based solutions. This paper explores the application of advanced AI techniques to a category of board games known as Hex and Counter Wargames. These are adversarial two-player simulations of real military conflicts that often involve complex scenarios requiring strategic decision-making. Unlike classical board games such as Chess and Go, these games feature intricate terrain/unit interactions, unit stacking, large maps of varying sizes, and simultaneous move and combat decisions involving hundreds of units. Our methodology is based on Alpha Zero [ Silver et al.(2018)Silver, Hubert, Schrittwieser, Antonoglou, Lai, Guez, Lanctot, Sifre, Kumaran, Graepel, Lillicrap, Simonyan, and Hassabis ] , but with a substantially different Neural Network architecture, adapted to much lower computing requirements and the specific needs of these types of games. The main contributions of our work are: • Network architecture - A dual head fully convolutional Recurrent Neural Network, adapted to this type of games, that can be used with boards of varying dimensions and needs less training resources. • State and action representations - Novel representations which can be utilized on many Hex and Counter Wargames, and that can be extended to accommodate specific game needs. • Generalization results - Significant results on the capacity of this type of architecture to generalize to different board and unit configurations serve as important groundwork for future research. • Game environment and AlphaZero implementation - We developed an environment for Hex and Counter Wargames, and a asynchronous AlphaZero implementation, based on the original paper, with extra features adapted to these specific games. This framework is freely available for use by other researchers 1 1 1 The code is available at the Github repository: https://github.com/guilherme439/NuZero . 1.1 Problem Setting Hex and Counter Wargames are adversarial boardgames played on hexagonally tiled maps that use different cardboard markers and counters to represent the several units and resources available on the map. These games are usually, but not limited to, simulations of historic military conflicts, commonly including large maps that represent multiple regions or countries, and rulebooks with many dozens or sometimes hundreds of pages (e.g.: [ Bannerman et al.(2023)Bannerman, Manka, Manka, and Potts. , Essig(2016) , Simonitch(2022) ] ). All the training and testing throughout this study was conducted under a sub-set of rules inspired by the Standard Combat Series ruleset [ Essig(2015) ] , that are common to a large amount of Hex and Counter Wargames and which we will now describe. The games are played on game boards that can be configured to have any desired dimensions. Each tile on the game board has a type of terrain with 3 attributes: attack modifier, defence modifier and movement cost. Additionally, certain tiles on the map yield victory points. Play lasts for a predetermined number of turns. After these turns, the player who controls the highest percentage of victory point tiles emerges victorious. Throughout the game, players place units on the map that they receive as reinforcements. Units are characterized by their three features: attack strength, defence strength and movement allowance. Groups of units can stack in the same tile up to a predefined limit. Beyond that, there are no stacking group restrictions, meaning that all units can stack with each other. Combat can be conducted between units on adjacent hexes. When a group of units attacks another, the total attack and defence strength of each group of units is calculated to determine the combat’s result. For the attackers, each unit multiplies its attack strength by the terrain’s attack modifier and adds it to the group’s total. The same thing happens in the group of defending units, using the defence strength and modifiers. The group that obtains the lower combined strength loses their strongest unit. If a draw happens, both groups lose a unit. Each player’s turn consists of two distinct phases: the movement and combat phases. Both the movement and combat phases comprise two sub-phases, resulting in a total of four sub-phases throughout each player’s turn. During the movement phase, players engage in the reinforcement and movement sub-phases, wherein they strategically place reinforcements and manoeuvre their units, respectively. Meanwhile, the combat phase entails sub-phases for: target selection (where the player chooses the tile to attack) and attacker selection (in which adjacent units are chosen to enter combat). Each turn is characterized by both players engaging in both phases, with each player completing their turn before the other can proceed. The only exception to this rule happens at the beginning of the game, before the first turn, where both players place their initial reinforcements on the map, without going through any other phases. Problem setting The term “Wargames” is used to describe games that attempt to realistically replicate military engagements. There are thousands of Wargames available, but our work focuses on a sub group called Hex and Counter Wargames. Hex and Counter Wargames are adversarial boardgames played on hexagonally tiled maps that use different cardboard markers and counters to represent the several units and resources available on the map. These games are usually, but not limited to, simulations of historic military conflicts and commonly include large maps that represent multiple regions or countries, and rulebooks with many dozens or even hundreds of pages. As an example of relatively simple Hex and Counter wargames one has the Standart Combat Series (SCS) [ Essig(2015)