Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning

Matej Straka, Martin Schmid · 2025

Read Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning by Matej Straka, Martin Schmid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06825.txt.

Introduction

Matej Straka and Martin Schmid build a real-time strategy game testbed and an agent that nearly tops its human ladder, in a 2025 paper. Generals.io, a browser conquest game, draws thousands of players weekly.

Their environment plugs into standard reinforcement learning toolkits and runs thousands of frames per second on plain hardware. The reference agent learned from human games first, then self-play. After 36 hours on one top GPU, it ranked in the top 0.003 percent of one-versus-one players. Potential-based reward shaping and memory features sped the climb. Code, examples, and tutorials are public. The authors offer the setup as an accessible but hard multi-agent benchmark.

Catalog record

CREATORMatej Straka, Martin Schmid
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06825

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning Matej Straka, Martin Schmid Abstract We introduce a real-time strategy game environment based on Generals.io, a game with thousands of weekly active players. Our environment is fully compatible with Gymnasium and PettingZoo and is capable of running thousands of frames per second on commodity hardware. We also present a reference agent, trained with supervised pre-training and self-play, which reached the top 0.003% of the 1v1 human leaderboard after only 36 hours on a single H100 GPU. To accelerate learning, we incorporate potential-based reward shaping and memory features. Our contributions of a modular RTS benchmark and a competitive baseline agent provide an accessible yet challenging platform for advancing multi-agent reinforcement learning research. The documented code, together with examples and tutorials, is available at https://github.com/strakam/generals-bots . 1 Introduction Games have played a pivotal role in the development of modern reinforcement learning (RL), acting as crucial benchmarks to advance the field of artificial intelligence. RL has been applied across various game types, starting with early successes in single-agent control like Atari ( Mnih et al. 2013 ) , and leading to significant milestones in perfect-information adversarial board games such as Go and Chess through self-play, exemplified by AlphaZero ( Silver et al. 2017 ) . Alongside these, RL has made strides in imperfect-information games such as poker ( Moravčík et al. 2017 ; Brown & Sandholm 2018 ) and Stratego ( Perolat et al. 2022 ) , which require reasoning under uncertainty and strategic deception. A distinct category emerged with games like Diplomacy ( Meta Fundamental AI Research Diplomacy Team 2022 ) , where RL has been utilized in environments where strategic alliances and verbal communication are key game mechanics, leveraging large language models (LLMs) to facilitate this communication. Another major venue for RL research has been large-scale, real-time multi-agent video games under partial observability, as evidenced by powerful systems developed for StarCraft II ( Vinyals et al. 2019 ) and Dota II ( Berner et al. 2019 ) . These diverse game environments collectively pose significant challenges—from low-level control and long-horizon planning under incomplete information to intricate multi-agent coordination—driving crucial advances in reinforcement learning and game theory. We present a real-time strategy environment based on the browser game Generals.io 1 1 1 https://generals.io . Our environment aims to provide an interesting benchmark with challenges comparable to those in large strategic video games such as StarCraft II, while being lightweight enough to allow RL experimentation on modest hardware. Moreover, Generals.io maintains an active community of several thousand weekly participants across multiple game formats. Each format hosts a quarterly championship tournament for top-ranked players, fostering a robust and dynamic competitive ecosystem. Although bots are allowed to compete, even the most advanced have yet to attain a top placement in these tournaments, underscoring the challenge and potential of this platform as a benchmark for agent development. There are two main contributions of this paper: first, we introduce a real-time strategy environment that is vectorized, compatible with Gymnasium ( Towers et al. 2024 ) and PettingZoo ( Terry et al. 2021 ) , and capable of running thousands of frames per second on commodity hardware; second, we develop a PPO-based agent that achieves top-tier performance on the 1v1 human leaderboard after 36 hours of training on a single H100 GPU. The remainder of the paper is structured as follows. In Section 2 , we situate our work within existing MARL benchmarks and prior Generals.io work. Section 3 formalizes the game mechanics and the partial observability model. In Section 4 , we introduce our environment. Section 5 then describes our reference agent—covering behavior cloning, self-play, reward shaping, feature design, and network architecture. Section 6 evaluates our agent against humans and existing bots. Finally, Section 7 summarizes our contributions and outlines the directions for future research. 2 Related Work Multi-agent RL environments can be broadly divided into cooperative, fully competitive, and mixed settings. In cooperative scenarios such as the StarCraft Multi-Agent Challenge (SMAC), agents rely on local observations to collaboratively micromanage unit control ( Samvelyan et al. 2019 ) ; fully competitive environments like FightLadder ( Li et al. 2024 ) involve direct competition between agents; and mixed settings, exemplified by hide-and-seek ( Baker et al. 2020 ) , feature teams that cooperate internally while competing against opponents to achieve their objectives. Generals can be played in both fully competitive (1v1, free-for-all) and mixed (e.g., 2v2) modes. Bhatia et al. 2023 introduce a Redis-based framework for data collection and bot integration with Generals.io; however, their choice of tools is ill-suited for machine learning applications, due to their lack of scalability. They also present a rule–based agent, Flobot , which was once competitive but is now substantially outperformed by more recent approaches. Later, Xu et al. 2018 highlighted Generals as an efficient and cost-effective research platform that offers challenges comparable to Dota 2 and StarCraft II, while supporting rapid simulation and a large online player base. They propose a Hierarchical Agent with Self-Play ( HASP ) and report a 77% win rate against Flobot . Prior to this work, the community-developed agent Human.exe 2 2 2 https://github.com/EklipZgit/generals-bot , created by hobbyist EklipZ , was considered state-of-the-art, consistently ranking among the top players on the Generals.io leaderboards. It offers a formidable challenge even to expert human opponents and obtains high winrates against both Flobot and HASP . Remarkably, Human.exe was engineered without any machine learning techniques, relying solely on heuristics and classical algorithms based on deep domain expertise. Moreover, Human.exe is capable of playing 1v1, free-for-all, and 2v2 formats. 3 Game Description Generals is played on an H\times W grid. Each player seeks to be the last to stand by capturing the opponent’s general . This is achieved by expanding territory, managing resources, and maneuvering armies to breach enemy defenses. In this section, we provide a detailed overview of the game’s mechanics. Grid Generation. At the start of each match, an H\times W grid is populated with four cell types: plain (traversable), mountain (impassable), general (player base cell), and castle (army generating cell). The grid generator reproduces terrain statistics from the web-based Generals.io implementation, populating the grid with approximately 20% mountain cells and 9–11 castles (each randomly initialized with 40–50 neutral units). Player generals are placed at least 15 breadth-first-search steps apart, and each player is guaranteed at least one castle within a BFS radius of 6. An example of a grid layout can be seen in Figure 1 . Figure 1 : Three zoom levels of the same game state are shown. Cells of different colors belong to different players. Each player has one base, depicted as a crown. Numbers in the cells represent the number of units in that cell. A player can move units to any neighboring cell, except in the direction of mountains. Ownership and Partial Observability. Cells may be owned by a player or remain neutral. A fog-of-war mechanic restricts the view of each player to (i) all cells they own and (ii) the immediate eight neighbor ("Moore") neighborhood around those cells; all other cells are hidden (see Figure 2 ). In parallel, a global scoreboard displays each player’s total owned cells and the aggregate army count,

Read the complete text