The Cell Must Go On: Agar.io for Continual Reinforcement Learning
Mohamed A. Mohamed, Kateryna Nekhomiazh, Vedant Vyas, Marcos M. José, Andrew Patterson, Marlos C. Machado · 2025
Read The Cell Must Go On: Agar.io for Continual Reinforcement Learning by Mohamed A. Mohamed, Kateryna Nekhomiazh, Vedant Vyas, Marcos M. José, Andrew Patterson, Marlos C. Machado (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18347.txt.
Introduction
Mohamed Mohamed and colleagues turn a browser game about growing cells into a testbed for agents that must keep learning forever. Continual reinforcement learning studies agents whose world keeps changing, so no fixed policy stays good. Existing setups mostly fake the change by swapping tasks between episodes, or offer worlds too small to be interesting. Writing in twenty twenty-five, the team adapts the game Agar.io, where a cell eats, grows, and risks being eaten in an endless, partly hidden, ever-shifting arena. Their platform supports both one long open-ended challenge and smaller tasks that isolate single skills.
Standard algorithms were benchmarked on the main challenge and the side tasks, and three continual learning methods were evaluated as well. The suite separates which ingredients of the game, such as partial sight or sudden threats, break which learners. The platform is offered as shared infrastructure for the field.
Catalog record
| CREATOR | Mohamed A. Mohamed, Kateryna Nekhomiazh, Vedant Vyas, Marcos M. José, Andrew Patterson, Marlos C. Machado |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18347
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Cell Must Go On: Agar.io for Continual Reinforcement Learning Mohamed A. Mohamed, Kateryna Nekhomiazh, Vedant Vyas, Marcos M. José, Andrew Patterson, Marlos C. Machado Abstract Continual reinforcement learning (RL) concerns agents that are expected to learn continually, rather than converge to a policy that is then fixed for evaluation. Such an approach is well suited to environments the agent perceives as changing , which renders any static policy ineffective over time. The few simulators explicitly designed for empirical research in continual RL are often limited in scope or complexity, and it is now common for researchers to modify episodic RL environments by artificially incorporating abrupt task changes during interaction. In this paper, we introduce AgarCL , a research platform for continual RL that allows for a progression of increasingly sophisticated behaviour. AgarCL is based on the game Agar.io, a non-episodic, high-dimensional problem featuring stochastic, ever-evolving dynamics, continuous actions, and partial observability. Additionally, we provide benchmark results reporting the performance of DQN, PPO, and SAC in both the primary, challenging continual RL problem, and across a suite of smaller tasks within AgarCL , each of which isolates aspects of the full environment and allow us to characterize the challenges posed by different aspects of the game. 1 Introduction Continual reinforcement learning (RL) is the RL setting where the agent faces problems in which it is expected to learn continually, rather than the more traditional setting where it learns a policy that is then fixed for evaluation or deployment. Continual RL can be seen either as a problem formulation khetarpal2022continualreinforcementlearningreview ; abel2023definition ; kumar2023continual or as a solution method for problems perceived by the agent as non-stationary. Such problems are often motivated by the big world hypothesis, which states that the “world” is bigger than the agent javed2024big ; in this case, continual adaptation is simply a more effective approach than any fixed policy sutton2007role ; janjua2024gvfs . Much of the progress in RL has been driven by empirical advances, with experimental results often shaping algorithmic innovations and research directions. As a result, evaluation platforms play a central role in accelerating progress in the field ( todorov2012mujoco ; bellemare2013arcade ; beatie2016deepmindlab , e.g.,) . In continual RL, most evaluation platforms are adaptations of environments from traditional RL research, modified to reflect the idea that the world is bigger than the agent. This is typically achieved by artificially introducing some non-stationarity to the environment—such as periodically switching the problem faced by the agent powers2022cora ; abbas2023loss ; anand2023prediction ; tomilin2023coom —or, more rarely, by designing new environments tailored specifically to continual RL research, with no vestigial notion of episodes ( platanios2020jelly ) . However, both of these approaches have limitations. While reusing existing environments—especially complex simulators—is appealing, introducing non-stationarity through hand-crafted switches that are disconnected from the agent’s behavior or the environment’s structure can feel overly artificial. It is akin to expecting someone to perform well when, without warning, they go from playing soccer to scuba diving. We rarely experience such abrupt phase shifts in our everyday lives. Figure 1: Agent-environment interface and main entities in AgarCL . The agent has access to one of two observation types: pixel-based or symbolic. The pixel-based observation includes four channels that represent different game entities: pellets, bots, viruses, and the agent itself. Alternatively, the agent can receive a symbolic observation, consisting of pre-processed features such as the distances to nearby enemies, pellets, and other entities. The reward function is defined as the change in the agent’s mass between two consecutive time steps. The action space in AgarCL is hybrid: at each time step, the agent selects an \langle x,y\rangle coordinate mimicking where a human player would point their mouse. Additionally, the agent decides whether to split, eject mass, or just move. In Section 3 , we provide detailed descriptions of these entities and the game dynamics, including the available actions. The second—designing environments specifically for continual RL—is promising in spirit, but existing instantiations tend to be limited in complexity or scope. Our work takes a complementary direction, aiming to preserve these purpose-built environments’ continual, non-episodic nature while introducing richer, ever-evolving stochastic dynamics, continuous actions, partial observability, resource-driven competition, and a scaffolded progression of behavioral complexity in a spatially structured world. Specifically, in this paper, we introduce a new evaluation platform based on the game Agar.io (see Figure 1 ). This platform, called AgarCL , was developed with a heavy emphasis on supporting research on continual RL. In this environment, the agent controls circular cells within a bounded, Petri dish-like arena. The agent perceives the world through high-dimensional, pixel-based observations, 1 1 1 Alternative observation types, with a more symbolic flavour, providing backwards compatibility to other platforms zhang2023gobigger , are also supported. We further discuss this option in Section 3 . and acts through both continuous and discrete actions. Navigation is determined by continuous actions represented as \langle x,y\rangle coordinates, while additional affordances such as splitting or ejecting mass are defined as discrete actions. In AgarCL , the reward is the difference in the agent’s mass between two consecutive time steps. Thus, an effective agent has a forever-growing mass. The dynamics of the game Agar.io are what make this environment interesting from a continual learning perspective. The agent ‘‘lives’’ in an environment full of other agents. 2 2 2 In AgarCL , the behaviour of other agents is determined by hand-coded policies. Future work will focus on supporting learning across multiple agents, making this environment multi-agent. Agents can increase their mass by either collecting stationary food pellets or absorbing other agents that are smaller than they are. Consequently, an agent’s primary objective is not only to increase their mass, but also to avoid larger cells that pose a threat. At each time step, the mass of a cell naturally decays, with the rate of decay increasing with size—because agents have the same density in the game, more mass implies greater size. Aside from its potential never-ending nature, and the complexity induced by other agents’ behaviours, the key property of AgarCL that makes it interesting from a continual RL perspective is how its dynamics change according to the agent’s mass. Bigger agents are slower, and their observation stream is zoomed out to keep their full body visible on screen. The consequence of an agent’s action (and even its observation stream) is constantly changing in a relatively smooth manner. Abrupt changes are also present—as when the agent absorbs another agent, drastically increasing its mass; when it splits itself into multiple cells on purpose; or when it splits due to a virus. Figure 2 and a Youtube video 3 3 3 https://www.youtube.com/watch?v=CGpvzHIqFLA depict some of the environment dynamics supported by AgarCL . We demonstrate the feasibility of AgarCL as both an evaluation platform and a challenge problem for continual RL by benchmarking three widely used algorithms—DQN ( DBLP:journals/nature/MnihKSRVBGRFOPB15 ) , PPO ( PPO ) , and SAC ( SAC ) —across multiple variations of the environment. In the core game, the Grand Arena, we find that none of these algorithms are able to learn an effective policy. We also evaluate their performance on several mini-games within AgarCL that isolate different aspects of the environment’s complexity—such as non-stationarity, exploration difficulty,