EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

Tristan Maidment 1, JB Lanier 1, Chase McDonald, Nathan Tsang 1, Eugene Vinitsky 3, Roy Fox 2, Albert Wang, Wesley N. Kerr 1 · 2026

Read EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games by Tristan Maidment 1, JB Lanier 1, Chase McDonald, Nathan Tsang 1, Eugene Vinitsky 3, Roy Fox 2, Albert Wang, Wesley N. Kerr 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23995.txt.

Introduction

Maidment and colleagues stabilize game playing AI that learns by playing itself, by anchoring it to a moving average of its own recent self, in a 2026 paper on reinforcement learning. In two player zero sum games with hidden information, regularized policy gradient methods trained in self play now rival specialized game solvers. The usual anchor is the uniform policy, but that wastes effort pulling the agent toward hopeless strategies. EMAgnet instead regularizes toward an exponential moving average of the agent's own past parameters.

The target trails the current strategy, pointing at competent recent play rather than uniform randomness. Across standard and modified benchmarks, EMAgnet reaches lower exploitability than PPO self play with uniform anchoring in the majority of environments. Gains are consistent where strictly dominated strategies abound, exactly where uniform regularization wastes the most.

Catalog record

CREATORTristan Maidment 1, JB Lanier 1, Chase McDonald, Nathan Tsang 1, Eugene Vinitsky 3, Roy Fox 2, Albert Wang, Wesley N. Kerr 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games Tristan Maidment 1, JB Lanier 1, Chase McDonald, Nathan Tsang 1, Eugene Vinitsky 3, Roy Fox 2, Albert Wang, Wesley N. Kerr 1 Abstract Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy’s parameters, providing an adaptive regularization target that evolves with the agent’s improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies. \optauthor and and and and , and and 1 Riot Games 2 University of California, Irvine 3 New York University \correspondingnote Corresponding authors: [email protected] , [email protected] 1 Introduction Solving two-player zero-sum imperfect-information games (IIGs) has driven notable advances in AI, from Poker ( Brown and Sandholm 2018 ; Brown et al. 2020 ) and Stratego ( Pérolat et al. 2022 ; Sokota et al. 2025 ) to real-time strategy games like StarCraft ( Vinyals et al. 2019 ) and Dota ( Berner et al. 2019 ) . Of these, multiple results have relied on self-play training stabilized through regularization towards target policies ( Pérolat et al. 2022 ; Sokota et al. 2025 ) , and recent work has demonstrated that with appropriate regularization, generic policy gradient methods can match or exceed other more specialized game-theoretic approaches ( Sokota et al. 2023 ; Rudolph et al. 2026 ) . Given this growing reliance on regularization, the choice of target policy becomes a key design decision. Sokota et al. 2023 and Rudolph et al. 2026 establish the uniform distribution, implemented as an entropy bonus, as a straightforward and effective regularization target. Rudolph et al. 2026 show PPO self-play outperforms more specialized game-solving methods such as R-NaD ( Pérolat et al. 2022 ) , PSRO ( Lanctot et al. 2017 ) , and NFSP ( Heinrich and Silver 2016 ) . However, the uniform target is strategically agnostic as it regularizes equally toward all strategies regardless of whether they are viable or strictly dominated. In games with large strategy spaces where most options are bad, the uniform target wastes regularization budget on irrelevant strategies (Figure 1 a,b). As games grow in complexity, the fraction of the strategy space that is strategically relevant tends to shrink, making this limitation increasingly consequential. In tabular settings, Sokota et al. 2023 explored a continuously moving “magnet” (the regularization target) that trails behind the current policy, demonstrating faster convergence than annealing uniform regularization strength. We introduce EMAgnet , which extends this concept to deep RL by maintaining a parameter-space exponential moving average (EMA) of the policy’s own network weights as the regularization target (Figure 1 c). The EMA magnet continuously adapts to the policy while adding minimal complexity to the standard PPO training loop. As the policy learns to avoid dominated strategies, the magnet also gradually stops regularizing toward them. At the same time, the EMA naturally accumulates a smoothed mixture over strategies encountered during self-play cycling, encouraging the policy to maintain coverage over strategically relevant options. This dual property, forgetting bad strategies while remembering good ones, is what distinguishes EMAgnet from a fixed regularization target. We extend the tabular moving magnet concept from Sokota et al. 2023 to deep RL with PPO ( Schulman et al. 2017 ) via parameter-space EMA regularization. We denote this new algorithm as PPO-EMAg , and we evaluate it against PPO self-play with uniform-magnet regularization under both linear annealing ( Rudolph et al. 2026 ) and power-law annealing ( Sokota et al. 2025 ) across standard two-player zero-sum benchmarks and modified benchmarks containing a large number of strictly dominated strategies ( Lanier et al. 2026 ) . PPO-EMAg matches or outperforms uniform baselines on standard benchmarks and outperforms them in games containing strictly dominated strategies. Figure 1: Self-play policy trajectories in Control Biased RPS ( Lanier et al. 2026 ) , where agents must solve gridworld navigation tasks to execute each RPS action or else forfeit. (a,b) Regularizing toward uniform forces the policy to use strictly dominated strategies that fail navigation and forfeit. By the time annealed regularization is weak enough to avoid forfeiting, the policy fails to explore and find the Nash equilibrium (green diamond). (c) PPO-EMAg applies constant regularization toward an EMA of the last-iterate (dashed orange), which regularizes toward viable actions the policy has chosen in the past, enabling convergence. 2 Preliminaries We consider two-player zero-sum games formalized as finite-horizon partially observable stochastic games (POSGs). A game is defined by the tuple \langle{\mathcal{S}},{\mathcal{A}},{\mathcal{O}},{\mathcal{R}},{\mathcal{T}},\Omega,T\rangle, where {\mathcal{S}} is the state space, {\mathcal{A}} is the action space, {\mathcal{O}} is the observation space, {\mathcal{R}}^{i}:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R} is the reward function for player i with {\mathcal{R}}^{i}=-{\mathcal{R}}^{-i} , {\mathcal{T}}:{\mathcal{S}}\times{\mathcal{A}}\to{\mathcal{S}} is the transition function, \Omega:{\mathcal{S}}\to{\mathcal{O}} is the observation function, and T is the episode horizon. From a sequence of observations and actions, each player constructs an information state z\in{\mathcal{Z}}=\cup_{t}({\mathcal{O}}\times{\mathcal{A}})^{t}\times{\mathcal{O}} that is sufficient for optimal decision-making. Each player i\in\{1,2\} acts according to a stochastic policy \pi_{i}:{\mathcal{Z}}\to\Delta({\mathcal{A}}) . We measure a joint policy \pi=(\pi_{1},\pi_{2}) by its exploitability , the mean incentive across players to deviate to a best response. A joint policy is a Nash equilibrium if and only if its exploitability is zero. 3 Method All methods in this work build on Proximal Policy Optimization ( Schulman et al. 2017 ) in symmetric self-play. We represent the policy with a neural network \pi_{\theta} and use a shared parameterization for both players, with player identity encoded in the observation. Our baseline, PPO-Uniform ( Rudolph et al. 2026 ) , augments the standard clipped PPO objective \mathcal{L}_{\text{PPO}}(\theta) with an entropy bonus \lambda_{H}H(\pi_{\theta}(\cdot\mid z)) that regularizes the policy toward the uniform distribution, where \lambda_{H}>0 may be held fixed or annealed over training. We propose replacing this fixed uniform target with an adaptive one. 3.1 PPO-EMAg The tabular moving magnet of Sokota et al. 2023 updates the magnet via a geometric average in policy space at each information state, \rho_{t+1}(h)\propto\rho_{t}(h)^{1-\tilde{\eta}}\pi_{t+1}(h)^{\tilde{\eta}} . With neural network policies, maintaining per-information-state policy averages is impractical. PPO-EMAg instead performs an arithmetic average in parameter space, replacing the uniform magnet with an exponential moving average (EMA) of the policy parameters. The objective becomes \mathcal{L}_{\text{PPO-EMAg}}(\theta)=\mathcal{L}_{\text{PPO}}(\theta)+\lambda_{\mathrm{KL}}\,\mathbb{E}_{z\sim\mathcal{T}}\!\bigl[D_{\mathrm{KL}}\bigl(\pi_{\theta_{\mathrm{mag}}}(\cdot\mid z)\,\|\,\pi_{\theta}(\cdot\mid z)\bigr)\bigr], (1) where \theta_{\mathrm{mag}} denotes the magnet parameters and \lambda_{\mathrm{KL}}>0 controls the regularization strength. After each PPO update, the magnet parameters are updated as \theta_{\mathrm{mag}}\leftarrow(1-\tau)\,\theta_{\mathrm{mag}}+\tau\,\theta, (2) with step size \tau\in(0,1] . The magnet is initialized to the same random weights as the policy, \theta_{\mathrm{mag}}\leftarrow\theta . The full training procedure is summarized in Algorithm B (Appendix). The key property of PPO-EMAg is that the regularization target adapts with the policy’s improving strategy. As the policy learns to avoid strictly dominated strategies, the EMA magnet also gradually stops regularizing toward those strategies.

Read the complete text