Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game
Tõnis Lees, Tambet Matiisen · 2026
Read Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game by Tõnis Lees, Tambet Matiisen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05476.txt.
Introduction
Lees and Matiisen adapt AlphaZero self-play to Tablut, an uneven board game where attackers hunt a king and defenders help him escape. Since one shared network must learn two clashing valuations, they split policy and value heads by role over a shared trunk. Fixes for role forgetting include board-symmetry augmentation, a larger replay buffer, and games against past selves. After 100 self-play rounds the model reaches a BayesElo rating of 1235 over a random baseline.
Catalog record
| CREATOR | Tõnis Lees, Tambet Matiisen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11223 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game Tõnis Lees, Tambet Matiisen Abstract 1 Introduction 5 introduced AlphaZero , a general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play with no domain-specific knowledge beyond the rules. A single neural network (\mathbf{p},v)=f_{\theta}(s) is trained iteratively: each iteration generates games via MCTS-guided self-play, then updates \theta to better predict game outcomes and the search-refined policy. In the original formulation, both players share a single policy and value head, and positions are canonicalized so the current player’s pieces always appear as “friendly”, which is natural for symmetric games where both sides face structurally identical decisions. In asymmetric games, where players differ in piece counts, objectives, and win conditions, this single head must learn two distinct evaluation functions, which can hinder learning efficiency and performance. This work investigates whether AlphaZero ’s self-play framework transfers to such a setting by applying it to Tablut, a historical board game played on a 9\times 9 board with 16 attackers against 8 defenders and a king. The attacker aims to capture the king; the defender aims to escort it to a corner (see Appendix A.1 ). The core algorithm transfers with one key modification—separate policy and value heads per player—but the asymmetric structure introduces training instabilities, such as catastrophic forgetting between roles, that were not reported in the original work. 2 Reproduction The neural network architecture closely follows 5 , with the main difference being a reduced residual trunk of 8 blocks with 128 filters to match Tablut’s lower complexity and more limited compute budget. The key modification is the use of separate policy and value heads for each player, (\mathbf{p_{a}},v_{a},\mathbf{p_{d}},v_{d})=f_{\theta}(s) , motivated by the fundamentally different objectives — king capture versus king escape — and the resulting asymmetric evaluation landscape. During MCTS, the head corresponding to the current player is selected, while the shared residual trunk learns common board features such as piece mobility and capture threats. Tablut’s rook-like piece movement allows direct reuse of the action encoding from 5 : each of the 81 squares has 32 directional planes (8 distances \times 4 directions), yielding an action space of 2592 possible moves per position. See Appendix A.3 for state representation planes. The system was implemented in JAX using Flax, Optax, Mctx, and Flashbax ( 1 ; 3 ; 6 ) . To enable hardware-accelerated self-play, game environments were fully vectorized. While 4 developed the Pgx library for this purpose, it does not natively support Tablut. Consequently, Tablut game logic was implemented from scratch by extending the Pgx base framework. Training was conducted on 2 NVIDIA H200 GPUs for 100 self-play iterations. In each iteration, generated states were stored in a replay buffer along with the MCTS-refined policy and the outcome reward of the corresponding game. The network was then trained on batches sampled from this buffer to predict game outcomes and policy distributions. MCTS was performed using the GumbelMuZero variant ( 2 ) with 128 simulations per move, which provides a more sample-efficient search than regular PUCT under limited compute budget, while retaining the AlphaZero -style policy and value targets. The loss function combines mean squared error for value and cross-entropy for policy, as in AlphaZero . AdamW with weight decay regularization was used for optimization. See Appendix 1 for all hyper-parameters. During training, performance against earlier checkpoints degraded — a phenomenon known as catastrophic forgetting in self-play. To stabilize training, C 4 data augmentation was applied (random 90^{\circ} board rotations), and the replay buffer was increased from 8 to 16 self-play iterations. Inspired by 7 , 25% of training games were played against randomly sampled past checkpoints, with the current model alternating between attacker and defender roles. See Appendix A.2 for an ablation of these contributions. Starting from the 20th iteration, the model was evaluated every 5 iterations against 4 opponents randomly sampled from a pool of up to 10 past checkpoints. The randomly initialized model (iteration 0) was retained as a fixed anchor. Following 5 , the BayesElo program was used to calculate the Elo ratings across iterations. BayesElo models draw rates and first-mover advantage, which is important for an asymmetric game where the inherent balance between sides is unknown. 3 Results and discussion Over 100 iterations ( 26 million frames), the model reached a BayesElo rating of 1235 relative to the randomly initialized baseline (see Figure 1(a) ). This indicates steady improvement, though the rating is only meaningful relative to this internal pool and not comparable to AlphaZero ’s absolute performance on Chess or Go. Policy entropy decreased from 3.05 to 1.47 and the average number of pieces remaining at game end dropped from 22 to 15 (see Appendix A.4 ), reflecting increasingly focused and decisive play. Up to iteration 75, attackers and defenders achieved similar evaluation win rates of roughly 70–80% against the pool of past checkpoints when playing as that side. After that point, the defender’s win rate declined to 52%, while the attacker’s climbed to 86%, and in self-play the attacker’s 10-iteration rolling average win rate reached about 65% by iteration 100 (see Figure 1(b) ). This suggests either that the Tablut ruleset used favours the attacker or that defender strategies are harder to learn in this setting; a separate balance analysis was not performed to distinguish these explanations. (a) Elo rating progression. (b) Game outcomes (win rates). Figure 1 : Training metrics over 100 iterations. The model shows steady Elo growth and an increasing win rate for the attacker in Tablut. The most challenging aspects of the reproduction were (i) down-scaling training from the much larger compute used by 5 and (ii) mitigating catastrophic forgetting in self-play, which the original paper does not discuss in detail. Among the hyperparameters, the MCTS simulation count proved most critical: reducing it below 128 degraded both search quality and the resulting policy targets, while larger counts were impractical on two GPUs. Increasing the replay buffer size and using C 4 data augmentation both substantially improved stability. Appendix A Appendix A.1 Tablut rules Tablut is a historic board game that was played in Northern Europe and its rules were first documented by Carl Linnaeus in 1732 in his diary. The original text, written in a mixture of Latin and Swedish, was subject to several problematic translations, leading to many different rule interpretations today. The rules used in this replication are the following: • The game is played on a 9x9 board with 8 defenders, 1 king and 16 attackers. Regular pieces are also called taflmen. • The king starts on the center square which is called the throne (see Figure 2 ). • The pieces move like rooks in chess, and they cannot pass through other pieces. • The objective of the defenders is to move the king to any of the corners of the board. • The objective of the attackers is to capture the king. • Any piece can be captured by sandwiching it between two enemy pieces. If a piece moves between two enemy pieces itself, then it is not captured. • The king is captured like any other piece and it can also participate in captures. • Only the king can step on the