A Multi-Agent Pokemon Tournament for Evaluating Strategic Reasoning of Large Language Models
Tadisetty Sai Yashwanth, Dhatri C · 2025
Read A Multi-Agent Pokemon Tournament for Evaluating Strategic Reasoning of Large Language Models by Tadisetty Sai Yashwanth, Dhatri C (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01623.txt.
Introduction
Tadisetty Sai Yashwanth and Dhatri C stage a tournament of game-playing chatbots in a 2025 paper. Their Pokémon League pits large language models against each other as trainers in turn-based battles.
The single-elimination bracket records far more than wins. Logs capture why each model built its team as it did, which moves it chose, and when it swapped fighters. That trail lets the authors compare reasoning style, adaptability, and tactical depth across models. Type matchups and rule limits keep the contest fair and legible. The authors offer the league as a fresh benchmark for strategic reasoning under uncertainty, a skill that standard question tests barely touch.
Catalog record
| CREATOR | Tadisetty Sai Yashwanth, Dhatri C |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Multi-Agent Pokemon Tournament for Evaluating Strategic Reasoning of Large Language Models Tadisetty Sai Yashwanth, Dhatri C Abstract This research presents LLM Pokemon League, a competitive tournament system that leverages Large Language Models (LLMs) as intelligent agents to simulate strategic decision-making in Pokémon battles. The platform is designed to analyze and compare the reasoning, adaptability, and tactical depth exhibited by different LLMs in a type-based, turn-based combat environment. By structuring the competition as a single-elimination tournament involving diverse AI trainers, the system captures detailed decision logs, including team-building rationale, action selection strategies, and switching decisions. The project enables rich exploration into comparative AI behavior, battle psychology, and meta-strategy development in constrained, rule-based game environments. Through this system, we investigate how modern LLMs understand, adapt, and optimize decisions under uncertainty, making Pokémon League a novel benchmark for AI research in strategic reasoning and competitive learning. I Introduction Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning, planning, and problem-solving across diverse domains. However, the systematic evaluation of their strategic decision-making in multi-agent adversarial environments remains limited. Traditional benchmarks for AI strategic reasoning, such as chess or poker, often rely on specialized algorithms and offer limited insight into the underlying reasoning processes of foundation models. To address this gap, we present LLM Pokémon League , a novel tournament framework that evaluates LLMs as strategic agents in Pokémon battles. This domain is characterized by complex type interactions, probabilistic outcomes, and multi-layered decision-making. As shown in Figure 1 , Pokémon battles feature a turn-based interface where each agent must decide which move or switch to make in real time, given partial information and uncertain outcomes. Pokémon battles offer several advantages as a research testbed: • well-defined rules with 18 interconnected types that create rich strategic complexity, • asymmetric information and resource management constraints, • a blend of deterministic relationships and stochastic elements, and • a compact yet expressive decision space well-suited for natural language reasoning. Fig. 1: Traditional Pokémon battle interface, where agents make sequential decisions involving attacks, switches, or predictions. This research seeks to answer several key questions: • How do different LLMs approach the task of strategic team building when selecting from a common pool of options? • What tactical heuristics and reasoning patterns emerge during battles when agents operate in a zero-shot setting? • How do divergent strategic philosophies such as raw offensive power versus a balanced team perform in competitive tournament settings? To explore these questions, we instantiate LLMs from the GPT, Claude, and Gemini families as tournament participants. Each model is tasked with team composition, battle tactics, and dynamic switching decisions. Crucially, our framework captures not only each agent’s actions but also its natural language explanations for those actions, enabling a direct analysis of strategic reasoning. The strategic depth of Pokémon emerges from a confluence of interconnected systems. Type effectiveness relationships, visualized in Figure 2 , form a complex advantage matrix that rewards deep combinatorial reasoning. Effective team building requires a careful balance of offensive coverage and defensive synergy. Real-time battles further compound the challenge, as agents must continuously predict opponent behavior while managing their limited resources. Fig. 2: The Pokémon type effectiveness chart, illustrating damage multipliers among 18 different types. Mastery of this system is critical for strategic play. Unlike traditional game AIs that rely on search-based methods or specialized architectures, our agents operate purely through natural language reasoning. This makes their decision-making processes not only interpretable, but also directly accessible for comparative analysis. The tournament structure enables deeper investigation into meta-game dynamics, opponent modeling, and adaptive strategies. Each match generates detailed logs of decisions and their accompanying rationales, allowing for analysis of reasoning consistency, adaptability, and strategy evolution across model architectures. This approach offers unprecedented visibility into how foundation models formulate and refine their strategies in adversarial contexts. Our contributions are threefold: 1. A multi-agent tournament framework leveraging LLMs for strategic competition in a complex, adversarial domain. 2. A methodology for capturing and analyzing natural language reasoning behind game-theoretic decisions. 3. Empirical insights into the strategic diversity and tendencies of modern LLMs, including the observed performance gap between balanced teams and those optimized for overwhelming statistical advantage. By combining structured gameplay with language-based explainability, Pokémon League serves both as a challenging benchmark for AI strategic capabilities and as a research platform for understanding foundation model decision-making in competitive environments. II Related Works The strategic reasoning capabilities of Large Language Models (LLMs) in adversarial multi-agent environments have gained increasing attention in recent years. A number of frameworks have been proposed to evaluate and enhance these abilities, particularly through the lens of game-theoretic or behavioral modeling. Jia et al. [ 1 ] introduced a behavioral game theory framework to disentangle LLMs’ reasoning abilities from contextual effects. Their evaluation of 22 state-of-the-art models revealed that model scale alone does not guarantee strategic proficiency, and that decision-making may reflect demographic biases. This highlights the importance of benchmarks that go beyond outcome prediction to reveal underlying reasoning, a goal our Pokémon League framework shares. Lee and Kader [ 2 ] analyzed LLMs in classic behavioral economics games. While models exhibited some basic strategic understanding, they struggled with higher-order reasoning. This limitation motivates the need for environments like Pokémon League , where strategy involves layered decision-making, uncertainty, and opponent modeling. Zhang et al. [ 3 ] proposed a recursive “K-Level Reasoning” framework, where LLMs form beliefs about other agents’ beliefs. This work demonstrates the potential for LLMs to engage in deeper strategic reasoning and directly informs our interest in meta-strategy dynamics within tournament play. Anthropic’s multi-agent research [ 4 ] shows that distributing reasoning across multiple LLMs can outperform single-agent setups on complex tasks. Our framework draws on this insight by instantiating multiple distinct models as competitors in a shared environment, allowing for the study of emergent, interacting strategies. Guan et al. [ 5 ] explored deliberative alignment by integrating symbolic reasoning with reinforcement learning to improve LLM safety and policy-aware reasoning. While not focused on games, their work underlines the importance of combining structural alignment and adaptive strategy, which is echoed in our natural language-explained battle decisions. Yuan et al. [ 6 ] developed a framework to trace LLM reasoning through multi-stage strategic games, emphasizing the importance of analyzing not just actions, but the rationale behind them. Pokémon League builds on this by capturing natural language explanations for every decision, enabling fine-grained interpretability of strategic thinking. Lv and Qihang [ 7 ] introduced an LLM-powered Pokémon agent trained using supervised fine-tuning and knowledge augmentation. While their approach improved move selection accuracy, it did not investigate cross-agent interactions or emergent tournament dynamics. In contrast, Pokémon League offers a competitive setting to study such dynamics among diverse models. Hu et al. [ 8 ] proposed PokéLLMon , a benchmark for adversarial Pokémon battles. They found that LLMs often lack domain grounding and exhibit inconsistency in sequential decisions. Pokémon League addresses these issues by analyzing consistency across full matches and emphasizing interpretable, model-generated rationales. Wu et al. [ 9 ] introduced a cognitive-inspired framework integrating episodic and working memory into LLMs to enhance long-term coherence in strategic reasoning. This aligns with our agent design, where memory of previous turns is embedded