SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks
Pengbo Shen, Yaqing Wang, Ni Mu, Yao Luan, Runpeng Xie, Senhao Yang, Lexiang Wang, Hao Hu, Shuang Xu, Yiqin Yang, Bo Xu · 2025
Read SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks by Pengbo Shen, Yaqing Wang, Ni Mu, Yao Luan, Runpeng Xie, Senhao Yang, Lexiang Wang, Hao Hu, Shuang Xu, Yiqin Yang, Bo Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10428.txt.
Introduction
Pengbo Shen and co-authors present a 2025 benchmark and training framework for language models playing StarCraft II, a complex strategy game. Older benchmarks trim the game down, dropping races, actions, or context, so scores overstate real skill. Their benchmark, SC2Arena, keeps all playable races and the full low-level action space. It also reshapes the text view of the game to ease spatial reasoning.
The companion framework, StarEvolve, splits play among a planner, an executor, and a verifier that checks and corrects moves. It improves by fine-tuning on high-scoring games it played itself. In tests on the new benchmark, StarEvolve outplayed rivals on strategic planning. The authors release their code and tools, hoping to push general agents that plan and adapt in real time.
Catalog record
| CREATOR | Pengbo Shen, Yaqing Wang, Ni Mu, Yao Luan, Runpeng Xie, Senhao Yang, Lexiang Wang, Hao Hu, Shuang Xu, Yiqin Yang, Bo Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10428
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SC2Arena and StarEvolve: Benchmark and Self-Improvement Framework for LLMs in Complex Decision-Making Tasks Pengbo Shen, Yaqing Wang, Ni Mu, Yao Luan, Runpeng Xie, Senhao Yang, Lexiang Wang, Hao Hu, Shuang Xu, Yiqin Yang, Bo Xu Abstract Evaluating large language models (LLMs) in complex decision-making is essential for advancing AI’s ability for strategic planning and real-time adaptation. However, existing benchmarks for tasks like StarCraft II fail to capture the game’s full complexity, such as its complete game context, diverse action spaces, and all playable races. To address this gap, we present SC2Arena, a benchmark that fully supports all playable races, low-level action spaces, and optimizes text-based observations to tackle spatial reasoning challenges. Complementing this, we introduce StarEvolve, a hierarchical framework that integrates strategic planning with tactical execution, featuring iterative self-correction and continuous improvement via fine-tuning on high-quality gameplay data. Its key components include a Planner-Executor-Verifier structure to break down gameplay, and a scoring system for selecting high-quality training samples. Comprehensive analysis using SC2Arena provides valuable insights into developing generalist agents that were not possible with previous benchmarks. Experimental results also demonstrate that our proposed StarEvolve achieves superior performance in strategic planning. Our code, environment, and algorithms are publicly available. 1 Introduction The rapid advancement of large language models (LLMs) ( Achiam et al. 2023 ; Guo et al. 2025 ) has accelerated research towards artificial general intelligence ( Mon-Williams et al. 2025 ; Zheng et al. 2025 ) . A critical milestone in this pursuit is the development of agents capable of strategic planning and real-time adaptation in complex environments. StarCraft II, a renowned real-time strategy game, provides an ideal testbed with its profound strategic depth, diverse unit types and interactions, vast decision space, and inherent demands for world modeling, as illustrated in Fig. 1 . These characteristics align closely with the LLMs’ core strengths ( Wei et al. 2022 ; Yao et al. 2023 ; Wang et al. 2023b ) , making it particularly suitable for LLM evaluation. Therefore, establishing benchmarks in StarCraft II to assess LLM performance is critical. However, existing LLM benchmarks in StarCraft II ( Ma et al. 2024 ; Li et al. 2024 ) often fail to capture the original game’s complexity, lacking support for full-length game context, complete low-level action spaces, comprehensive race configuration, and agent-vs-agent mechanisms. This limits their ability to evaluate the strategic reasoning and adaptability of LLM agents. Figure 1: An illustration of StarCraft II. StarCraft II is a highly complex and competitive real-time strategy game, where players choose one of three distinct races (Terran, Zerg, Protoss) to gather resources, construct buildings, produce units, and defeat enemies. Each race has unique traits and can produce race-specific buildings and military units. To bridge this gap, we propose SC2Arena , a comprehensive StarCraft II benchmark for evaluating LLMs’ decision-making ability. With a standardized JSON interface, it supports ① full-length game context, ② all playable races (Terran, Protoss, Zerg), and ③ complete low-level actions, preserving the game’s strategic depth. Crucially, SC2Arena addresses inherent challenges of text-based agents by optimizing observations: proximity-based unit ordering captures units’ spatial relationships, and worker aggregation reduces information overload. Additionally, it enables agent-vs-agent gameplay, facilitating direct competition and an agent ranking mechanism to better evaluate performance and strategic adaptability. Furthermore, we introduce StarEvolve , a closed-loop agent framework that enables self-correction and self-improvement for strategic decision-making. It employs a hierarchical architecture: a Planner generates high-level strategic commands, while an Executor converts them into precise low-level actions. Both components integrate Verifier modules that perform iterative self-correction to ensure syntactic correction and feasibility. Crucially, StarEvolve achieves continuous self-improvement by collecting gameplay data and performing supervised fine-tuning (SFT). We design an RL-inspired scoring function to prioritize high-quality training samples, which quantifies action impact through discounted value increments across key game-state metrics. Utilizing these data, we fine-tune the Planner, Executor, and Verifiers to continuously improve their performance. In summary, our contributions are twofold: (1) We propose SC2Arena, a comprehensive benchmark for evaluating LLMs in StarCraft II, supporting full-length game context, all races, full action spaces, and agent-vs-agent gameplay. (2) We propose StarEvolve, a closed-loop agent framework, which integrates strategic planning with tactical execution. It achieves self-correction with iterative decision refinement, and also continuous self-improvement via SFT, enabling LLM agents to defeat challenging opponents. 2 Related Work LLM as autonomous agents. Recent advances in LLMs have empowered autonomous agents to succeed in diverse domains, including conversational games ( Xu et al. 2023 ; Wang et al. 2023c ) , webpage navigation ( Lai et al. 2024 ) , and open-world environments ( Wang et al. 2023a ) . Concurrently, benchmarks ( Liu et al. 2023 ; Zhou et al. 2023 ; Light et al. 2023 ; Rismanchian et al. 2024 ) have emerged to evaluate these agents’ planning and reasoning capabilities. Despite these successes, LLM agents remain relatively underexplored in real-time adversarial games, which require integration of long-term strategic planning and precise tactical execution, under partial observability and time constraints. AI in StarCraft II. StarCraft II remains a classical AI research environment ( Ontanón et al. 2013 ) due to its expansive action space and real-time decision complexity. Prior work primarily leveraged reinforcement learning (RL), achieving strong micro-management performance through multi-agent RL ( Samvelyan et al. 2019 ; Rashid et al. 2020 ; Yu et al. 2022 ) . Subsequent efforts like ( Vinyals et al. 2019 ) scaled RL to full-length game contexts, although this approach required massive training resources and carefully engineered rewards. In contrast, LLMs offer complementary strengths to strategic planning by leveraging inherent world knowledge and reasoning capabilities ( Yildirim and Paul 2024 ; Yuan et al. 2024 ) . Yet, current benchmarks for evaluating LLM decision-making in StarCraft II ( Ma et al. 2024 ; Li et al. 2024 ) have limitations, including the abstraction of low-level tactical execution, a lack of support for full-length game contexts and full race configurations, and insufficient agent-vs-agent mechanisms. To bridge these gaps, we propose SC2Arena, a comprehensive StarCraft II benchmark to evaluate agents across diverse dimensions. We further propose StarEvolve, a self-improving agent framework for this benchmark, advancing LLM-driven decision-making in complex environments. Figure 2: Illustration of SC2Arena design. (a) SC2Arena provides a standardized and extensible text-based interface where actions are defined in a unified JSON format. (b) To convert game states into textual observations suitable for LLMs, SC2Arena introduces two key techniques: Unit aggregation and Proximity-based unit ordering. Table 1: Comparison of benchmarks in StarCraft II from four dimensions. Benchmark Agent-vs-agent Low-level action space All races Full-length game context SMAC ( Samvelyan et al. 2019 ) ✗ ✓ ✗ ✗ LLM-PySC2 ( Li et al. 2024 ) ✗ ✓ ✗ ✓ TextStarCraft II ( Ma et al. 2024 ) ✗ ✗ ✗ ✓ SC2Arena (ours) ✓ ✓ ✓ ✓ 3 Benchmark: SC2Arena In this section, we propose SC2Arena, a structured text-based benchmark designed to evaluate LLM agents in StarCraft II, with standardized observation and action interfaces. SC2Arena supports full-length game contexts, complete low-level action spaces, all three races and agent-vs-agent gameplay, thus preserving the original game’s strategic depth and complexity. 3.1 Introduction to SC2Arena Overview of StarCraft II. StarCraft II is a highly complex and competitive real-time strategy game, where players control one