Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
Yang Li, Xing Chen, Yutao Liu, Gege Qi, Yanxian BI, Zizhe Wang, Yunjian Zhang, Yao Zhu · 2026
Read Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments by Yang Li, Xing Chen, Yutao Liu, Gege Qi, Yanxian BI, Zizhe Wang, Yunjian Zhang, Yao Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09337.txt.
Introduction
Yang Li and colleagues introduce the STAR Benchmark, which tests language models as agents in one-on-one zero-sum contests rather than on static puzzles. It frames reasoning as repeated, adaptive choice under time pressure, in both turn-based and real-time modes. A modular design with a standard API and full execution engine keeps runs reproducible and tasks customizable. A scoring suite grades not just wins but how well the agent plays, including how fast and cleanly it acts.
Catalog record
| CREATOR | Yang Li, Xing Chen, Yutao Liu, Gege Qi, Yanxian BI, Zizhe Wang, Yunjian Zhang, Yao Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52227 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09337
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments Yang Li, Xing Chen, Yutao Liu, Gege Qi, Yanxian BI, Zizhe Wang, Yunjian Zhang, Yao Zhu Abstract Large Language Models (LLMs) have achieved strong performance on static reasoning benchmarks, yet their effectiveness as interactive agents operating in adversarial, time-sensitive environments remains poorly understood. Existing evaluations largely treat reasoning as a single-shot capability, overlooking the challenges of opponent-aware decision-making, temporal constraints, and execution under pressure. This paper introduces Strategic Tactical Agent Reasoning (STAR) Benchmark, a multi-agent evaluation framework that assesses LLMs through 1v1 zero-sum competitive interactions, framing reasoning as an iterative, adaptive decision-making process. STAR supports both turn-based and real-time settings, enabling controlled analysis of long-horizon strategic planning and fast-paced tactical execution within a unified environment. Built on a modular architecture with a standardized API and fully implemented execution engine, STAR facilitates reproducible evaluation and flexible task customization. To move beyond binary win–loss outcomes, we introduce a Strategic Evaluation Suite that assesses not only competitive success but also the quality of strategic behavior, such as execution efficiency and outcome stability. Extensive pairwise evaluations reveal a pronounced strategy–execution gap: while reasoning-intensive models dominate turn-based settings, their inference latency often leads to inferior performance in real-time scenarios, where faster instruction-tuned models prevail. These results show that strategic intelligence in interactive environments depends not only on reasoning depth, but also on the ability to translate plans into timely actions, positioning STAR as a principled benchmark for studying this trade-off in competitive, dynamic settings. Codes are available at https://github.com/star-nexus/star . Figure 1 : Visualization interface of STAR (Strategic Tactical Agent Reasoning Benchmark). STAR is a LLM PvP(Player versus Player) environment with multiple types of maps, evaluating the capability of LLMs to perform iterative reasoning and strategic decision-making in dynamic and zero-sum multi-agent environments. 1 Introduction Large Language Models (LLMs) have achieved remarkable progress in recent years ( 6 ; 25 ; 27 ) , driven by large-scale pretraining and advances in reasoning-oriented techniques such as Chain-of-Thought prompting ( 23 ) and architectural optimization ( 13 ) . As a result, modern LLMs demonstrate strong performance across a wide range of static benchmarks, including mathematical reasoning ( 5 ) , code generation ( 15 ) , and factual knowledge evaluation ( 22 ) . These successes have established reasoning as one of the central capabilities in contemporary language models. However, most existing evaluations implicitly treat reasoning as a single-shot, context-independent process: the model observes a fixed input, produces an answer, and is scored in isolation. Such settings fail to reflect scenarios where reasoning must unfold over time, react to an opponent, and adapt to continuously evolving states. In particular, they overlook a critical dimension of intelligence: strategic reasoning under adversarial interaction, where outcomes depend not only on the correctness of individual decisions, but also on anticipation, adaptation, and long-horizon planning. As LLMs are increasingly used as decision-making agents, this gap between static evaluation and interactive reasoning becomes increasingly pronounced. A closer examination reveals several limitations shared by existing benchmarks. First, most evaluations focus on single-agent performance in non-adversarial environments, providing little insight into a model’s ability to reason about and respond to competing agents. Second, temporal structure is largely ignored: models are typically allowed unconstrained inference time, masking the trade-off between reasoning depth and decision timeliness that arises in sequential decision-making. Third, evaluation metrics are often coarse-grained (e.g., accuracy or success rate), making it difficult to disentangle how a model wins or loses, and whether its reasoning leads to robust strategic outcomes. To address these limitations, we introduce the Strategic Tactical Agent Reasoning Benchmark (STAR), a multi-agent evaluation framework designed to assess LLMs through 1v1 competitive interactions in a zero-sum environment. STAR formulates strategic reasoning as an iterative decision-making process, where two agents repeatedly interact, observe each other’s actions, and adapt their strategies over time. This design enables direct measurement of a model’s ability to perform opponent-aware reasoning, maintain situational consistency, and plan over extended horizons, where capabilities that are difficult to capture in static benchmarks. Architecturally, STAR is organized into four decoupled layers, including framework, environment, communication, and agent, which together provide a clean separation between game logic, interaction protocols, and model-specific behaviors. Importantly, STAR includes a fully implemented underlying execution environment, allowing users to define new tasks and rules on top of the existing infrastructure without reimplementing core mechanics. Through a unified interaction interface and open API, LLMs can be seamlessly deployed as autonomous agents across different task instantiations. Beyond empirical win rates, STAR incorporates a strategic evaluation suite named Performance-Weighted ELO Rating (PWER) tailored for multi-turn adversarial settings, capturing not only competitive outcomes but also the efficiency and stability of strategic execution. By grounding evaluation in interactive competition and supporting user-defined tasks, STAR functions not only as a benchmark but as a general-purpose platform for studying strategic reasoning and agent behavior in dynamic, adversarial environments. We conduct extensive pairwise evaluations of a diverse set of representative LLMs on the STAR benchmark through systematically designed 1v1 competitive experiments. The results reveal a clear correlation between a model’s reasoning quality and its competitive performance, while simultaneously exposing critical failure modes that are invisible in static evaluations. Notably, models that excel in isolated reasoning tasks do not necessarily translate this capability into superior outcomes in adversarial interactions, underscoring that strategic reasoning must be understood as a dynamic, agent-centric process rather than a static competence. Overall, these findings demonstrate that STAR can effectively differentiate models based on their ability to convert reasoning sophistication into actionable strategic advantage, providing a complementary and necessary perspective for evaluating the strategic reasoning capabilities of modern LLMs. Figure 1 illustrates a representative encounter between two LLM agents, highlighting their decision-making behaviors in a dynamic competitive setting. Our contributions are summarized as follows: • We introduce STAR, the first benchmark that evaluates Large Language Models in explicit zero-sum, 1v1 competitive environments, where reasoning quality is assessed through direct adversarial interaction rather than static, single-agent prompts. By framing reasoning as an iterative, opponent-aware decision process, STAR enables systematic evaluation of strategic consistency, adaptation, and long-horizon planning capabilities that are fundamentally inaccessible to existing static benchmarks. • We design a layered and modular agentic framework for strategic evaluation, consisting of decoupled framework, environment, communication, and agent layers. STAR provides a fully implemented execution environment and a unified interaction API, allowing users to easily define new tasks and deploy heterogeneous LLM-based agents, thereby ensuring scalability, extensibility, and fair comparison across models. • We propose a multi-dimensional strategic evaluation suite tailored to adversarial settings, which goes beyond empirical win rates to quantify execution efficiency and strategic stability during interaction. This evaluation paradigm exposes critical gaps between abstract reasoning ability and effective action under time and adversarial pressure, offering fine-grained diagnostic insights into how and why LLMs succeed or fail as interactive agents. Figure 2 : Overview of the STAR benchmark architecture. The system relies on four decoupled layers designed for high extensibility and interoperability. The Framework Layer serves as the foundational ECS engine, empowering researchers to construct diverse game scenarios in the Environment Layer by reusing core simulation components. Crucially, the Protocol Layer establishes a standardized interface that bridges the Agent Layer with