CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs

Hongtao Liu, Zhicheng Du, Zihe Wang, Weiran Shen {ht6,duzhicheng,wang.zihe,shenweiran}@ruc.edu.cn Gaoling School of Artificial IntelligenceRenmin University of China · 2025

Read CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs by Hongtao Liu, Zhicheng Du, Zihe Wang, Weiran Shen {ht6,duzhicheng,wang.zihe,shenweiran}@ruc.edu.cn Gaoling School of Artificial IntelligenceRenmin University of China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11944.txt.

Introduction

Winning a game against an opponent is a shaky way to grade a language model's strategy, since the score depends on who sits across the table. Hongtao Liu, Zhicheng Du, Zihe Wang, and Weiran Shen borrow a yardstick from behavioral economics: the cognitive hierarchy, where thinkers are ranked by how many steps ahead they reason.

Their benchmark, CHBench, probes six leading models across fifteen simple games and infers each model's reasoning depth. The depths came out consistent no matter the opponent, which supports the yardstick's sturdiness. Two add-ons pulled in opposite directions: letting models chat mid-game dulled their strategy, while giving them memory sharpened it. The authors offer the benchmark as a steadier way to compare strategic minds.

Catalog record

CREATORHongtao Liu, Zhicheng Du, Zihe Wang, Weiran Shen {ht6,duzhicheng,wang.zihe,shenweiran}@ruc.edu.cn Gaoling School of Artificial IntelligenceRenmin University of China
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs Hongtao Liu, Zhicheng Du, Zihe Wang, Weiran Shen {ht6,duzhicheng,wang.zihe,shenweiran}@ruc.edu.cn Gaoling School of Artificial IntelligenceRenmin University of China Abstract Game-playing ability serves as an indicator for evaluating the strategic reasoning capability of large language models (LLMs). While most existing studies rely on utility performance metrics, which are not robust enough due to variations in opponent behavior and game structure. To address this limitation, we propose Cognitive Hierarchy Benchmark (CHBench) , a novel evaluation framework inspired by the cognitive hierarchy models from behavioral economics. We hypothesize that agents have bounded rationality — different agents behave at varying reasoning depths/levels. We evaluate LLMs’ strategic reasoning through a three-phase systematic framework, utilizing behavioral data from six state-of-the-art LLMs across fifteen carefully selected normal-form games. Experiments show that LLMs exhibit consistent strategic reasoning levels across diverse opponents, confirming the framework’s robustness and generalization capability. We also analyze the effects of two key mechanisms (Chat Mechanism and Memory Mechanism) on strategic reasoning performance. Results indicate that the Chat Mechanism significantly degrades strategic reasoning, whereas the Memory Mechanism enhances it. These insights position CHBench as a promising tool for evaluating LLM capabilities, with significant potential for future research and practical applications. 1 Introduction The strategic reasoning ability of large language models (LLMs) has attracted much research attention. Such ability not only helps us understand how intelligent LLMs are, but also enables sophisticated applications such as realistic human behavior simulations and autonomous decision-making (e.g., interactive simulacra of human behavior [ 2 , 24 ] , constituting an essential pathway toward Artificial General Intelligence (AGI). Although recent years have seen huge improvements in the strategic reasoning ability of LLMs, how to evaluate such ability of LLMs quantitatively remains a critical challenge. Game theory [ 13 , 22 , 23 , 31 ] is a field of research that studies strategic reasoning and interactions among rational decision-makers. The mathematical framework of game theory helps to model key aspects in intelligent decision-making, such as modeling opponent behaviors and long-term planning under uncertain and competitive environments. Thus, the ability to play games could serve as a measure for evaluating the strategic reasoning capability of LLMs. Indeed, many existing studies focus on game-theoretic reasoning abilities of LLMs, especially in the setting of normal-form games or repeated games due to their relatively simple representations and the connection to classic game theory [ 2 , 8 , 20 , 21 , 25 ] . Figure 1 : The full architecture of CHBench framework. Most existing studies focus on the utility performance of the LLMs across different normal-form games to evaluate the strategic reasoning capability [ 9 , 10 , 16 , 18 , 38 ] . This approach is reasonable in its own right since game players always aim to maximize their utilities. However, the robustness of the method is limited due to both the game utility structure and uncertainties in the opponent’s behaviors [ 14 , 26 , 27 , 33 , 35 ] . Here we provide a symmetric game to show the limit of this approach. Table 1 : A symmetric two-player game A B A -8 -8 0 5 B 5 0 1 1 Example 1 . Consider the game shown in Table 1 . There are two players in the game (the row player and the column player), both with two available actions (action A and action B). The two numbers in each cell represent the utilities of the row player and the column player, respectively. It is easy to check that the Nash equilibrium of the game is for both players to choose action A with probability \frac{1}{3} and action B with probability \frac{2}{3} . Suppose that the row player is naïve and chooses both actions uniformly randomly, while the column player is able to compute the Nash equilibrium and uses the equilibrium strategy. In this case, the expected utilities of the row player and the column player are \frac{2}{3} and -\frac{1}{6} . Even if the column player is able to accurately predict the row player’s strategy and best responds accordingly (which is to choose action B with probability 1), the expected utility of the row player is 3, while the column player’s expected utility is only 0.5. In either case, the column player has stronger strategic reasoning capability, but still obtains a much lower expected utility than the row player. To address this limitation, we propose the Cognitive Hierarchy Benchmark (CHBench) framework to evaluate the strategic reasoning capability of LLMs. The cognitive hierarchy model [ 4 ] and its level- k extension originate in behavioral economics, and are used to describe human thought processes. To the best of our knowledge, this paper is the first work that adopts such models to evaluate the strategic reasoning capability of LLMs. Unlike standard game theory (which assumes perfect rationality), the cognitive hierarchy model considers limited human rationality – different people think at different levels of strategic depth. This model focuses on the players’ ability to model and predict the behaviors of their opponents rather than the obtained utility, and thus aligns better with our goal to measure the reasoning capability of LLMs. In the CHBench framework, each pure strategic player performs iterative best response to the strategies of others. For instance, consider a two-player game, player 1 at level k adopts the best response strategy against player 2 ’s strategies at levels below k . Specifically, we examine two distinct CH models: the Level-K CH model and the Poisson CH model. The main difference between the two models lies in their behavioral assumptions: the Level-K CH model assumes that each level - k player performs iterative best response to level- (k-1) players ( myopic iterative best response ), whereas the Poisson CH model assumes that each player performs iterative best response to all players at below level- k ( omniscient iterative best response ). Figure 1 illustrates the architecture of CHBench. The implementation of CHBench framework contains three phases: the data collection phase, the optimization phase, and the evaluation phase. During the former two phases, we collect behavioral data from LLMs across various game matrices and optimize the model parameters using the method of Maximum Likelihood Estimation (MLE). In the evaluation phase, given any normal-form game and the specified parameters, the model generates the level distribution for each agent, and the predicted strategy at each level for each agent. The mean of the level distribution represents the agent’s average strategic reasoning capability in the game, while the variance indicates the stability of the agent’s strategic behaviors. Section 3 introduces the process of three phases. We evaluate the strategic reasoning capability of six LLMs across different games and mechanisms. Our key findings are briefly summarized below. First, we show that setting the maximum level of cognitive hierarchy framework at \hat{k}=4 is appropriate as increasing the maximum level yields negligible improvements in likelihood while amplifying strategic instability. Second, we observe that the same LLM exhibit only slight fluctuation in expected strategies when facing different opponents in the same game. Confirming that our framework effectively measures the consistency of the strategic reasoning capability of LLMs. Finally, we propose two mechanisms to explore

Read the complete text