Measuring General Intelligence with Generated Games
Vivek Verma, David Huang, William Chen, Dan Klein, Nicholas Tomlin, Computer Science Divison · 2025
Read Measuring General Intelligence with Generated Games by Vivek Verma, David Huang, William Chen, Dan Klein, Nicholas Tomlin, Computer Science Divison (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07215.txt.
Introduction
Vivek Verma and co-authors build a fresh test for machine smarts in this 2025 paper. Most tests reuse fixed tasks, which models can memorize. Their test instead makes new games on demand, so each run can hold fresh tasks.
A large model first writes rules for a novel game, then codes it as a playable world. Self-play agents train on the game to set a bar. Test models then read the rules and play against those agents. Top chat models won few games, while reasoning models won about a third. The team shares the games and code so others can grow the test.
Catalog record
| CREATOR | Vivek Verma, David Huang, William Chen, Dan Klein, Nicholas Tomlin, Computer Science Divison |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring General Intelligence with Generated Games Vivek Verma, David Huang, William Chen, Dan Klein, Nicholas Tomlin, Computer Science Divison Abstract We present gg-bench , a collection of game environments designed to evaluate general reasoning capabilities in language models. Unlike most static benchmarks, gg-bench is a data generating process where new evaluation instances can be generated at will. In particular, gg-bench is synthetically generated by (1) using a large language model (LLM) to generate natural language descriptions of novel games, (2) using the LLM to implement each game in code as a Gym environment, and (3) training reinforcement learning (RL) agents via self-play on the generated games. We evaluate language models by their winrate against these RL agents by prompting models with the game description, current board state, and a list of valid moves, after which models output the moves they wish to take. gg-bench is challenging: state-of-the-art LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench using in-context learning, while reasoning models such as o1, o3-mini and DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games, data generation process, and evaluation code in order to support future modeling work and expansion of our benchmark. 1 Introduction Early researchers in artificial intelligence had broad ambitions of building systems capable of performing at or above human levels across arbitrary tasks. Often credited with the creation of the field of artificial intelligence, John McCarthy conjectured in his 1955 Dartmouth Conference proposal that “every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it” [ 24 ] . However, in the subsequent decades, AI research narrowed significantly, focusing on more specific problem domains like chess, rule-based expert systems like DENDRAL [ 7 ] , and knowledge engineering efforts like Cyc [ 22 , 31 ] . Concerned that the field had strayed too far from its initial ambitions, Goertzel and Pennachin [16] coined the term “artificial general intelligence” in the early 2000s and urged researchers to move beyond “narrow AI.” While the definition and usage of this term have been hotly debated in both AI and psychology [ 39 , 21 , 15 ] , in this work we follow Chollet [10] and use general intelligence to refer to the ability of a system to generalize and act in unseen contexts and environments. Figure 1 : Overview of our benchmark creation process. We start by generating descriptions of two-player strategy games, after which we generate implementations of these games as Gym environments. Lastly, we employ self-play reinforcement learning to train agents on these games In recent years, large language models (LLMs) have emerged as a potential stepping stone toward artificial general intelligence, and their performance on a wide variety of popular benchmarks has drastically increased in recent years [ 6 ] . However, a growing concern is that these gains might not reflect true advancements in their ability to generalize to new domains, but might instead simply be the result of training on larger and more relevant datasets [ 10 ] . In other words, many tasks that were previously viewed as tests of out-of-domain generalization have now been moved into the training distributions of our models. As a result, it is an open question whether today’s models can adapt and generalize to novel tasks in a way that would satisfy our definition of a generally intelligent system. In this paper, we propose a scalable approach for evaluating whether models can generalize to new domains, leveraging a key observation: LLMs are capable of generating complex tasks that they themselves are incapable of solving. Under this view, benchmarks are not static lists of questions but data generating processes , such that individual task instances can be regenerated at will. This approach allows us to generate new tasks in the result of data contamination, and also provides the possibility of generating more difficult tasks as stronger language models are developed and released. We present gg-bench , an evaluation benchmark consisting of games generated entirely by LLMs. The benchmark is created by first using LLMs to generate descriptions of two-player, turn-based strategy games designed to be played in a console environment. Then, using the same model, we generate Python implementations of each game in the form of Gym environments [ 4 ] . After this, we use self-play reinforcement learning to train agents on each of these games via proximal policy optimization [ 35 ] . Finally, in order to evaluate whether a target model can generalize to act in these generated games, we evaluate its winrate against the trained RL agents. gg-bench is challenging: state-of-the-art LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates between 7.5% and 9% on gg-bench using in-context learning, while reasoning models such as DeepSeek-R1 and o1 achieve average winrates between 31% and 36%. We analyze the diversity of generated games and identify common failure patterns of language models, revealing that their primary shortcomings are an inability to effectively strategize over multiple turns and to correctly generalize from game descriptions to new gameplay scenarios. Lastly, we release the dataset, code for generating the dataset and our experiments at https://github.com/vivek3141/gg-bench . ⬇ # Number Duel ## Objective Be the first player to ** capture all of your opponent ’ s numbers **. Utilize strategic selection and timing to outmaneuver your opponent . Victory is achieved when your opponent has no numbers remaining in their set . ## Setup 1. ** Number Range Selection **: - Determine the value of ** N **, the maximum number in each player ’ s set . A recommended starting value is ** N = 10**. 2. ** Initial Number Sets **: - Each player receives a set of unique numbers ranging from **1 to N ** inclusive . … ### Example Game Setup - ** N = 5** - ** Player 1’ s Numbers **: ‘{1, 2, 3, 4, 5}‘ - ** Player 2’ s Numbers **: ‘{1, 2, 3, 4, 5}‘ - ** First Attacker **: Player 1 ### Round 1 - ** Player 1** ( Attacker ) selects **3**. - ** Player 2** ( Defender ) selects **2**. - ** Reveal **: - Player 1: **3** - Player 2: **2** - ** Outcome **: - 3 ( Attacker ) > 2 ( Defender ): Attack successful . - ** Player 2’ s number 2 is captured **. - Player 1’ s number **3 remains ** in their set . … (a) An example game description from gg-bench . Some parts of the description are elided with ... markers. ⬇ class CustomEnv ( gym . Env ): def __init__ ( self , N =10): … self . action_space = spaces . Discrete ( N ) self . observation_space = spaces . Box ( low =0, high =1, shape =(2 * self . N + 1,), dtype = np . float32 ) self . reset () def reset ( self , seed = None ): … def step ( self , action ): … def render ( self ): output = [] output . append ( f "Current ␣ role: ␣ {’Attacker’ ␣ if