ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search

Sam Earle 1 , Ahmed Khalifa 2 , Muhammad Umair Nasir 3 , Zehua Jiang 1 , Graham Todd 1 , Andrzej Banburski-Fahey 4 , Julian Togelius 1 · 2025

Read ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search by Sam Earle 1 , Ahmed Khalifa 2 , Muhammad Umair Nasir 3 , Zehua Jiang 1 , Graham Todd 1 , Andrzej Banburski-Fahey 4 , Julian Togelius 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06524.txt.

Introduction

Sam Earle, Ahmed Khalifa, and colleagues present a system that designs small puzzle games on its own in a 2025 study. Automatic game design has leaned on large pre-trained models under constant human watch. Letting a system generate, test, and revise games over longer stretches, linked directly to a game engine, remains an open goal.

Their system, ScriptDoctor, uses a language model plus tree search to write games in PuzzleScript, a tight language for tile-based puzzles. Generated games are played and tested inside the engine without a person in the loop. The work shows a fuller pipeline for machine game creation.

Catalog record

CREATORSam Earle 1 , Ahmed Khalifa 2 , Muhammad Umair Nasir 3 , Zehua Jiang 1 , Graham Todd 1 , Andrzej Banburski-Fahey 4 , Julian Togelius 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search Sam Earle 1 , Ahmed Khalifa 2 , Muhammad Umair Nasir 3 , Zehua Jiang 1 , Graham Todd 1 , Andrzej Banburski-Fahey 4 , Julian Togelius 1 Abstract There is much interest in using large pre-trained models in Automatic Game Design (AGD), whether via the generation of code, assets, or more abstract conceptualization of design ideas. But so far this interest largely stems from the ad hoc use of such generative models under persistent human supervision. Much work remains to show how these tools can be integrated into longer-time-horizon AGD pipelines, in which systems interface with game engines to test generated content autonomously. To this end, we introduce ScriptDoctor, a Large Language Model (LLM)-driven system for automatically generating and testing games in PuzzleScript, an expressive but highly constrained description language for turn-based puzzle games over 2D gridworlds. ScriptDoctor generates and tests game design ideas in an iterative loop, where human-authored examples are used to ground the system’s output, compilation errors from the PuzzleScript engine are used to elicit functional code, and search-based agents play-test generated games. ScriptDoctor serves as a concrete example of the potential of automated, open-ended LLM-based workflows in generating novel game content. I Introduction Can large language models (LLMs) create complete games? Spend some time scrolling through the right social media feeds, and you will see ample examples of this happening. So, apparently, yes, they can. Yet, many questions are hard to answer. How good are these games? We do not yet have the capabilities to play and test games in general, so this requires human evaluation. With the very large number of small games in the training set of a modern LLM, there remains the suspicion that the generated games are at best a minor variation on something it has already seen. The lack of automated evaluation is also limiting our ability to understand how to make an LLM produce the games we want. In this paper, we seek to shed light on some of these questions not by investigating unconstrained game generation in JavaScript or Python, but by using a model organism of sorts. We investigate LLM-based game generation in a domain-specific language that allows for the creation of complete games in a manageable amount of text, which allows for generating diverse high-quality games, which allows for automatic game-testing through AI-based game-playing, and for which we have an approximate understanding of which games have ever been made public. Our model organism is PuzzleScript 1 1 1 https://www.puzzlescript.net/ , a domain-specific language for puzzle games. This has multiple advantages. PuzzleScript was created as a way of prototyping puzzle games, and has since been used by hundreds of game designers to create thousands of games, some of them quite good and not all of them puzzle games. Importantly, the PuzzleScript engine has a standard format for input and output, and is lightweight enough to allow rapid rollouts. Fig. 1 : The ScriptDoctor automatic game generation pipeline. An LLM iterates on PuzzleScript code given feedback from the compiler and a search-based player agent. Its output is parsed and repaired where possible according to a context-free grammar, and its prompt is augmented by documentation, examples of human games, and design ideas generated by a “brainstorming” agent. II Related work Generating game rules or complete games is relatively uncommon in PCG work [ 1 ] . This is due to the complexity of the task and the difficulty of algorithmically evaluating overall game quality. Most game generation is based on the search-based paradigm [ 2 ] using playability information yielded from simulation and/or expert knowledge. [ 3 ] creates novel board games while [ 4 ] generates rules for Pac-Man-like games. A number of subsequent papers describe attempts to generate game rules or full games using similar ideas [ 5 , 6 , 7 , 8 , 9 ] . Particularly relevant to this paper is [ 10 ] , which generates PuzzleScript games. They fixed the number of game objects, and used a genetic algorithm to evolve a small subset of game rules. They use a constructive level generator that takes the generated rules and uses expert knowledge to place objects. The final levels and rules are evaluated using an automated player and compared to expert knowledge about the domain and game player data. The authors argue that having a better orchestration [ 11 ] between levels and rules using co-evolution [ 12 ] might improve the results. Even so, the possibility space of generated games is limited by dependency on hand-crafted constructive algorithms. With the rise of LLMs [ 13 ] , one might think that the models would have some heuristics, perhaps implicit, for what makes a game good. This pushed researchers to explore the capabilities of these models in games [ 14 ] . [ 15 ] few-shot prompt LLMs to generate a simple maze game for the General Video Game framework [ 16 ] . [ 17 ] leverage LLMs’ understanding of programming languages to generate arcade games written in a simple JavaScript game engine. [ 18 ] mix search-based methods [ 2 ] with LLMs to generate new games in the Ludii framework using a fine-tuned model. This work sets itself apart from previous ones as it is not just producing game rules but also levels, and to some extent, visual aesthetics and narrative. We attempt to give LLMs the power to orchestrate between several different facets of the game [ 11 ] and produce a full PuzzleScript game in an unconstrained fashion. While this work investigates how the pre-trained knowledge of LLMs can be leveraged out of the box to support designers/developers, future work might investigate fine-tuning LLMs for this purpose, as in GAVEL [ 18 ] . III Methods In this section, we outline ScriptDoctor, a pipeline for the automatic, iterative generation of entire PuzzleScript games that leverages feedback from a tree search-based player agent. A schematic overview of the system is depicted in 1 . Our system is developed as a fork of the original PuzzleScript repository 2 2 2 https://github.com/increpare/PuzzleScript . We communicate with the existing game engine via a Python server. The server is responsible for controlling the hyperparameters of the game generation process and querying LLMs. On the client side, we compile the generated PuzzleScript games, and compilation errors or warnings from the engine are returned as potential feedback for the LLM. If the game compiles, each level is tested for playability using a breadth-first search algorithm. The algorithm runs until a solution is found or it explores 1M unique nodes. The results (solvability, number of expanded nodes, and solution length) from the BFS algorithm are sent back to the server to help with the generation process. The language model is given 10 total chances to output functional code. A generated script is deemed successful when it compiles successfully and each of its levels admits a solution of length >10 . At this point, the game generation trial is terminated. Otherwise, the LLM is prompted with the code that it generated at the prior iteration, any compilation warnings and errors that appeared in the

Read the complete text