Baba is LLM: Reasoning in a Game with Dynamic Rules
Fien van Wetten, Aske Plaat, Max van Duijn · 2025
Read Baba is LLM: Reasoning in a Game with Dynamic Rules by Fien van Wetten, Aske Plaat, Max van Duijn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19095.txt.
Introduction
Fien van Wetten, Aske Plaat, and Max van Duijn test language models on the puzzle game Baba is You, in a 2025 paper. In the game, players move text blocks to rewrite the rules as they play. Winning takes both language skill and step-by-step reasoning.
Six models were tried with three kinds of prompts, from simple to rule-rich. Two smaller models were also fine-tuned on game data. The largest model did best, while small untuned models often missed how rule changes work. Fine-tuning helped models analyze levels but not solve them much better. The authors conclude that reasoning about changing rules stays hard for current models.
Catalog record
| CREATOR | Fien van Wetten, Aske Plaat, Max van Duijn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Baba is LLM: Reasoning in a Game with Dynamic Rules Fien van Wetten, Aske Plaat, Max van Duijn Abstract Large language models (LLMs) are known to perform well on language tasks, but struggle with reasoning tasks. This paper explores the ability of LLMs to play the 2D puzzle game Baba is You , in which players manipulate rules by rearranging text blocks that define object properties. Given that this rule-manipulation relies on language abilities and reasoning, it is a compelling challenge for LLMs. Six LLMs are evaluated using different prompt types, including (1) simple, (2) rule-extended and (3) action-extended prompts. In addition, two models (Mistral, OLMo) are finetuned using textual and structural data from the game. Results show that while larger models (particularly GPT-4o) perform better in reasoning and puzzle solving, smaller unadapted models struggle to recognize game mechanics or apply rule changes. Finetuning improves the ability to analyze the game levels, but does not significantly improve solution formulation. We conclude that even for state-of-the-art and finetuned LLMs, reasoning about dynamic rule changes is difficult (specifically, understanding the use-mention distinction). The results provide insights into the applicability of LLMs to complex problem-solving tasks and highlight the suitability of games with dynamically changing rules for testing reasoning and reflection by LLMs. 1 Introduction Artificial Intelligence (AI) has a long history in using games as benchmarks for reasoning, decision-making, and problem-solving capabilities ( Campbell et al. 2002 ; Silver et al. 2016 ; Silver et al. 2018 ; Brown and Sandholm 2018 ; Berner et al. 2019 ; Schrittwieser et al. 2020 ) . This paper investigates the use of large language models (LLMs) in the 2D puzzle game Baba is You ( Teikari 2019 ) . In this game players must alter rules by manipulating text blocks. Solving puzzles in this environment requires understanding how rule changes affect the game state and to apply that understanding dynamically, implying a form of reasoning in which the model should be able to reflect on the effects of its own actions. LLMs, based on the transformer architecture ( Vaswani et al. 2023 ) have demonstrated strong performance in natural language processing tasks including text generation, machine translation, conversational agents and code generation ( Naveed et al. 2024 ) . Techniques such as finetuning ( Xu et al. 2023 ) , reinforcement learning with human feedback (RLHF) ( Chaudhari et al. 2024 ) , and prompt-based learning ( Kamath et al. 2024 ) have been developed to improve the performance of the transformer architecture. Beyond natural language processing, LLMs are also emerging as agents in games ( Bakhtin et al. 2022 ; Topsakal et al. 2024 ; Marincioni et al. 2024 ; Müller-Brockhausen et al. 2023 ) . Baba is You is compelling because of its dynamic rule system relying on two-level language-based mechanics, which can be understood in terms of the classical mention versus use distinction ( Wilson 2017 ; Saka 1998 ) . Board games in general are based on pushing around pieces associated with a fixed meaning (mention). However, in Baba is You certain pieces can form a new game rule when they are aligned (use). Unlike games with fixed rules, Baba is You allows players to rewrite the logic of the game by manipulating the pieces. While LLM’s strong language and general pattern-learning abilities suggest potential ( Mirchandani et al. 2023 ) , an initial study by Cloos et al. 2024 showed indeed that state-of-the-art LLMs struggle with the reasoning aspects of Baba is You , failing to generalize rule manipulation. This paper evaluates how well LLMs solve Baba is You puzzles. We use two approaches: prompt-based learning and finetuning. We test six LLMs (GPT-4o, Gemini-Flash 1.5, OLMo 2 13B and 7B, Mistral 7B and Mixtral 8x7B) across three prompt types. We additionally finetune Mistral 7B and OLMo 7B using game data. Our contributions are as follows: • Comparing different prompts in six LLMs, we find that prompt-based learning achieves weak results when dynamic rule changes are necessary—even for LLMs with enhanced reasoning capacities; • Using a dataset for finetuning, we find that finetuning on two open LLMs is able to improve performance somewhat; • Reasoning about dynamic rules changes, remains a challenging problem for current Reasoning LLMs; the deceptively simple puzzle game of Baba is You offers a challenging testbed for Reasoning LLMs. All training scripts, prompts, and finetuning datasets of this work are publicly available ( van Wetten 2025a ; van Wetten 2025c ; van Wetten 2025b ) . 2 Related work With the advent of LLMs, a new type of learning has emerged: prompt-based (or in-context) learning ( Kamath et al. 2024 ) . This type of learning occurs at inference time, using a structured prompt that includes a task description, optional examples, and a query. To enhance LLM reasoning, chain of thought (CoT) prompting was introduced, where the model is guided to generated intermediate steps before answering ( Wei et al. 2022 ) . In few-shot CoT prompts include a task or question, followed by a step-by-step reasoning example along with the final answer, and ending with a similar question or task. This approach showed better performance on complex reasoning tasks for large models. Kojima et al. 2022 proposed a zero-shot CoT template for reasoning, they unlock the reasoning step by adding the Let’s think step by step sentence at the end of each prompt. Wang et al. 2023a introduce plan and solve prompting (PS), another zero-shot CoT method. It prompts the model to first plan a solution and then execute it, using the sentences Let’s first understand the problem and devise a plan to solve the problem. Then, let’s carry out the plan and solve the problem step by step . They extend these sentences with more detailed instructions to reduce errors in the reasoning step. Further approaches on reasoning in LLMs can be found in ( Chu et al. 2024 ; Dong et al. 2023 ; Huang and Chang 2023 ; Plaat et al. 2024 ) . Another approach to teaching a pretrained LLM to perform a new task is finetuning, where the parameters of the model are adjusted ( Jeong 2024 ) . Finetuning adapts pretrained LLMs to specific task with smaller, domain-specific datasets. A common method is supervised finetuning (SFT), where labeled examples guide learning. Instruction-tuning, a variant of SFT, trains models on (Instruction, Output) pairs, where Instruction is a human instruction and Output is the desired response by the LLM for that instruction ( Zhang et al. 2024 ) . Full finetuning updates all model parameters. This technique can be costly, as the pretrained model often contains billions of parameters, see early models such as GPT ( Radford et al. 2018 ) . Parameter-efficient finetuning (PEFT) offers a lighter alternative by modifying only a small subset of parameters ( Han et al. 2024 ) . A notable PEFT method is LoRA ( Hu et al. 2021 ) , which inserts low-rank matrices to approximate weight updates, reducing memory and compute costs. The rise of AI agents achieving dominance in gaming begins in the 1990s with Deep Blue ( Campbell et al. 2002 ) , which defeated world chess champion Garry kasparov using