A Close Look At World Model Recovery in Supervised Fine-Tuned LLM Planners

Patrick Emami [email protected], Nan Qiang [email protected], Peter Graf [email protected] · 2026

Read A Close Look At World Model Recovery in Supervised Fine-Tuned LLM Planners by Patrick Emami [email protected], Nan Qiang [email protected], Peter Graf [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03685.txt.

Introduction

Emami, Qiang, and Graf probe whether fine-tuned language model planners truly learn world models. Supervised fine-tuning improves end-to-end classical planning in such models. But producing plans is not the same as representing the problem. Planning tasks are complex enough that this gap has been hard to study. A set of interpretability experiments now interrogates both inner states and outputs.

Fine-tuning on valid action sequences lets models encode action validity in a linear way. Some state predicates are encoded too. Models that fail to use output scores for validity may still split valid from invalid actions inside. Wider state coverage during tuning, such as random walk data, sharpens world model recovery. The study gives a recipe for interpretability work on planning models. It also informs debates on how knowledge sits inside language models.

Catalog record

CREATORPatrick Emami [email protected], Nan Qiang [email protected], Peter Graf [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03685

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Close Look At World Model Recovery in Supervised Fine-Tuned LLM Planners Patrick Emami [email protected], Nan Qiang [email protected], Peter Graf [email protected] Abstract Supervised fine-tuning (SFT) improves end-to-end classical planning in large language models (LLMs), but do these models also learn to represent and reason about the planning problems they are solving? Due to the relative complexity of classical planning problems and the challenge that end-to-end plan generation poses for LLMs, it has been difficult to explore this question. In our work, we devise and perform a series of interpretability experiments that holistically interrogate world model recovery by examining both internal representations and generative capabilities of fine-tuned LLMs. We find that: a) Supervised fine-tuning on valid action sequences enables LLMs to linearly encode action validity and some state predicates. b) Models that struggle to use output probabilities for classifying action validity may still learn internal representations that separate valid from invalid actions. c) Broader state space coverage during fine-tuning, such as from random walk data, yields more accurate recovery of the underlying world model. In summary, this work contributes a recipe for applying interpretability techniques to planning LLMs and generates insights that shed light on open questions about how knowledge is represented in LLMs. 1 Introduction Reliable automation of complex workflows with large language models (LLMs) depends on their ability to successfully generate plans. LLMs that plan can help humans write software, recommend contingencies in the face of emergencies, and contribute to open-ended scientific discovery. However, current pretrained LLMs have significant difficulty with generating plans in zero-shot and few-shot settings ( 36 ) . Performance decreases as problem complexity increases beyond what is seen during training and LLMs hallucinate intermediate steps of plans even when arriving at the goal ( 23 ) . This has driven interest in fine-tuning to improve the planning abilities of LLMs ( 26 ; 9 ) . There is reason for optimism; supervised fine-tuning (SFT) and reinforcement learning (RL) fine-tuning, the two major LLM fine-tuning paradigms, have been used to greatly enhance the reasoning capabilities of LLMs in other domains, such as math problem solving and coding abilities. LLMs explicitly fine-tuned to solve difficult problems that require deliberation show noticeable improvements over their non-reasoning counterparts when evaluated on planning benchmarks ( 38 ) . An important question is whether LLMs that are effective planners after fine-tuning are also representing and reasoning about planning problems. One way in which LLMs might reason about a planning problem is by using a world model to decide which actions to take while forming a plan. There is evidence that suggests that LLMs do not natively reason this way about planning problems ( 9 ) . However, a popular hypothesis is that, under favorable training conditions, LLMs can learn to use world models to make predictions. Recent work ( 34 ; 18 ) studies world model recovery in board-game-playing transformers, equating recovery in Othello and Chess with the identification of internal representations that encode the game’s state. Compelling evidence that LLMs may be recovering world models to some extent is now accruing. For example, they appear to learn internal representations of meaning that are updated during dialogue ( 17 ) , as well as linear representations of numbers ( 14 ) and even their own hallucinations ( 3 ) . 39 shows that for a synthetic logic puzzle domain, LLMs learn to reason from fine-tuning while also learning interpretable representation essential to solving the puzzles. There are previous studies which suggest a different perspective: transformers are liable to learn shortcut solutions to automata ( 20 ; 35 ) and logical reasoning problems ( 43 ) , motivating our close look in the classical planning setting. In this work, we raise the following question in the context of classical planning: Does SFT with examples of plans induce world model recovery in LLMs? We adapt two popular definitions of world model recovery ( 34 ; 35 ) : 1. Internal representations : We say that a fine-tuned LLM has recovered a planning world model if it has learned representations that encode the truth values of state predicates and validity of actions in states. 2. Generation : We say that a fine-tuned LLM has recovered a planning world model if it only assigns high probability to sequences of valid actions and low probability to sequences containing invalid actions. Figure 1 : Overview of our interpretability experiments examining world model recovery in supervised fine-tuned LLM planners. We examine both internal representations and generative capabilities to assess how well models capture environment dynamics. (a) We study whether planning LLMs proficient at planning can also classify actions as valid or invalid. (b) We also investigate whether the internal representations of planning LLMs linearly encode truth values of state predicates. We find that: 1) supervised fine-tuning on valid action sequences enables linear encoding of action validity and certain state predicates; 2) models may learn internal representations that distinguish valid from invalid actions even when output probabilities fail to do so; and 3) broader state space coverage during fine-tuning (e.g., random walk data) improves recovery of the underlying world model. (a) (b) Our paper develops a framework for studying world model recovery under these definitions in classical planning, and contributes new insights to this ongoing debate by fine-tuning a collection of gemma2-9b-instruct ( 31 ) models with SFT on end-to-end plan generation. To quantify world model recovery from the lens of internal representation quality (1), we train linear probes on the hidden states of fine-tuned LLMs to predict the truth values of state predicates and the validity of actions at intermediate steps of plans. We also quantify world model recovery from the perspective of LLMs as generative models (2) by adopting a metric that classifies valid actions and invalid actions using generated token probabilities. To develop our understanding of which aspects of the fine-tuning process impact world model recovery, we fine-tune LLMs using three different data distributions and also explore fine-tuning with a chain-of-thought-style technique where intermediate states are generated between actions. Our experiments, which span two classical planning environments, show that 1. Supervised fine-tuning directly on sequences of valid actions is sufficient for an LLM to learn internal representations that linearly encode action validity and the truth values for some (but not all) state predicates. 2. LLMs that have difficulty with using predicted probabilities to distinguish between valid and invalid actions, may still have high quality internal representations capable of making this distinction. 3. Fine-tuning on planning data with good state space coverage (e.g., generated by random walks) generally achieves superior in-distribution and out-of-distribution world model recovery. Our results are discussed in further detail in Section 6 . The rest of this paper is organized as follows. Section 2 provides technical background on classical planning. We describe our SFT setup in Section 3 and the evaluation task and metrics in Section 4 . Then, Section 5 introduces our two planning environments and presents planning performance results. Our interpretability results are provided in Section 6 . We review related literature in Section 7 and then conclude in Section 8 . 2 Goal-directed deterministic classical planning In this section, we introduce the goal-directed deterministic classical planning problem. An example of

Read the complete text