Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models

Youwei Liu, Jian Wang, Hanlin Wang, Beichen Guo, Wenjie Li · 2026

Read Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models by Youwei Liu, Jian Wang, Hanlin Wang, Beichen Guo, Wenjie Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08955.txt.

Introduction

Youwei Liu and four co-authors teach agents to look ahead in a fake world before they act. A world model is a learned sim of how a place changes when an agent acts. With it, an agent can plan with no risk to the real place. Past use was short: one step, or a fixed span of steps, no more.

Their method is called Imagine-then-Plan. The agent shifts its look span to fit the task at hand. Hard tasks get a long look. Easy tasks get a short one. A reader who builds agents gets a plan rule that bends with need. It may lift task scores while it keeps sim cost in check.

Catalog record

CREATORYouwei Liu, Jian Wang, Hanlin Wang, Beichen Guo, Wenjie Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models Youwei Liu, Jian Wang, Hanlin Wang, Beichen Guo, Wenjie Li Abstract Recent advances in world models have shown promise for modeling future dynamics of environmental states, enabling agents to reason and act without accessing real environments. Current methods mainly perform single-step or fixed-horizon rollouts, leaving their potential for complex task planning under-exploited. We propose Imagine-then-Plan ( ITP ), a unified framework for agent learning via lookahead imagination, where an agent’s policy model interacts with the learned world model, yielding multi-step “imagined” trajectories. Since the imagination horizon may vary by tasks and stages, we introduce a novel adaptive lookahead mechanism by trading off the ultimate goal and task progress. The resulting imagined trajectories provide rich signals about future consequences, such as achieved progress and potential conflicts, which are fused with current observations, formulating a partially observable and imaginable Markov decision process to guide policy learning. We instantiate ITP with both training-free and reinforcement-trained variants. Extensive experiments across representative agent benchmarks demonstrate that ITP significantly outperforms competitive baselines. Further analyses validate that our adaptive lookahead largely enhances agents’ reasoning capability, providing valuable insights into addressing broader, complex tasks. Our code and data will be released. 1 1 footnotetext: Corresponding author. 1 Introduction The emergence of Large Language Models (LLMs) has sparked a paradigm shift in autonomous agents, enabling them to reason and interact across a wide range of digital and physical environments ( 14 ; 21 ; 9 ) . LLM-based agents primarily leverage immediate observations and historical interaction traces to facilitate decision-making ( 33 ; 18 ) . Despite the impressive performance achieved, most agents remain constrained by shallow grounding , a state where they perceive the environment but lack a deep, causal understanding of how their current actions will ultimately reshape the environment. Without the ability to project into the future, agents are prone to catastrophic failures, discovering erroneous actions or state conflicts only after they have been irreversibly executed. Figure 1: Comparison between our ITP framework and conventional agent learning frameworks. To bridge this grounding gap, world models ( 13 ; 11 ) that focus on future state modeling have emerged as a promising solution, enabling agents to simulate environmental dynamics and “rehearse” actions in a mental sandbox ( 27 ) . Contemporary methods typically employ world models for single-step verification or with fixed-horizon rollouts 12 ; 17 ; 35 . Such rigid strategies are often suboptimal, as they fail to capture long-term dependencies in complex tasks (e.g., household embodied tasks). Furthermore, they are prone to suffer from high computational costs with unnecessary deep rollouts for trivial actions. A truly intelligent agent should be deliberative, allocating deep foresight adaptively to resolve potential state conflicts and account for long-term dependencies in pivotal, high-stakes decisions. In this paper, we propose Imagine-then-Plan ( ITP ), a framework that empowers agents to perform task planning with world model-based foresight. The core of ITP is to move beyond passive observation and perform a proactive, deliberative “rehearsal” phase, where decisions are conditioned on both present observations and potential futures. This requires the agent to internally imagine multi-step future trajectories by looking ahead. Unlike previous rigid approaches, ITP introduces an adaptive lookahead mechanism that dynamically scales the imagination horizon by trading off the ultimate goal and estimated task progress. More importantly, this shift necessitates a new conceptualization of the agent’s decision-making process. We move beyond the Partially Observable Markov Decision Process (POMDP) ( 1 ; 20 ; 24 ) toward a Partially Observable and Imaginable MDP (POIMDP). As illustrated in Figure 1 , the agent’s action policy is optimized over a dual-stream representation: the concrete present ( observable ) and the foresighted future ( imaginable ). These imagined trajectories provide rich signals, such as anticipating goal progress or detecting potential bottlenecks, allowing the agent to close the loop between planning action sequences and estimating their possible consequences. This implicit feedback enables the agent to perform self-correction when necessary, significantly reducing the need for expensive interactions with the real environment. We instantiate ITP in two variants: a training-free ( \texttt{ITP}_{\text{I}} ) variant that uses reflection as the adaptive lookahead for plug-and-play enhancement of LLM agents, and a reinforcement-trained ( \texttt{ITP}_{\text{R}} ) variant that leverages imagined futures to optimize the agent policy more effectively and more efficiently. Extensive experiments demonstrate that both variants significantly improve task success rates. Further analyses validate the vital role of our adaptive lookahead mechanism. Our contributions are summarized as follows: 1) We conceptualize the partially observable and imaginable Markov decision process (POIMDP), laying a solid foundation for integrating imagined futures and historical interactions into agent decision-making. 2) We propose Imagine-then-Plan ( ITP ), a framework that incorporates world model-based imagination with an adaptive lookahead mechanism, which provides deliberative guidance for action policy planning. 3) We demonstrate through training-free and reinforcement-trained variants that ITP significantly improves the success rates of LLM-based agents, providing valuable insights into addressing complex, long-horizon tasks. 2 Preliminaries Problem Formulation. The reasoning process of LLM agents is often formulated as a Partially Observable Markov Decision Process (POMDP), defined by (\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},\mathcal{R}) . Here, \mathcal{S} denotes the environment state space, \mathcal{A} the action space, and \mathcal{O} the observation space. \mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} represents the state transition function, and \mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow[0,1] denotes the reward function that evaluates task performance. At each time step t , the agent receives an observation o_{t}\in\mathcal{O} and selects an action a_{t}\sim\pi_{\theta}(\cdot|h_{t}) , where h_{t}=(o_{1},a_{1},\dots,o_{t}) denotes the interaction history. Executing a_{t} induces a transition to a new latent state s_{t+1}\sim\mathcal{T}(s_{t},a_{t}) , from which the environment emits the next observation o_{t+1} . The interaction terminates when the agent reaches a terminal state or exceeds a predefined maximum number of steps. LLMs as World Models. A world model is a predictive model of environment dynamics that estimates future states conditioned on actions 13 . In text-based environments, the environment state is typically represented as text and observable to the agent. We treat textual observations as state representations, denoted as s_{t} at time step t . Under this formulation, LLMs can be interpreted as world models, as they capture transition regularities by predicting the next state given the current interaction context 35 ; 15 . Concretely, a textual world model parameterized by an autoregressive LLM defines the conditional distribution p_{\phi}(s_{t+1}|s_{t},a_{t}) . The distribution is factorized at the token level, and the model generates the next state sequentially. Such world models serve as powerful proxies for environment dynamics and enable planning in language-based agents. Figure 2: Overview of the proposed Imageine-then-Plan ( ITP ) framework. It consists of two variants: (a) \texttt{ITP}_{\text{I}} , which is training-free and enables LLM agents to learn from the imagination at inference time. (b) \texttt{ITP}_{\text{R}} , which leverages imagined futures to optimize the action policy more effectively and more efficiently. 3 Method We propose I magine- t hen- P lan ( ITP ), a framework that equips LLM-based agents with adaptive lookahead via learned world models. As illustrated in Figure 2 , ITP enables agents to condition decisions on both the observable present and imagined future trajectories. 3.1 World Model Training As shown in Figure 2 , we first fine-tune a base LLM on expert demonstrations \mathcal{D}_{\mathrm{exp}} to obtain an initial agent

Read the complete text