Dyna-Think : Synergizing Reasoning, Acting, and World Model Simulation in AI Agents

Xiao Yu, Baolin Peng, Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao & Zhou Yu · 2025

Read Dyna-Think : Synergizing Reasoning, Acting, and World Model Simulation in AI Agents by Xiao Yu, Baolin Peng, Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao & Zhou Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00320.txt.

Introduction

Xiao Yu and co-authors teach AI agents to daydream before acting in this 2025 paper. Long computer tasks need planning, yet the reasoning tricks that shine in math do not transfer cleanly. Their framework, Dyna-Think, weaves an inner world model into the agent's thinking loop, so each planned action gets simulated first.

Two training stages build the skill. One rewrites an expert reasoner's traces to stress simulation of the chosen action, and the agent imitates those traces. The next stage first sharpens world modeling through prediction and critique tasks, then polishes action choice. On two computer-use benchmarks, the agent matched a top reasoner's best-of-many scores while using half the words.

Catalog record

CREATORXiao Yu, Baolin Peng, Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao & Zhou Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00320

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dyna-Think : Synergizing Reasoning, Acting, and World Model Simulation in AI Agents Xiao Yu, Baolin Peng, Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao & Zhou Yu Abstract Recent progress in reasoning with large language models (LLMs), such as DeepSeek-R1, demonstrates impressive capabilities in domains like mathematics and coding, by exhibiting complex cognitive behaviors such as verification, goal decomposition, and self-reflection. However, it is unclear what behavior is effective and what behavior is missing for agentic tasks. In this work, we propose Dyna-Think, a thinking framework that integrates planning with an internal world model with reasoning and acting to enhance AI agent performance. To enable Dyna-Think, we propose Dyna-Think Imitation Learning ( DIT ) and Dyna-Think Dyna Training (DDT). To initialize a policy with Dyna-Think, DIT reconstructs the thinking process of R1 to focus on performing world model simulation relevant to the proposed (and planned) action, and trains the policy using this reconstructed data. To enhance Dyna-Think, DDT uses a two-stage training process to first improve the agent’s world modeling ability for next state prediction and critique generation, and then improve the agent’s action via policy training. We evaluate our methods on OSWorld, and demonstrate that Dyna-Think improves the agent’s in-domain and out-of-domain performance, achieving similar best-of-n performance compared to R1 while generating 2x less tokens on average. Our extensive empirical studies reveal that 1) using critique generation for world model training is effective to improving policy performance; and 2) improving a world model is effective to improving the performance of its AI agent. Our results suggest a promising research direction to integrate world-model simulation into AI agents to enhance their reasoning, planning, and acting capabilities. 1 Introduction Autonomous AI agents powered by large language models (LLMs) have offered substantial promise in real-world applications, automating digital tasks such as software engineering ( Jimenez et al. 2024 ; Wang et al. 2024b ; Yang et al. 2024a ) , web navigation ( Liu et al. 2018 ; Yao et al. 2023a ; Zhou et al. 2024b ; Koh et al. 2024a ) , computer-use ( Xie et al. 2024 ; Anthropic 2025b ; Qin et al. 2025 ) , and mobile device control ( Rawles et al. 2023 ; Rawles et al. 2024 ; Trivedi et al. 2024 ) . As many computer use tasks require agents to interact with complex environments to achieve long-term objectives, one critical challenge is to reason and act efficiently over a large decision space. Recent methods of test-time scaling ( Snell et al. 2024 ) offer a potentially promising solution. For example, Yu et al. 2023 ; Yu et al. 2024b ; Zhou et al. 2024a show that LLM agents can significantly improve performance using search algorithms such as Monte Carlo Tree Search (MCTS) to allow additional interactions with the environment before decision-making. However, these methods require a large amount of expensive, time-consuming interactions with a real (or a separately learned) world model, limiting their applicability in real-world scenarios. Alternatively, many recent work ( OpenAI 2024 ; Valmeekam et al. 2024 ; Zhou et al. 2025 ) finds that LLM agents can also improve their performance by “thinking” longer, effectively internalizing parts of search into their reasoning process. For example, models such as OpenAI o1/o3 ( OpenAI 2024 ; OpenAI 2025a ) , Claude-3.7-Sonnet ( Anthropic 2025a ) , and DeepSeek-R1 ( DeepSeek-AI et al. 2025 ) generate significantly longer reasoning chains when faced with challenging tasks, exhibiting behaviors such as self-reflection, goal decomposition, verification, exploration, and more ( Gandhi et al. 2025 ) . Although many of these behaviors are crucial for solving challenging long-horizon tasks, recent work also finds that these models suffer from problems such as overthinking and fact-ignoring ( Cuadron et al. 2025 ; Zhou et al. 2025 ) , and are less efficient in certain tasks compared to non-reasoning models such as GPT-4o ( Zhou et al. 2025 ) . It is thus unclear what type of “thinking” is crucial to an agent’s performance for long-horizon tasks, and how to improve it via learning from experiences. Figure 1 : Dyna-Think focus on integrating world model simulation (shown in green ) with reasoning and acting. Behaviors that is not necessary to cause/has unclear contribution to the final action is shown in orange . After training, we find Dyna-Think achieves similar BoN performance compared to R1, while generating 2x less tokens on average and being only 32B in size. The successes in Go, games, and robotics reveal that modeling and interacting with the environment is key to solving challenging long-horizon tasks. The Dyna algorithms ( Sutton 1991 ) offer a promising approach to combine real-world interactions with planning via simulation to improve an agent’s policy. However, modeling the whole environment for tasks such as computer-use (e.g., predicting screen content after entering ‘amazon.com’ in chrome) is challenging ( Pathak et al. 2017 ; Wang et al. 2024a ; Fang et al. 2025 ; Gu et al. 2025 ) . In contrast, research in cognitive sciences ( Marr 1982 ; Rao & Ballard 1999 ) shows that the human brain encodes a compressed representation of the external world, capturing only statistical regularities and meaningful structures related to current tasks. We thus propose Dyna-Think, a thinking framework that performs “compressed” world model simulation based on model-generated states and critiques (rewards), and integrates it with reasoning and acting to improve an AI agent’s performance. Figure 1 illustrates the difference between the Dyna-Think agent and DeepSeek-R1 using an example. The Dyna-Think framework consists of Dyna-Think Imitation Learning ( DIT ) and Dyna-Think Dyna Training (DDT). DIT improves a baseline agent policy by first distilling simulated experiences from R1-generated thinking tokens, and then revising the baseline policy using the simulated experiences via supervised learning. We finds models after DIT achieve a similar performance to R1-distilled models, but with 2X less thinking tokens on average. DDT is an extension to the Dyna-Q method ( Sutton1998 ) that combines policy learning and world model training. Similar to Dyna-Q, DDT first constructs policy and world model training data using online rollouts in a real (digital) environment. Different from Dyna-Q, DDT is applicable to the problems with arbitrarily large state-action space by leveraging the prior knowledge encoded in a pre-trained LLM, and performs both policy learning and world model training based on a single LLM . During world model training, the model encodes state transitions by optimizing training objectives of next-state prediction and critiquing the states generated by its own. During policy learning, the agent’s actions are improved using successful rollouts via reinforcement learning. We evaluate Dyna-Think on the challenging OSWorld benchmark ( Xie et al. 2024 ) . Results show that Dyna-Think improves the agent’s in-domain and out-of-domain performance compared to only performing policy training (e.g., reinforcement learning) or training a separate world model (e.g., as in Dyna-Q), and that Dyna-Think leads to highly capable and cost-effective agents. For example, our 32B-parameter Dyna-Think model achieves a similar best-of-n performance to the much larger 685B-parameter R1 model, with 2x less tokens on average. A detailed analysis also reveals that 1) the critique-style world model training is effective in improving policy; and 2) the AI agents equipped with a stronger world

Read the complete text