MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systems
Anirudh Chari, Suraj Reddy, Aditya Tiwari, Richard Lian, Brian Zhou · 2025
Read MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systems by Anirudh Chari, Suraj Reddy, Aditya Tiwari, Richard Lian, Brian Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19318.txt.
Introduction
Five researchers present a planning system that lets game agents learn from their own past attempts, in a 2025 paper. Language models can plan tasks for embodied agents out of the box, but they forget everything between tries. In open worlds like Minecraft, that amnesia hurts. The team's system, MINDSTORES, keeps a store of past experiences written in plain language.
Each memory entry bundles the situation, the task, the plan, and how it turned out. When a new task arrives, the planner retrieves matching memories and reasons over them to sharpen its plan. Tests in a Minecraft research world showed the system learning and applying lessons better than rival memory-based planners. It kept the flexibility of zero-shot planning while adding steady improvement. The authors frame it as a step toward agents that grow through natural experience.
Catalog record
| CREATOR | Anirudh Chari, Suraj Reddy, Aditya Tiwari, Richard Lian, Brian Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systems Anirudh Chari, Suraj Reddy, Aditya Tiwari, Richard Lian, Brian Zhou Abstract While large language models (LLMs) have shown promising capabilities as zero-shot planners for embodied agents, their inability to learn from experience and build persistent mental models limits their robustness in complex open-world environments like Minecraft. We introduce MINDSTORES, an experience-augmented planning framework that enables embodied agents to build and leverage mental models through natural interaction with their environment. Drawing inspiration from how humans construct and refine cognitive mental models, our approach extends existing zero-shot LLM planning by maintaining a database of past experiences that informs future planning iterations. The key innovation is representing accumulated experiences as natural language embeddings of (state, task, plan, outcome) tuples, which can then be efficiently retrieved and reasoned over by an LLM planner to generate insights and guide plan refinement for novel states and tasks. Through extensive experiments in the MineDojo environment, a simulation environment for agents in Minecraft that provides low-level controls for Minecraft, we find that MINDSTORES learns and applies its knowledge significantly better than existing memory-based LLM planners while maintaining the flexibility and generalization benefits of zero-shot approaches, representing an important step toward more capable embodied AI systems that can learn continuously through natural experience. 1 Introduction Recent advances in large language models (LLMs) have demonstrated enhanced capabilities in reasoning ( Plaat et al., 2024 ; Huang & Chang, 2023 ) , planning ( Sel et al., 2025 ) , and decision-making ( Huang et al., 2024 ) through methods that strengthen analytical depth. Among the numerous domains of active innovation, the success of AI agents serves as a critical benchmark for assessing our progress toward generally capable artificial intelligence ( Brown et al., 2020 ) . Building embodied agents—AI systems with physical form—that learn continuously from real-world interactions through persistent memory and adaptive reasoning remains a fundamental challenge in the future of artificial intelligence. Classical approaches, such as reinforcement learning ( Dulac-Arnold et al., 2021 ) and symbolic planning ( Zheng et al., 2025 ) , struggle with scalability, irreversible errors, and rigid assumptions in complex environments. A promising paradigm for such agents leverages LLMs as high-level planners ( Jeurissen et al., 2024 ) : the LLM decomposes abstract goals into step-by-step plans (e.g., “mine wood \rightarrow craft tools \rightarrow smelt iron”), while a low-level controller translates these plans into environment-specific actions (e.g., movement, object interaction). This “brain and body” architecture capitalizes on the LLM’s capacity for structured reasoning while grounding its outputs in the dynamics of the physical world—a critical capability for real-world applications like robotic manipulation ( Shentu et al., 2024 ; Bhat et al., 2024 ; Wang et al., 2024b ) , autonomous navigation ( Zawalski et al., 2024 ) , and adaptive disaster response. Input Review Experience Plan Generation Predict Outcome Execute Record Failure Management Next Iteration Task: Mine Iron Review: Previous experiences for mining needs a pickaxe! Plan: Mine , craft , mine , craft , mine Predict: Probable failure, might die from hunger while finding iron ore. Review: Find food before searching for iron ore. Plan: Mine , craft , mine , craft , craft , hunt , mine Predict: Probable success! Execution: Store relevant information back into database to refer to later. Figure 1: Overview of the MINDSTORES planning architecture. The left shows the iterative experiential learning pipeline leveraging the experience database. Database-related methods are in orange, planning steps are in green, and Minecraft steps are in red. The right shows an example applying this pipeline to an example task in Minecraft. While recent LLM-based agents show promise in generating action plans for embodied tasks, many lack experiential learning, i.e., the ability to apply insights from past experiences to planning for future tasks. Unlike humans—who build mental models to generalize insights, avoid errors, and reason counterfactually (e.g., “Crafting a stone pickaxe first would enable iron mining”)—existing agents cannot synthesize persistent representations of past interactions. This gap hinders their ability to tackle long-horizon tasks in open worlds like Minecraft, where success requires inferring objectives, recovering from failures, and transferring insights across scenarios. Minecraft exemplifies these challenges: agents must explore procedural terrains, infer task dependencies (e.g., stone tools before iron mining), and adapt to unforeseen challenges. Current LLM planners, namely zero-shot architectures like DEPS ( Wang et al., 2024c ) , exhibit critical flaws: (1) they lack persistent mental models, causing repetitive errors (e.g., using wooden pickaxes for iron mining); and (2) they underutilize LLMs’ reasoning to synthesize experiential insights, producing brittle plans. To address these limitations, we propose MINDSTORES, a framework that leverages LLMs to construct dynamic mental models—internal representations guiding reasoning and decision-making, inspired by human cognition. Just as humans build simplified models of reality to anticipate events and solve problems, our approach equips agents to actively interpret experiences through structured reasoning. By analyzing failures (e.g., “Wooden pickaxes break mining iron”), inferring causal rules (e.g., “Stone tools are prerequisites”), and predicting outcomes, the LLM transforms raw interaction data into adaptive principles. MINDSTORES augments planners with an experience database storing natural language tuples (state, task, plan, outcome) and operates cyclically: observe, retrieve relevant experiences, synthesize context-aware plans, act, and log outcomes. This closed-loop process enables semantic analysis of memories, iterative strategy refinement, and outcome prediction, bridging the gap between static planning and experiential learning while grounding agent reasoning in human-like cognitive foundations. Hence, our key contributions are as follows: • A cognitive-inspired formulation of artificial mental models to enable natural-language memory accumulation and transfer learning. • MINDSTORES , a novel open-world LLM planner leveraging the above formulation to develop lifelong learning embodied agents. • Extensive evaluation of MINDSTORES in Minecraft, demonstrating a 9.4% mean improvement in open-world planning tasks over existing methods. In the remainder of this paper, we detail the theoretical foundations of mental models in Section 2 , present the MINDSTORES architecture in Section 3 , and validate its performance through experiments in Sections 4 and 5 . Our findings underscore the critical role of memory-informed reasoning in developing lifelong learning agents for open-world environments. 2 Background 2.1 Open-World Planning for Embodied Agents Planning for embodied agents in open-world environments presents unique challenges due to the unbounded action space, long-horizon dependencies, and complex environmental dynamics. In environments like Minecraft, agents must reason about sequences of actions that may span dozens of steps, where early mistakes can render entire trajectories infeasible ( Fan et al., 2022 ) . Traditional planning approaches that rely on explicit state representations and value functions struggle in such domains due to the combinatorial explosion of possible states and actions. The key challenges in open-world planning stem from two main factors. First, the need for accurate multi-step reasoning due to long-term dependencies between actions presents a significant hurdle. Second, the requirement to consider the agent’s current state and capabilities when ordering parallel sub-goals within a plan poses additional complexity. Consider the example of crafting a diamond pickaxe in Minecraft: the process requires first obtaining wood, then crafting planks and sticks, mining stone with a wooden pickaxe, crafting a stone pickaxe, mining iron ore, smelting iron ingots, and finally crafting the iron pickaxe—a sequence that can easily fail if