Agentic Episodic Control

Xidong Yang, Wenhao Li, Junjie Sheng, Chuyun Shen, Yun Hua, Xiangfeng Wang · 2025

Read Agentic Episodic Control by Xidong Yang, Wenhao Li, Junjie Sheng, Chuyun Shen, Yun Hua, Xiangfeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01442.txt.

Introduction

Xidong Yang and co-authors present Agentic Episodic Control, a reinforcement learning design that uses large language models to use memory better. Reinforcement learning often learns slowly from few trials and generalizes poorly. Past episodic methods add external memory, but shallow encoders bottleneck what is stored and indiscriminate retrieval wastes that memory.

A language model augmenter turns raw observations into rich semantic representations. A critical state recognizer then selects only valuable past experiences for recall. Memory use shifts from passive similarity matching to strategic, context aware recall. Across five text based BabyAI environments, the method learns two to six times more efficiently than rivals. It alone solves a hard unlock task at over 90 percent success and holds up under distribution shifts.

Catalog record

CREATORXidong Yang, Wenhao Li, Junjie Sheng, Chuyun Shen, Yun Hua, Xiangfeng Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01442

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agentic Episodic Control Xidong Yang, Wenhao Li, Junjie Sheng, Chuyun Shen, Yun Hua, Xiangfeng Wang Abstract Reinforcement learning (RL) has driven breakthroughs in AI, from game-play to scientific discovery and AI alignment. However, its broader applicability remains limited by challenges such as low data efficiency and poor generalizability. Recent advances suggest that large language models, with their rich world knowledge and reasoning capabilities, could complement RL by enabling semantic state modeling and task-agnostic planning. In this work, we propose the Agentic Episodic Control (AEC), a novel architecture that integrates RL with LLMs to enhance decision-making. The AEC can leverage a large language model (LLM) to map the observations into language-grounded embeddings, which further can be stored in an episodic memory for rapid retrieval of high-value experiences. Simultaneously, a World-Graph working memory module is utilized to capture structured environmental dynamics in order to enhance relational reasoning. Furthermore, a lightweight critical state detector dynamically arbitrates between the episodic memory recall and the world-model-guided exploration. On the whole, by combining the trial-and-error learning scheme with LLM-derived semantic priors, the proposed AEC can improve both data efficiency and generalizability in reinforcement learning. In experiments on BabyAI-Text benchmark tasks, AEC demonstrates substantial improvements over existing baselines, especially on complex and generalization tasks like FindObj, where it outperforms the best baseline by up to 76%. The proposed AEC framework bridges the strengths of numeric reinforcement learning and symbolic reasoning, which provides a pathway toward more adaptable and sample-efficient agents. 1 Introduction Reinforcement learning (RL) has emerged as a cornerstone of modern AI, driving many of its most significant breakthroughs through trial and error. RL has driven landmark achievements in game-playing, with systems like AlphaGo and AlphaZero mastering Go, Chess, and Shogi purely from self-play silver2016mastering ; silver2017mastering . Beyond games, RL has been successfully applied to robotics loppenberg2024dynamic ; ju2022transferring , embodied agents gupta2021embodied , algorithm discovery fawzi2022discovering ; mankowitz2023faster , and scientific applications tunyasuvunakool2021highly . Furthermore, RL is recently central to aligning large language models (LLMs) with human intent, as seen in recent reinforcement learning from human feedback (RLHF) approaches ouyang2022training ; trung2024reft ; shao2024deepseekmath . These advances underscore the influence of RL across domains—spanning decision-making, reasoning, and human-AI alignment. Despite its impressive achievements, RL still falls short of human-level learning in two key respects, i.e., data efficiency schwarzer2021pretraining ; ball2023efficient and generalization korkmaz2024survey ; ghosh2021generalization . While humans can learn from sparse experiences and transfer knowledge across domains, RL agents usually require millions of interactions to converge and struggle to adapt to novel or compositional tasks. These limitations reflect a deeper deficiency: the absence of cognitive mechanisms critical to human learning, including symbolic abstraction binder2011neurobiology , structured memory EICHENBAUM2004109 , and selective retrieval of relevant experiences kuhl2009strategic . Crucially, these limitations also highlight a fundamental gap: whereas humans ground their decisions in symbolic language and prior knowledge, traditional RL agents operate on low-level state-action mappings with no semantic abstraction. To address these limitations, researchers have proposed mechanisms such as Neural Episodic Control (NEC) pritzel2017neural , Differentiable Neural Dictionary (DND) ritter2018been , and episodic reinforcement learning frameworks lengyel2008hippocampal . These approaches typically incorporate external memory modules to enhance sampling efficiency. However, they always rely on shallow state representations (e.g., raw observations or low-dimensional embeddings) and lack mechanisms for strategic, semantic-level retrieval of experiences ritter2018been ; hung2019optimizing . Even advanced methods that integrate meta-learning with episodic memory are typically confined to specific task structures, making it challenging to generalize to more complex scenarios with long-term dependencies mishra2018simple ; wang2018prefrontal . In contrast, human episodic memory is supported by high-fidelity, semantically structured neural representations during encoding, and is shaped by interactions with prior knowledge and context xue2018neural ; ramey2022episodic . These interactions enable humans to form more distinctive and durable memory traces, facilitating generalization and inference beyond surface-level similarity. Cognitive neuroscience suggests that human learning emerges from the coordination of multiple systems: semantic memory for general world knowledge rogers2004structure , episodic memory for context-specific recall tulving2002episodic , and working memory for real-time reasoning and relational representation miller2018working . These systems enable humans to interpret novel environments, recall useful past experiences, and plan actions in a goal-directed manner. For instance, when faced with a new instruction, a person can decompose it using prior knowledge, retrieve similar situations, and reason over possible outcomes—all without repeated trial-and-error. Recent advances in LLMs offer an opportunity to integrate similar capabilities into RL agents. Trained on massive text corpora, LLMs encode rich semantic knowledge and support high-level reasoning, such as logical inference patel2024multi , commonsense reasoning wang2023gemini , and abstract problem solving imani2023mathprompter . Functionally, they resemble the brain’s semantic memory system, allowing agents to interpret environmental states in structured, symbolic terms. This motivates a shift from traditional RL architectures toward cognitively inspired frameworks that integrate language-driven understanding with structured memory control. Such a design emulates the human learning loop: understanding the world semantically, maintaining structured memory of entities and events, and making decisions based on both present context and relevant past experience. We introduce Agentic Episodic Control (AEC) —a framework that operationalizes this principle by integrating the LLM-based semantic encoder, dual-memory systems (episodic and graph-structured working memory), and a critical-state recognition mechanism that dynamically switches between exploration and memory recall. Besides, AEC bridges symbolic reasoning and efficient decision-making by combining an LLM-based encoder for semantic abstraction, a graph-structured working memory for relational reasoning, and a critical-state recognizer that selectively triggers episodic recall when it matters most. Our contributions are three-fold: - We proposed LLM - grounded state encoding for episodic control, coupling deep semantic understanding with episodic memory, for the first time leveraging linguistic priors to break the limitations of traditional episodic-memory RL. - We present a novel dual - memory decision architecture. Based on the critical state recognition mechanism, the agent combines long-term episodic memory with graph-based working memory, enabling it to both recall past high-reward scenarios and reason about immediately perceived structures, thereby achieving remarkable performance in non-stationary tasks. - We evaluate our method on BabyAI-Text. Our method significantly outperforms strong baselines in both sample efficiency and generalization. In addition, we conduct a preliminary exploration of cross-task memory transfer, suggesting the potential for reusable semantic representations. 2 Related Work 2.1 Neural Episodic Control Episodic control refers to capturing and retaining the most influential experiences during the learning process to guide an agent’s decision-making. Model-free episodic control (MFEC) blundell2016model employs episodic memory buffers that hold experience tuples, compressing states via Gaussian random projection or variational auto-encoders. NEC pritzel2017neural introduced a differentiable neural dictionary that stores past experiences, with convolutional networks used for state encoding. Subsequent studies modify parameterised models through episodic memory lin2018episodic ; hansen2018fast ; chen2022deep ; liang2024episodic ; lee2019sample . Recently, research has shifted to state representation in neural episodic control. Episodic Reinforcement Learning with Associative Memory (ERLAM) zhu2020episodic associates related trajectories to form policies with efficient reasoning and propagates new values through a graph built over memory items. Neural Episodic Control with State Abstraction (NECSA) li2023neural performs state abstraction on grid-based environments to exploit topological structures and enhance policy learning and generalization. However, human episodic memory encodes experiences with prior knowledge and semantic detail xue2018neural ; ramey2022episodic . We therefore introduce an LLM-based semantic encoder, aligned with this

Read the complete text