Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov · 2024

Read Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation by Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06531.txt.

Introduction

Egor Cherepanov and four co-authors bring order to the fuzzy word memory in reinforcement learning. Their work appeared in twenty twenty-four. Agents need memory to act on the past, yet the field uses the term loosely and tests it inconsistently, so claims about smart agents often mislead.

Borrowing categories from cognitive science, the team splits memory into long-term and short-term, and into knowing-that versus knowing-how kinds. Each published agent design is sorted into the scheme. A standard test protocol follows, so different memories face the same honest exam. Experiments with several agents show what goes wrong when the protocol is skipped: scores inflate and comparisons collapse. The authors offer the scheme as common ground.

Catalog record

CREATOREgor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06531

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, Aleksandr I. Panov Abstract The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the use of past information, adaptation to novel environments, and improved sample efficiency. However, the term “memory” encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent’s memory, leads to erroneous judgments about agents’ memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term vs. short-term memory and declarative vs. procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to. 1 Introduction Reinforcement Learning (RL) effectively addresses problems within the Markov Decision Process (MDP) framework. However, applying RL to tasks with partial observability remains challenging, requiring agents to efficiently process their interaction history ( Esslinger et al., 2022 ; Hausknecht and Stone, 2015 ; Ni et al., 2021 ) . In complex environments with noisy observations and long episodes, storing and retrieving important information becomes crucial ( Goyal et al., 2022 ; Graves et al., 2016 ) . Yet, the concept of “memory” in RL literature lacks unified definition. Some works define it as the ability to handle dependencies within a fixed context ( Esslinger et al., 2022 ; Ni et al., 2023 ) , others as the ability to use out-of-context information ( Parisotto et al., 2020 ) , and in Meta-RL, as the ability to adapt to new environments ( Team et al., 2023 ) . However, in the absence of clear definitions and standardized evaluation protocols, claims about memory capacity in RL agents remain vague and often misleading. Memory is frequently attributed to architectural features like recurrence or attention, yet without proper isolation of memory effects, such assumptions can be incorrect. For instance, an agent might appear to exhibit long-term memory simply due to task configurations that allow shortcuts or overlap with short-term context. As a result, many empirical evaluations risk conflating different memory mechanisms or failing to detect architectural limitations. This hinders progress in developing truly memory-capable agents and comparing models in a fair and reproducible manner. In this work, we attempt to unify and clarify the concept of memory in RL agents by treating memory as an intrinsic attribute of memory-enhanced agents, directly linking memory type classification to the agent’s internal mechanisms. These specific memory types - short-term vs. long-term and declarative vs. procedural - can be rigorously assessed through experiments in memory-intensive environments. Our classification, based on temporal dependencies and the nature of the recalled information, provides a structured framework for distinguishing memory types, enabling fair comparisons, diagnosing architectural limitations, and guiding principled improvements. It is important to clarify that our goal is not to replicate the full complexity of human memory. Rather, we selectively adapt well-established memory concepts from neuroscience - such as short-term, long-term, declarative, and procedural memory - that are already informally used in RL, but lack precise definitions and formal grounding ( Fortunato et al., 2020 ; Kang et al., 2024b ; Ni et al., 2023 ) . In summary, our main contributions are as follows: 1. We provide formal definitions of key memory types in RL – specifically, short-term (STM) vs. long-term (LTM) and declarative vs. procedural memory – grounded in neuroscience and formalized for RL settings – Section 4 . 2. We introduce a task-level decoupling of Memory Decision-Making (Memory DM) and Meta-RL , clarifying the behavioral role of memory in each category – Section 4 . 3. We propose a principled experimental methodology for evaluating STM and LTM in Memory DM tasks, including precise criteria for identifying memory boundaries – Section 4.2 . 4. We show that neglecting the proposed methodology can mislead conclusions about agent memory capabilities, highlighting the importance of proper experimental configuration – Section 5 . 2 Background 2.1 Memory of Humans and Agents Many RL studies reference memory types from cognitive science, such as long-term ( Lampinen et al., 2021 ; Ni et al., 2023 ) , working ( Graves et al., 2014 ) , associative ( Polson, 1975 ) , and episodic memory ( Pritzel et al., 2017 ) , but often reduce them to vague temporal categories (e.g., short vs. long-term), with short-term spanning a few steps and long-term hundreds. This oversimplification, ignoring the relative nature of memory, complicates meaningful evaluation. To resolve this, we formalize agent memory types and introduce a principled evaluation framework. 2.1.1 Memory in Cognitive Science Human adaptive behavior depends heavily on memory, which governs how knowledge and skills are acquired, retained, and reused ( Parr et al., 2020 ; Parr et al., 2022 ) . Memory exists in many forms, each of which relies on different neural mechanisms. Neuroscience and cognitive psychology distinguish memory by the temporal scales at which information is stored and accessed, and by the type of information that is stored. Abstracting from this distinction, a high-level definition of human memory is as follows: “ memory – is the ability to retain information and recall it at a later time ” . This definition aligns with how memory is typically understood in RL, and we adopt it to define types of RL agent memory. In neuroscience, memory is classified by timescale and behavioral function, distinguishing short-term memory, lasting seconds, from long-term memory, which can persist for a lifetime ( Davis and Squire, 1984 ) . It is also divided into declarative (explicit) and procedural (implicit) forms ( Graf and Schacter, 1985 ) : the former involves consciously recalled facts and events, while the latter relates to unconscious skills like riding a bike or skiing. Though these distinctions are well-established in neuroscience, RL requires precise, testable definitions. In what follows, we adapt these cognitive categories into a formal framework suitable for RL agents. 2.1.2 Memory in RL Memory in RL encompasses diverse agent capabilities, but its definition varies across studies. In many POMDPs, agents must retain key information to act effectively later within the same environment. This typically involves two kinds of temporal dependencies: 1) within a bounded time window (e.g., transformer context ( Esslinger et al., 2022 ; Ni et al., 2023 ; Yue et al., 2024 ) ); 2) beyond the current context, requiring persistent storage or recall ( Parisotto et al., 2020 ; Sorokin et al., 2022 ) . As noted in Section 2.1.1 , STM and LTM correspond to different temporal scopes of declarative memory (see Figure 1 ). In contrast, Meta-RL Figure 1: STM vs. LTM. t_{e} - event start, t_{r} - recall time; K - context length, \xi – correlation horizon. If the event lies beyond K ,

Read the complete text