Re:Frame – Retrieving Experience From Associative Memory

Daniil Zelezetsky, Egor Cherepanov, Alexey K. Kovalev, Aleksandr I. Panov · 2025

Read Re:Frame – Retrieving Experience From Associative Memory by Daniil Zelezetsky, Egor Cherepanov, Alexey K. Kovalev, Aleksandr I. Panov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19344.txt.

Introduction

Daniil Zelezetsky and co-authors improve offline reinforcement learning by retrieving memories of expert behavior in a 2025 paper. Offline learning trains agents from logged data, which is often suboptimal because expert datasets are unavailable or impractical to collect. Agents then struggle to generalize from imperfect trajectories.

The authors show that even a tiny amount of expert experience can substantially lift performance. Their method, Re:Frame, retrieves relevant expert moments from an associative memory while learning from the abundant weaker data. Borrowing the right memory at the right time bridges the gap between cheap data and expert behavior.

Catalog record

CREATORDaniil Zelezetsky, Egor Cherepanov, Alexey K. Kovalev, Aleksandr I. Panov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Re:Frame – Retrieving Experience From Associative Memory Daniil Zelezetsky, Egor Cherepanov, Alexey K. Kovalev, Aleksandr I. Panov Abstract Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame ( R etrieving E xperience Fr om A ssociative Me mory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them into decision-making; the same AMB is queried at evaluation. This requires no environment interaction and no modifications to the backbone architecture. On D4RL MuJoCo tasks, using as few as 60 expert trajectories (0.1% of a 6000-trajectory dataset), Re:Frame consistently improves over a strong Decision Transformer baseline in three of four settings, with gains up to +10.7 normalized points. These results show that Re:Frame offers a simple and data-efficient way to inject scarce expert knowledge and substantially improve offline RL from low-quality datasets. 1 Introduction Figure 1: Re:Frame overview. A standard offline RL agent trained on suboptimal data is augmented with an Associative Memory Buffer holding a few expert data. At each step, the agent queries memory, retrieves and decodes an expert data, and integrates it to guide the next action, enriching low-quality training with scarce expert knowledge. Memory is fundamental to human intelligence, enabling us to accumulate experiences, learn from interactions, and make informed decisions in complex environments [ tulving2002episodic , SQUIRE2004171 , BADDELEY2010R136 ] . It allows seamless integration of past knowledge with present observations to guide adaptive behavior [ eichenbaum2017memory , parr2020prefrontal , parr2022active ] . A central mechanism is associative memory—the ability to recall and apply relevant experiences by linking them to the current context [ associative_mem , memory_humnan ] , enabling efficient and robust decisions even with limited information. In reinforcement learning (RL), significant advances have led to agents achieving superhuman performance in settings where exploration is possible and large amounts of high-quality data can be collected [ Mnih2015HumanlevelCT , Silver2017MasteringTG ] . However, in offline RL, agents must learn entirely from a fixed dataset without further environment interaction [ levine2020offline ] . This creates a fundamental limitation: the performance of offline RL algorithms depends directly on the quality of available data [ janner2021offline , cherepanov2023recurrent , kachaev2025a ] . In practice, collecting large datasets of expert demonstrations is rarely feasible. Safety, cost, and scalability constraints make it difficult to obtain sufficient high-return trajectories, especially in complex or real-world domains [ kumar2020conservative ] . Instead, practitioners typically face datasets dominated by medium- or low-quality behavior, with only a handful of expert trajectories available. This mismatch between abundant suboptimal and scarce expert data has motivated the development of methods that explicitly leverage mixed-quality datasets [ nair2020awac , kostrikov2021offline , zhou2021plas ] . Yet, effectively incorporating a very small number of expert trajectories into training remains a challenging open problem. Motivated by the cognitive role of associative memory, we propose Re:Frame ( R etrieving E xperience Fr om A ssociative Me mory), a simple and general framework for offline RL. Re:Frame augments an existing policy architecture (e.g., Decision Transformer (DT) [ chen2021decision ] ) with an external Associative Memory Buffer ( AMB ) that stores only a tiny fraction of expert demonstrations (e.g., 0.1% of the training dataset size). During learning on suboptimal data, the agent learns to retrieve and integrate relevant expert experiences through content-based associations, effectively using memory to compensate for the lack of exploration and the scarcity of expert trajectories. On standard D4RL [ fu2020d4rl ] benchmarks, this approach consistently improves over DT in the low-expert regime, with gains up to +10.7 normalized points. Our contribution can be summarized as follows: • We introduce Re:Frame , an associative memory framework for offline RL that enables efficient retrieval of expert demonstrations, addressing the challenge of training with predominantly suboptimal datasets and only a handful of expert trajectories. • We demonstrate that Re:Frame can be seamlessly integrated with existing architectures, significantly improving decision-making performance in offline RL benchmarks while using as little as 0.1% of expert data. 2 Related Works 2.1 Associative Memory Associative memory has long inspired research in machine learning. Hopfield Networks [ hopfield1982neural ] pioneered associative recall, Neural Turing Machines [ graves2014neural ] added external memory with content- and location-based addressing, and Associative Recurrent Memory Transformer [ rodkin2024associative ] extended transformers with associative memory for long-term dependencies. The principles of associative memory have also been applied to RL [ cherepanov2024unraveling ] . Associative Memory Prioritized Experience Replay [ li2022associative ] speeds up prioritized replay, while Fast Weight Memory [ schlag2020learning ] augments Long Short-Term Memory [ lstm ] for meta-learning and associative inference. Self-attentive Associative Memory [ le2020self ] combines item and relational memory, and Associative Search Network [ barto1981associative ] enables autonomous sensory-motor learning from reinforcement signals. Episodic Reinforcement Learning with Associative Memory [ Zhu2020EpisodicRL ] builds a graph-based memory to connect experiences and propagate values. While prior work shows the promise of associative memory in RL, such methods are tied to specific architectures. In contrast, Re:Frame is architecture-agnostic, using a memory buffer that retrieves experiences by similarity rather than temporal order or graph links, enabling flexible, context-aware long-term recall. 2.2 Offline RL with Limited Expert Data In offline RL, agents learn from a fixed dataset without exploration, so performance hinges on the quality and composition of trajectories. Classical methods - Advantage-Weighted Actor-Critic [ nair2020awac ] , Implicit Q-Learning (IQL) [ kostrikov2021offline ] , and Twin Delayed DDPG + Behavior Cloning (TD3+BC) [ fujimoto2021minimalist ] - constrain policy improvement toward the behavior distribution or weight updates by advantages/values, enabling learning from mixed-quality data with only limited expert demonstrations. Recent work addresses the setting of few expert trajectories among many suboptimal ones. Guided Offline Reinforcement Learning [ yang2023hundreds ] adapts constraints using a small expert set, Hybrid Policy Optimization [ yang2023hybrid ] refines imperfect demos with online updates, and Optimal Transport for Offline Imitation Learning [ luo2023optimal ] aligns datasets with handfuls of expert examples. Equivariant Offline Reinforcement Learning [ tangri2024equivariant ] adds symmetry priors, Guided Data Augmentation [ corrado2023guided ] generates expert-like rollouts from small seeds. Re:Frame instead equips agents with a tiny expert associative memory (about 0.1% of the dataset) that they learn to query, directly addressing the “few experts + many suboptimal” regime without modifying the backbone. Algorithm 1 Re:Frame-DT Integration 1: AMB \mathcal{B}\in\mathbb{R}^{T\times N} , trajectory \tau 2: Predicted actions \hat{a} 3: Memory Retrieval: 4: for (R_{t},o_{t},a_{t})\in\tau do 5: R^{\prime}_{t}\leftarrow\text{RtgEnc1}(R_{t}) , o^{\prime}_{t}\leftarrow\text{ObsEnc1}(o_{t}) 6: h^{*}_{t}\leftarrow\text{Linear}(\text{concat}(R^{\prime}_{t},o^{\prime}_{t})) 7: h^{\prime}_{t}\leftarrow\arg\min_{h\in\mathcal{B}}\|h^{*}_{t}-h\|_{2}^{2} 8: a^{\prime}_{t}\leftarrow\text{ActDec1}(h^{\prime}_{t}) 9: a^{\prime\prime}_{t}\leftarrow\text{Linear}(a^{\prime}_{t})\in\mathbb{R}^{1\times D} 10: end for 11: Action Generation: 12: \tau^{\prime}\leftarrow\text{Embed Sequence}(\tau) 13: a^{*}\leftarrow\text{Transformer}(\tau^{\prime})\in\mathbb{R}^{T\times D} 14: \hat{a}\leftarrow\text{ActHead}(a^{*}+a^{\prime\prime})\in\mathbb{R}^{T\times D} 15: return \hat{a} with loss \mathcal{L}(a,\hat{a}) 3 Background Offline Reinforcement Learning. Offline RL

Read the complete text