Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning
Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi · 2025
Read Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning by Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19732.txt.
Introduction
Gunshi Gupta and four co-authors propose Memo, a transformer architecture that gives embodied agents a working memory without blowing past context limits. Agents acting over long periods drown in visual input, most of it irrelevant, while a transformer's context window fills quickly. Memo teaches the model to compress its experience as it goes.
During training, periodic summarization tokens are interleaved with the inputs, so the model learns to write and later retrieve compact memory summaries. On a gridworld meta-learning benchmark and a photo-realistic indoor navigation task, Memo beat plain long-context transformer baselines while using less compute and storage. It also handled longer contexts at test time and stayed robust when old history had to be truncated.
Catalog record
| CREATOR | Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning Gunshi Gupta, Karmesh Yadav, Zsolt Kira, Yarin Gal, Rahaf Aljundi Abstract To enable embodied agents to operate effectively over extended timeframes, it is crucial to develop models that form and access memories to stay contextualized in their environment. In the current paradigm of training transformer-based policies for embodied sequential decision-making tasks, visual inputs often overwhelm the context limits of transformers, while humans can maintain and utilize a lifetime of experience compressed as memories. Significant compression is possible in principle, as much of the input is irrelevant and can be abstracted. However, existing approaches predominantly focus on either recurrent models with fixed-size memory or transformers with full-context reliance. In this work, we propose Memo, a transformer-based architecture and training recipe for reinforcement learning (RL) on memory-intensive, long-horizon tasks. Memo incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training. We demonstrate Memo’s effectiveness on a grid-world meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings. Memo outperforms naive long-context transformer baselines while being more compute and storage efficient. Additionally, Memo generalizes better to longer contexts at inference time and remains robust in streaming settings, where historical context must be truncated to fit inference constraints. 1 Introduction Humans intuitively prioritize and retain memories relevant to their current tasks, filtering out irrelevant details such as the color of a house’s walls along a route unless it serves as a crucial navigation landmark. Similarly, reinforcement learning (RL) agents must learn to selectively retain and access task-specific memories guided by task objectives rather than predefined rules. This ability is especially critical for long-horizon tasks, where delayed rewards make credit assignment challenging and necessitate efficient training and inference over extended timescales to effectively capture the relationship between past and future events. Recently, Transformers have become the go-to framework for sequence modeling due to their flexible and expressive encoding, free from the fixed-size constraints of recurrent neural networks (RNNs) [ 8 , 14 ] . However, it assumes access to all past information at each step rather than selectively retaining relevant memories, which can lead to inefficiencies and make it harder to extract task-relevant information for decision-making. Their quadratic attention complexity further limits scalability, making it challenging to process long contexts efficiently, especially when gradients must propagate over large sequences. At test time, Transformers face another practical challenge: storing an increasingly large key-value (KV) cache is computationally expensive, and attending over ever-growing contexts requires the ability to generalize beyond the temporal patterns seen during training. To address these challenges, we propose Memo , a novel framework that enhances transformer-based agents by enabling them to summarize and index past experiences through specialized summary tokens. Memo introduces a simple yet effective training procedure where transformers learn to compress their experience by periodically generating summary tokens that encode relevant past information. These summary tokens are stored in a dedicated memory buffer, allowing the model to attend to condensed representations of prior states instead of maintaining a full-context cache. This enables Memo to maintain a compact memory footprint at inference, reducing computational costs while preserving long-horizon reasoning capabilities. Inspired by advancements in extending context lengths in language models, Memo adapts these principles to reinforcement learning (RL), addressing the unique challenges presented by RL tasks. Specifically, Memo integrates with both on-policy and off-policy RL agents, leveraging learned compression mechanisms to enhance sample efficiency and generalization in sequential decision-making settings. We demonstrate its effectiveness across diverse tasks, including a grid world and a 3D indoor navigation task in Habitat. Memo matches or outperforms the naive transformer baseline, which requires storing its entire context, leading to an 8-10 \times larger cache. Despite using significantly less memory, Memo achieves better in-context learning (ICL) behavior, higher success rates, and shorter path lengths in unseen environments. On large-scale navigation tasks, we show that Memo also exhibits superior robustness in streaming settings where the cached context is truncated, demonstrating improved adaptability under limited inference budgets. Our contributions are summarized as follows: • We propose Memo, a framework that enables transformers to learn to summarize and store task-relevant past experiences, reducing computational costs while maintaining long-horizon reasoning capabilities. • We evaluate Memo on sequential decision-making tasks, demonstrating its efficiency and improved in-context learning behavior compared to transformers that require full-context storage. • We show that Memo is a general and versatile memory augmentation technique applicable to both on-policy and off-policy RL agents for long-horizon tasks. 2 Related Work In this section we briefly review research on extending context limits of transformers in language modeling, as well as the state of the art in long sequence modeling and decision making in RL. Extending Transformer Contexts in language : A significant limitation of current large language models is their restricted input context length, prompting research into methods for scaling and generalizing to larger contexts [ 33 , 21 , 13 , 27 ] . Key-value (KV) caching can reduce recomputation at training [ 9 ] or inference time [ 23 ] , by storing and reusing self-attention outputs during decoding. [ 25 ] further reduce memory by maintaining a compressed cache, though this compression is not completely task-guided. Beyond compute efficiency, recent work explores dynamic context extension. Recurrent Memory Transformer (RMT) [ 4 ] introduces summarization tokens to periodically compress and propagate prior context while discarding older tokens. Autocompressors (AC) [ 7 ] extend this by accumulating generated summaries across context windows, avoiding RMT’s fixed-size memory but still truncating gradient propagation through summaries. We extend this context summarization approach to the more involved setting of reinforcement learning (RL), where summarization must support decision-making and handle credit assignment over long horizons. Unlike AC, which fine-tunes pretrained models on a supervised token prediction task with limited gradient propagation—and aims to match but does not surpass full-context transformer baselines—we train from scratch, integrating summarization directly into the RL optimization process and propagating gradients across all summaries. This enables more effective task-driven memory formation, surpassing full-context transformer baselines in both efficiency and scalability. Transformers in RL : Transformers, while widely used in self-supervised language modeling [ 10 , 3 ] , have seen slower adoption in reinforcement learning (RL) due to challenges like sparse rewards, optimization instability, and limited batch sizes in on-policy RL. However, as we move to more complex tasks that demand long-horizon planning and reasoning over past experiences, transformers have shown promising initial results, outperforming recurrent models in POMDPs when encoding longer temporal dependencies [ 22 ] . Figure 1 : Architecture Diagram of Memo. The frames O_{1-3l_{seg}} depict the input sensor observations to the agent at timesteps 1-3l_{seg} . The figure depicts the information flow between three consecutive segments of a much longer context of inputs provided to a transformer during training. The summary tokens bottleneck the information passed on from one chunk of inputs to the next. To stabilize transformer training in partially observable RL, Parisotto et al. 2020 used KV caching while restricting gradient propagation to a small window of time steps, limiting the learning of long-term dependencies. More recently, ReLIC [ 11 ] improved convergence of on-policy RL algorithms by unfreezing the historical context within a rollout and