MemER: Scaling Up Memory for Robot Control via Experience Retrieval
Ajay Sridhar, Jennifer PanSatvik SharmaChelsea FinnStanford University {ajaysri, jrpan}@stanford.edu · 2025
Read MemER: Scaling Up Memory for Robot Control via Experience Retrieval by Ajay Sridhar, Jennifer PanSatvik SharmaChelsea FinnStanford University {ajaysri, jrpan}@stanford.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20328.txt.
Introduction
Ajay Sridhar and three co-authors present MemER, a hierarchical framework that gives robot policies a usable memory of past experience. People rely on memory for long tasks, but conditioning a robot on its full observation history is expensive and brittle, while random subsampling keeps irrelevant frames. Their high-level policy instead learns to select and track relevant keyframes from its own experience.
Those keyframes, together with the most recent frames, guide text instructions that a low-level policy executes, and the design fits existing vision-language-action models. Trained with demonstrations plus minimal language annotations, the system beats prior methods on three real-world manipulation tasks that require minutes of memory. The authors release videos and code for the approach.
Catalog record
| CREATOR | Ajay Sridhar, Jennifer PanSatvik SharmaChelsea FinnStanford University {ajaysri, jrpan}@stanford.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20328
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MemER: Scaling Up Memory for Robot Control via Experience Retrieval Ajay Sridhar, Jennifer PanSatvik SharmaChelsea FinnStanford University {ajaysri, jrpan}@stanford.edu Abstract Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and π 0.5 \pi_{0.5} as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/ . 1 Introduction In recent times, we have seen significant strides in the language-following and generalization capabilities of robotic manipulation policies ( Brohan et al. 2023 ; Intelligence et al. 2025 ; Kim et al. 2024 ; NVIDIA et al. 2025 ; Team et al. 2025 ) . While these policies are improving for real-world deployment, a critical limitation remains: the absence of long-term memory. Memory allows humans to handle the inherent partial observability found in their environment. For instance, if a person wanted to make a sandwich, they would have to recall where they saw the jar of peanut butter or the knife, especially if these items had not been recently viewed. The ability to form and retrieve memories is a crucial step towards robots solving complex, multi-step tasks. The goal of this paper is to provide an effective way to enable existing generalist policies to solve tasks that require long-term visual memory. Because conditioning on long sequences of high-dimensional image and video sequences is computationally expensive, many existing generalist end-to-end policies are trained with little to no visual history. The high memory cost makes training prohibitively expensive and model deployment unusably slow. Furthermore, long observation histories can often introduce a form of overfitting — shortcut reliance on spurious correlations between inputs and demonstrator actions ( Torne et al. 2025 ) . The policy misgeneralizes under its own state distribution, leading to performance degradation during deployment due to compounding covariate shift between states visited by the demonstrator policy and the learned policy. The suboptimal policy will generate histories that differ from those seen by the expert, which is only made worse as the observation history lengthens. Some past works have shown it is possible to expand the observation context of their policy via auxiliary losses ( Torne et al. 2025 ) , or by finetuning pretrained foundation models for action prediction with native memory capabilities ( Fang et al. 2025 ) . Although these methods significantly increase the types of tasks a robot can execute, they are challenging to naively scale to long histories. To overcome this, policies must learn to filter out and store task-relevant information from the full historical context to prevent the memory footprint from exploding on tasks that require long-range dependencies. To this end, we propose approaching long-term memory for robotic policies with a hierarchical framework. The high-level policy is a finetuned video-understanding VLM trained to output subtasks and, most importantly, to select keyframes from its fixed recent context that represent important information it will need to remember to solve the task. The low-level policy is a generalist robot policy finetuned to execute the subtask specified by the high-level policy, handling the robot-specific challenges of the task that require high-frequency inference such as kinematic control. We take advantage of the fact that these open-source VLMs are finetuned on large amounts of video understanding data. With this strong prior, we find that we only need 50 teleoperated robot demonstrations with subtask annotations to adapt these VLMs to accomplish robot-specific memory-based tasks ( Bai et al. 2025 ) . Our contribution is MemER, a framework for scaling up Mem ory in robotic control via E xperience R etrieval. We demonstrate MemER’s ability to effectively utilize task-relevant past information on three complex long-horizon tasks that require up to a few of minutes of memory. To the best of our knowledge, our real-world manipulation tasks necessitate reasoning over longer horizons than prior work. Figure 1: MemER Overview. We present MemER, a framework for scaling up memory in robotic control. MemER is able to utilize task-relevant past information to effectively reason through three complex long-horizon tasks, using a single policy trained with only a small number of expert demonstrations. 2 Related Work Memory for Long-Context Robot Policies. Memory is essential for generalist robots to complete complex tasks. Prior work primarily studies memory in the context of comparatively short-horizon tasks. For example, Torne et al. 2025 and Fang et al. 2025 use different approaches to extend the context of imitation policies from a few frames to at most two dozen. We further investigate tasks which require building memory from hundreds of frames. Similar to our approach, Fang et al. 2025 stores previous frames in memory. However, these are at most the N=10 most recent frames of context, while our method can choose to include task-relevant frames spanning the entire episode, which could contain over a thousand frames. Another body of work investigates the compression of images in the policy’s context. Li et al. 2025a compresses similar observations in pixel space, which works well for stationary cameras but not for wrist-mounted cameras, which are necessary for many dexterous robotic tasks. Zheng et al. 2025a compresses all the past observations into 2D visual traces of the motion of the end-effector and moving objects in the scene. Traces help the model reason about recent spatiotemporal changes, but this approach is hard to extend tasks that require long-term memory. Memory has also played a major role in robotic navigation research. Some navigation works represent memory with an explicit geometric and/or semantic map of the environment Henry et al. 2012 ; Yu et al. 2024 . Spatial maps of the environment are hard to apply in manipulation tasks since the robot often modifies the environment. Other works directly prompt API-Based VLMs with video context to decide where the robot should navigate ( Chiang et al. 2024 ; Sharma et al. 2023 ; Chen et al. 2024 ) . We found that existing API-Based VLMs are not sufficient to reason about robot affordances for our long-horizon manipulation tasks, so we resort to finetuning open-weight models. Foundation Models in Robotics. Recent progress in vision-language-action models (VLAs) have allowed for impressive generality in robotics. VLAs combine web-scale pretraining with expressive action decoding mechanisms to execute real-world tasks. Most VLAs usually follow one of two paradigms: 1) single end-to-end model which takes images and a language task as input and outputs actions ( Black et al. 2024 ; Brohan et al. 2023 ) or 2) a high-level policy that takes images and a language task as input