MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agents
Vishnu Sashank Dorbala, Dinesh Manocha · 2026
Read MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agents by Vishnu Sashank Dorbala, Dinesh Manocha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20831.txt.
Introduction
Vishnu Sashank Dorbala and Dinesh Manocha give embodied agents a memory that a model actively runs. Their 2026 paper presents MemCtrl. Foundation models personalize their choices through in-context learning. But the context window is small, so memory must be compressed and fetched. Systems like RAG treat memory as a large offline store. That view fits embodied agents poorly.
An embodied agent, such as a robot, works online under tight limits. It has strict memory and compute budgets. MemCtrl uses a multimodal model as an active memory controller instead. The model decides what to keep, drop, and fetch as events unfold. The paper argues this active style suits agents that act in the world. It points toward robots that remember what matters and forget the rest.
Catalog record
| CREATOR | Vishnu Sashank Dorbala, Dinesh Manocha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agents Vishnu Sashank Dorbala, Dinesh Manocha Abstract Foundation models rely on in-context learning for personalized decision making. The limited size of this context window necessitates memory compression and retrieval systems like RAG. These systems however often treat memory as large offline storage spaces, which is unfavorable for embodied agents that are expected to operate under strict memory and compute constraints, online. In this work, we propose MemCtrl, a novel framework that uses Multimodal Large Language Models (MLLMs) for pruning memory online. MemCtrl augments MLLMs with a trainable memory head μ \mu that acts as a gate to determine which observations or reflections to retain, update, or discard during exploration. We evaluate with training two types of μ \mu , 1) via an offline expert, and 2) via online RL, and observe significant improvement in overall embodied task completion ability on μ \mu -augmented MLLMs. In particular, on augmenting two low performing MLLMs with MemCtrl on multiple subsets of the EmbodiedBench benchmark, we observe that μ \mu -augmented MLLMs show an improvement of around 16 % 16\% on average, with over 20 % 20\% on specific instruction subsets. Finally, we present an qualitative analysis on the memory fragments collected by μ \mu , noting the superior performance of μ \mu augmented MLLMs on long and complex instruction types. 1 Introduction Figure 1: Overview : We present MemCtrl, a novel memory filterting scheme to improve decision making performance on small MLLMs tackling embodied tasks. Our approach proposes a trainable memory head (green box labeled “Memory”) that learns to actively filter out redundant observations on-the-go. This form of active filtering alleviates issues with inefficient retrieval from stored observations, while also enabling scalability as a detachable memory head. An overarching goal of Embodied AI is the development of a generalist agent that can perform consistently well with high success on diverse tasks, environments and instructions ( 25 ) . A common paradigm to achieve this has been to utilize foundation models to develop task solving frameworks 18 ; 32 . While a few of these methods generalize well to diverse tasks and instructions ( 5 ; 33 ) , they are constrained by high training costs, prohibiting them from quickly being able to adapt to novel real-time settings where the data is out of distribution. Further, finetuning large foundation models incurs significant computational capacity, proving to be a significant hurdle in the democratization of these methods ( 12 ) , especially in the context of robotics, where computation on the edge is of vital importance. Figure 2: Comparison with Prior Work : We present MemCtrl, a novel approach to train “memory heads” to filter observations on the go. Prior work either used the entirety of stored observations as context (left) or filtered them via a variety of Retrieval Augmented Generation (RAG) based schemes ( red arrows), both of which assume the parsing of large amounts of data offline. MemCtrl introduces transferrable heads to use on MLLM backbone ( green arrows) to actively filter observations. An alternative, more feasible paradigm has been a modular system where foundation models are used in conjunction with memory banks ( 35 ; 28 ) of past experiences and reflections. Foundation models including very large Multimodal Large Language Models (MLLMs) such as LLaMA 4 26 and Deepseek V3 3 are limited by the size of their context window, and developing methods to refine and selectively pass memory as context is an active area of research ( 30 ) . Prior work in this area includes use of intrinsic model editing techniques for memory injection ( 17 ; 16 ) , or extrinsic interactions with episodic logs, Retrieval-Augmented Generated (RAG) 8 , or long-range latent states ( 20 ; 27 ) . While both paradigms have shown improved performance in multi-step reasoning, their implementation on embodied robot agents raises practical issues. Embodied agents providing assistance often use small models ( <20 B parameters) that work locally on-device, with often limited or only cloud access to large memory storage. Moreover, these agents need to generalize to novel settings, making it preferable to have modular, lightweight segments that are easily transferrable . To model a more efficient memory framework for embodied agents, we draw inspiration from how humans store memories of experiences. While performing various embodied tasks, humans do not accumulate every observation for later retrieval, but rather learn to actively filter out only certain vital fragments that we assume to be relevant to our task 10 . When queried, we reconstruct the missing fragments of memory through commonsense reasoning. This makes us humans highly efficient reasoners even with limited storage. We aim to endow compact embodied agents with a similar ability: rather than relying on large external memory banks or complex retrieval pipelines, the agent must actively learn to store vital memories while filtering redundant ones on the go. Learning this skill across a wide range of tasks would enable scalable self-improvement under tight computational and memory budgets. Main Results: To address these issues, we present MemCtrl , a transferrable memory augmentation scheme that aims to improve the embodied decision making performance of small models. MemCtrl introduces a trainable memory head that learns to selectively store memories of importance , increasing both parameter and memory efficiency for self-improving embodied agents. Our contributions are as follows: • Active Memory Filtering: We introduce two lightweight memory heads \mu trained on top of a frozen MLLM backbone to actively filter observations to determine which to keep and which to discard in memory. Unlike prior retrieval-based work involving filtering large observational data offline, \mu enables the MLLM to engage in real-time filtering, which is particularly useful in memory-constrained settings involving small models. • Transferrable Heads: \mu is model-agnostic and attaches to any off-the-shelf MLLM without having to finetune or edit the backbone to remove redundant observations for more prudent decision making. This modular design allows MemCtrl to transfer across embodied setups and vision-language backbones, enabling its scalable transfer across embodied agents in diverse settings. • Improving Small Model Performance: Finally, attaching \mu to the worst performing agents on the Habitat 21 and ALFRED 24 splits of EmbodiedBench 32 shows a significant improvement on task performance of around 16\% on average, while also storing significantly fewer observations. This efficiency makes MemCtrl favorable for real-world deployment. 2 Related Work 2.1 MLLMs in Embodied AI Several works in recent literature have leveraged large language models (LLMs) for high-level robot planning. For example, 1 introduce a framework (SayCan) where an LLM translates natural-language instructions into feasible robot actions constrained by a set of learned skills. This approach demonstrated that LLMs can provide semantic task knowledge, but it relies on a fixed library of affordance-grounded skills and struggles with adapting to novel situations. Recent multimodal LLMs extend this idea by directly integrating visual inputs. For instance, PaLM-E ( 5 ) is a vision-language model that outputs robotic actions, achieving good generalizability across manipulation and navigation tasks. However, PaLM-E requires a lot of training data, limiting its real-time adaptability. Similarly, RT-2 ( 36 ) augments a vision-language model with web-scale pretraining to create a vision-language-action (VLA) agent for improved zero-shot object understanding. However,