Learning to Forget – Hierarchical Episodic Memory for Lifelong Robot Deployment

Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour · 2026

Read Learning to Forget – Hierarchical Episodic Memory for Lifelong Robot Deployment by Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11306.txt.

Introduction

Bärmann and three colleagues give humanoid robots a memory that learns what to forget. Their H2-EMV framework builds hierarchical episodic memory from continuous perception, drops details by language-model relevance under learned natural-language rules, and revises those rules from user feedback. Tests on simulated household tasks and 20.5 hours of real recordings from ARMAR-7 cut memory size by 45 percent and query compute by 35 percent while holding question-answering accuracy, which rose 70 percent on second-round queries.

Catalog record

CREATORLeonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12849
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning to Forget – Hierarchical Episodic Memory for Lifelong Robot Deployment Leonard Bärmann, Joana Plewnia, Alex Waibel, Tamim Asfour Abstract Robots must verbalize their past experiences when users ask “Where did you put my keys?” or “Why did the task fail?” Yet maintaining life-long episodic memory (EM) from continuous multimodal perception quickly exceeds storage limits and makes real-time query impractical, calling for selective forgetting that adapts to users’ notions of relevance. We present H 2 -Emv , a framework enabling humanoids to learn what to remember through user interaction. Our approach incrementally constructs hierarchical EM, selectively forgets using language-model-based relevance estimation conditioned on learned natural-language rules, and updates these rules given user feedback about forgotten details. Evaluations on simulated household tasks and 20.5-hour-long real‑world recordings from Armar -7 demonstrate that H 2 -Emv maintains question-answering accuracy while reducing memory size by 45% and query-time compute by 35%. Critically, performance improves over time—accuracy increases 70% in second-round queries by adapting to user-specific priorities—demonstrating that learned forgetting enables scalable, personalized EM for long-term human-robot collaboration. keywords Episodic Memory, Forgetting, Experience Verbalization, Human-Robot Interaction, Incremental Learning 1 Introduction Episodic Memory (EM) enables humans to remember, organize, and recall past events and experiences, and therefore is a key aspect in human cognition tulving_episodic_1972 . In everyday interactions with others, we use EM to talk about and reflect upon previous experiences. To transfer this capability to artificial agents such as humanoid robots, they need the ability to store and organize their continuous stream of incoming events in an EM and verbalize it appropriately when receiving a user’s request, i.e., perform the task of EM Verbalization (EMV) rosenthal_verbalization_2016 ; dechant_toward_2021 ; barmann_deep_2021 . EM in humans is not unlimited – we manage information by selectively forgetting irrelevant details. While an artificial system could outperform a human’s memory here, an ever-growing EM would similarly bring any realistic system to its storage and compute limits when running for extended periods of time. Therefore, we argue that selective forgetting is a desirable property of an artificial EM to limit storage requirements, compute at question time, and even improve precision by storing only the relevant details. However, there is no generally accepted definition of relevance, as this may greatly differ between use cases and users of a system. Because of this, an EM system needs to incrementally learn what its user considers relevant based on its usage and the provided feedback. Existing systems for EMV of a robot’s experiences are either end-to-end trained neural networks barmann_deep_2021 ; dechant_learning_2023 , or utilize text-based representations of EM in combination with natural language templates rosenthal_verbalization_2016 ; zhu_autonomous_2017 ; katuwandeniya_what_2025 ; plewnia_combining_2025 or LLMs liu_reflect_2023 ; wang_i_2024 ; barmann_episodic_2025 for verbalization. To tackle the problem of scaling to very long sequences of events, previous works explored retrieval-augmented generation liu_reflect_2023 or tree-based representations of EM barmann_episodic_2025 . While these approaches effectively structure the search problem at question time, they still produce an ever-growing EM when deployed in a lifelong setting, and will eventually also suffer from too many memories. Further, none of these systems prioritizes memories based on an incrementally learned, user-defined relevance. Therefore, we introduce H 2 -Emv , an architecture modeling the EM of an autonomous agent, featuring (1) online memory construction, (2) decay-based forgetting with relevance estimation, and (3) incremental learning of a relevance measure from user feedback. Specifically, we extend the hierarchical, tree-based EM representation of H-Emv barmann_episodic_2025 : First, we transfer the offline, “retrospective” EM construction of H-Emv to an online setting by updating the history tree incrementally as new events occur. Secondly, inspired by the literature on forgetting plewnia_forgetting_2024 ; freedman_2011 , we introduce a decay-based forgetting mechanism with relevance estimation. Each node within the history tree is assigned a default lifetime, and the relevance for retention of expired nodes is estimated using an LLM conditioned on a set of natural-language relevance rules. Since there is no universal set of such rules, we finally propose to incrementally learn a relevance measure from user feedback. Specifically, when a user asks a question and the queried information has already been forgotten, they can provide feedback to improve future forgetting and summarization processes. To evaluate our system, we use simulated recordings of household tasks from TEACh padmakumar_teach_2022 as well as real-world recordings from our humanoid robot Armar -7 asfour_armar_2017 . Through a systematic ablation study, we show that each contribution is effective: Online history tree construction shifts the EM construction effort away from question time, thereby reducing QA latency, while also introducing a useful recency bias. Relevance-based forgetting effectively reduces memory size while retaining information relevant for later QA. Finally, feedback-based relevance learning leads to notable performance improvements over time, successfully adapting to the user’s needs from single or only a few feedback instances. We further demonstrate our system deployed on Armar -7 , showcasing its real-world usage while also highlighting remaining challenges for future work. 2 Results We first describe H 2 -Emv , then present simulated and real‑world experiments, and finally discuss a multi‑week deployment on the real-world humanoid robot Armar -7 . Modeling & verbalizing EM with H 2 -EMV (a) System architecture and typical information flow of H 2 -Emv : From the robot’s observations, online history tree construction incrementally groups and summarizes new nodes in context of existing ones. Nodes that reached their expiration date are passed to relevance estimation and may be forgotten. Both online history tree construction and relevance estimation are conditioned on a set of natural language relevance rules learned during dialog: Given a question ( “?” ), the dialog manager invokes EMV to explore the history tree. If relevant details are forgotten, the dialog manager indicates this ( ​) and the user provides feedback ( “!” ), triggering relevance rule learning, thereby improving future summarization and forgetting. (b) The time-aware incremental grouping and summarization algorithm (Alg. ) adjusts the summaries on layer \ell+1 given a new node n_{0}^{\ell} on layer \ell (c) Forgetting inspects the history tree top-down and estimates relevance of expired nodes to extend relevant and drop irrelevant ones Figure 1 : H 2 -Emv models and verbalizes episodic memory of a robot using a tree-like data structure constructed online, managing storage and compute efficiency by forgetting of old items based on a relevance measure learned from user feedback. H 2 -Emv equips an autonomous agent with the capability to build an EM in an online setting, prioritize relevant memories during summarization, forget irrelevant memories to save storage, efficiently answer questions about its past (EMV), and incrementally learn an evolving relevance measure from task and user feedback. Building on previous work barmann_episodic_2025 , from a stream of observations, we derive a history tree , a hierarchical data structure to represent EM, by recursive grouping and summarization. Given a subsequent question, an agentic LLM is used to efficiently explore the history tree to find the answer. Figure 1 provides an overview of our system (details in ). To quantitatively analyze the performance of our system, we evaluate it under two settings: Using simulated agent recordings from TEACh padmakumar_teach_2022 , we randomly concatenate episodes to form long histories and automatically create diverse QA samples asking for past observations and actions. Similarly, we create

Read the complete text