Human-Inspired Context-Selective Multimodal Memory for Social Robots Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1080
Hangyeol Kang, Slava Voloshynovskiy, Nadia Magnenat Thalmann · 2026
Read Human-Inspired Context-Selective Multimodal Memory for Social Robots Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1080 by Hangyeol Kang, Slava Voloshynovskiy, Nadia Magnenat Thalmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12081.txt.
Introduction
Kang, Voloshynovskiy, and Thalmann propose a context-aware memory architecture for social robots, holding textual and visual episodic traces, picking salient or novel scenes. Most robots rely on plain text-based memory, curbing tailored contact. Linking moments with single users enables social recall and more natural, grounded talk. On curated social scenes, this storage reaches a Spearman correlation of 0.506, above human level at 0.415 and beating older image memory models. In tests, fusion lifts Recall@1 by up to 13 percent over single-mode text or image retrieval, with real-time speed.
Catalog record
| CREATOR | Hangyeol Kang, Slava Voloshynovskiy, Nadia Magnenat Thalmann |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42107 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12081
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-Inspired Context-Selective Multimodal Memory for Social Robots Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 1080 Hangyeol Kang, Slava Voloshynovskiy, Nadia Magnenat Thalmann Abstract . Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the context. However, most current social robots and embodied agents rely on non-selective, text-based memory, limiting their ability to support personalized, context-aware interactions. Drawing inspiration from cognitive neuroscience, we propose a context-selective, multimodal memory architecture for social robots that captures and retrieves both textual and visual episodic traces, prioritizing moments characterized by high emotional salience or scene novelty. By associating these memories with individual users, our system enables socially personalized recall and more natural, grounded dialogue. We evaluate the selective storage mechanism using a curated dataset of social scenarios, achieving a Spearman correlation of 0.506, surpassing human consistency ( ρ = 0.415 \rho=0.415 ) and outperforming existing image memorability models. In multimodal retrieval experiments, our fusion approach improves Recall@1 by up to 13% over unimodal text or image retrieval. Runtime evaluations confirm that the system maintains real-time performance. Qualitative analyses further demonstrate that the proposed framework produces richer and more socially relevant responses than baseline models. This work advances memory design for social robots by bridging human-inspired selectivity and multimodal retrieval to enhance long-term, personalized human-robot interaction. \doi JQIR9876 C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Memory plays a central role in human cognition and social interaction. The ability to selectively store and recall past experiences enables people to sustain meaningful relationships, learn from previous encounters, and adapt flexibly to changing contexts ( 50 ) . For artificial agents, particularly embodied robots and assistive AI systems, equipping them with memory capabilities is essential for fostering natural, effective, and personalized interactions with humans ( 27 ) . In real-world environments, agents operate within a continuous stream of multimodal sensory information while engaging in complex social exchanges. To act as credible social partners, robots and AI need to remember not only factual events but also how and when those were meaningful to specific individuals. This requires memory systems that go beyond non-selective or text-based storage, enabling multimodal retention and retrieval of contextually meaningful experiences. For embodied agents such as social robots, contextually selective memory grounded in both perceptual and social cues is critical for developing user-aligned behavior. Despite recent advances in conversational AI and embodied agents, most existing systems still rely on non-selective, text-based memory ( 49 ) . Although some frameworks have introduced visual memory modules, they often remain object-centric, interval-based, or non-selective—storing scene snapshots or generic visual information at fixed intervals regardless of contextual importance ( 34 ; 17 ) . Many also convert visual memories into textual descriptions, constraining the robot’s ability to answer detailed or flexible queries when relevant cues were never explicitly transcribed ( 33 ) . This lack of contextual selectivity limits the depth of social interaction that such agents can achieve, as object- or scene-centric memory alone cannot capture what makes an experience personally or socially meaningful. Few systems prioritize what is memorable from a human perspective, neglecting factors such as emotional salience, novelty, and user-centered relevance ( 43 ) . As a result, they often miss opportunities to support richer, more personalized, and emotionally resonant interactions with humans. Insights from cognitive neuroscience demonstrate that human memory is inherently selective: people do not store every moment of experience, but instead encode memories according to intrinsic and extrinsic cues ( 53 ) . Intrinsic cues, such as the perceptual complexity or distinctiveness of a scene, and extrinsic factors, such as emotional intensity, novelty, or personal relevance, are known to strongly influence which events are remembered and which are forgotten ( 28 ; 26 ; 4 ) . This selectivity allows people to prioritize significant, emotionally resonant, or socially important experiences, an ability that is essential for adaptive social functioning and efficient use of cognitive resources ( 53 ) . For social robots and embodied AI, adopting a similar selective approach is essential. Real-world environments are dynamic and information-rich, and storing every sensory input or conversational exchange is neither scalable nor meaningful. Memory systems for social agents, therefore, need to integrate both selectivity and multimodality, capturing not only textual or visual information but also when an experience holds emotional or contextual significance for the user. Without such mechanisms, artificial agents risk being overwhelmed by irrelevant details and failing to form genuine, user-aligned social connections. Consequently, there is a clear need for frameworks that integrate selective and multimodal memory storage and retrieval inspired by how humans prioritize, store, and recall their most memorable experiences. In this work, we introduce SUMMER (Selectivity Unified Multimodal Memory for Embodied Robots), an end-to-end framework that equips social robots with context-selective and multimodal memory capabilities. Grounded in principles of human memory selectivity, the framework stores and retrieves information that is not only factual but also socially and emotionally meaningful. SUMMER operates without additional model training or fine-tuning, enabling efficient integration into diverse robotic platforms. Its lightweight and modular design ensures practical deployment in real-world social and embodied agents. To systematically evaluate the framework, we conduct four complementary studies. First, we curate a pilot dataset of social interaction scenarios to analyze how emotional salience, novelty, and scene complexity influence selective memory storage. Second, we benchmark multimodal retrieval on public datasets (Flickr8k ( 13 ) , Flickr30k ( 57 ) , and MS COCO ( 32 ) ), comparing fusion-based retrieval against unimodal text and image baselines. Third, we assess runtime performance using the pilot dataset to verify the framework’s suitability for real-time social interaction. Finally, we perform qualitative evaluations on the same dataset, comparing responses generated by the proposed framework with those of a baseline vision–language model to illustrate its ability to produce richer and more socially grounded outputs. Together, these evaluations demonstrate the framework’s capacity to capture socially meaningful experiences, retrieve contextually relevant information, and operate efficiently in dynamic, human-centered environments. Our main technical contributions are as follows: • We present an end-to-end framework for multimodal memory storage and retrieval in social robots, integrating selective capture guided by contextual cues. • We design a train-free multimodal retrieval mechanism that enables robots to respond to broader, human-like memory queries by leveraging both textual and visual modalities. • We conduct comprehensive quantitative and qualitative evaluations using both public benchmarks and a new human-annotated pilot dataset focused on socially relevant, user-centered memories. 2. Related Work 2.1. Memory Systems in Social Robots Memory enables social robots to sustain coherent interactions, personalize behavior, and adapt to human partners over time. Early approaches focused on text-based conversational memory, using context windows or prompt-based retrieval to maintain dialogue coherence in large language models (LLMs) ( 38 ) . Frameworks such as MemoryBank ( 60 ) and Memory Sandbox ( 18 ) extend this concept with long-term conversational traces, autonomous recall, and user-level inspection. However, text-centric memories often lose perceptual richness when multimodal inputs are reduced to language ( 20 ) . To enable more structured reasoning, symbolic or knowledge-based systems explicitly represent events and facts using graphs