Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions
Jeongeun Lee, Chanyoung Park, Dongha Lee · 2026
Read Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions by Jeongeun Lee, Chanyoung Park, Dongha Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26256.txt.
Introduction
Jeongeun Lee, Chanyoung Park, and Dongha Lee propose POLAR (2026), a memory-augmented framework that personalizes embodied agents over long user contact. Past interactions enter a multimodal knowledge graph holding semantic memory for personal context and visual ideas, plus episodic memory for past trajectories. At run time, POLAR pulls relevant memories to read the current request and steer execution. Tests across several model backbones and settings show steady gains, largest when tasks demand reasoning across interactions or tracking shifting user context.
Catalog record
| CREATOR | Jeongeun Lee, Chanyoung Park, Dongha Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30914 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions Jeongeun Lee, Chanyoung Park, Dongha Lee Abstract Multimodal large language model (MLLM)-based embodied agents have shown strong potential for solving complex tasks in physical environments. However, personalized assistance requires more than following generic instructions or recognizing object at the category level. In real-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to ground user-intended instances from personalized context accumulated over time. In this work, we propose Polar , a multimodal memory-augmented framework for personalized embodied agents over long-term user interactions. Polar organizes prior interactions into a multimodal knowledge graph with semantic memory for personalized context, and episodic memory for past embodied experiences such as agent trajectories. Therefore Polar retrieves candidate object memories to ground the user-intended target instance and guide subsequent planning. We evaluate Polar across multiple MLLM backbones and diverse evaluation scenarios to study how prior interactions should be represented and used for long-term personalization. Results show that current inputs alone are insufficient for personalized instance grounding, while raw prior interactions are difficult to use directly due to their unstructured form. Polar improves performance by converting prior interactions into task-relevant memory. Figure 1 : Personalization over long-term user interactions. In daily life, users often refer to objects through personal context accumulated over prior interactions rather than explicit target references. When multiple similar objects are present, conventional embodied agents may fail to determine which specific instance the user intends, as they focus on finding the category “shoes” rather than “which” shoes. This motivates personalized instance grounding, where embodied agents use long-term interaction history to resolve user intent. 1 Introduction With great advancements of multimodal large language models (MLLMs) 52 ; 24 ; 5 , MLLM-based embodied agents have been increasingly studied for solving complex tasks while interacting with physical environments 36 ; 17 ; 71 ; 49 ; 63 . By integrating visual perception and multimodal reasoning for vision-driven decision making, these agents are expected to assist human in daily life, including locating 4 ; 45 , navigating to 68 ; 15 ; 43 , and delivering objects 51 ; 27 in response to user requests. While prior works have largely focused on solving generic instructions, however, category-level instance recognition is not enough for personalized assistance. As illustrated in Figure 1 , when a user asks for shoe under specific circumstances and multiple pairs of shoes exist in the house, the agent struggles to determine which pair to bring without personalized context. Since such natural language-only instructions often fail to fully cover user request 30 , some embodied agents learn to perform tasks on specific target instances with instance-level grounding 31 ; 6 ; 50 . Still, these methods generally assume that the target reference is explicitly and directly given. In real-world, however, such personalized context is more often conveyed implicitly through previous interactions, requiring the agent to interpret them to perform personalized instructions accordingly. Figure 2 : Preliminary experiments on the PinNED dataset 6 . Success Rate measures correct target-instance navigation, while Category Match counts cases that reach the correct category object but the wrong instance. 0-turn and 10-turn denote the number of intervening interactions between the target reference and the final target instruction. Since the final instruction contains no reference, the agent must identify the target from prior interactions. We implement the agent in NavBench 42 . Although leveraging long-term context is important for MLLM-based embodied agents, prior works have shown that MLLMs often degrade in long-context settings 26 ; 58 ; 7 . They often fail to reason effectively over accumulated multimodal context and instead rely on superficial cues 58 ; 7 ; 56 . They also struggle to track how information evolves over time, making it difficult to capture updates or resolve conflicts across interactions 19 . To further examine whether this challenge also arises in embodied multi-turn interactions, we provide a preliminary analysis in Figure 2 . The results show that multiple MLLM-based embodied agents often fail to ground the intended target instance from prior interactions, highlighting their difficulty in using accumulated user-specific information for personalized instance grounding. For personalized embodied assistance, the agent should (1) manage information over long-term interactions to build personalized knowledge and (2) use that knowledge to interpret and adapt to individual user requests. This requires more than simply remembering prior interactions, requiring the agent to organize them into task-relevant memory structures that can be selectively retrieved and applied to current instruction following and planning. Therefore, in this work, we propose Polar , a multimodal memory-augmented framework for p ers o na l ized MLLM-based embodied a gent for long-term use r interactions. To this end, Polar leverages multimodal memory to accumulate personalized knowledge from prior interactions and retrieve relevant information for task execution. While many existing memory methods primarily focus on compressing past interactions, the key idea of Polar is to convert them into reusable personalized knowledge that can directly support future embodied tasks. In particular, the memory organizes visual concepts together with personalized context (i.e., semantic memory) and agent trajectories (i.e., episodic memory) 53 . We organize these memories into a multimodal knowledge graph that connects semantic and episodic memories across objects and interactions. We study how Polar structures prior interactions for long-term personalized embodied agents across multiple MLLM backbones and evaluation scenarios. The results show that current observations alone are insufficient for personalized instance grounding. Raw prior interactions can contain useful user-specific information, but their unstructured form makes them unstable to use directly for the current task. Polar improves performance by converting prior interactions into task-relevant memory. Further analyses show that semantic memory provides finer-grained retrieval of user-specific information, while episodic memory converts past trajectories into planning-relevant experience. These findings suggest that long-term personalization requires memory representations that support both personalized instance grounding and subsequent embodied planning. 2 Related Works Vision-driven Embodied Agents. Early works on embodied agents have primarily leveraged large language models (LLMs) with its high reasoning and planning capabilities 47 ; 21 ; 23 ; 41 . They show that LLMs can decompose complex instructions 1 ; 9 ; 69 , reason over environment feedback 48 ; 8 ; 29 , and compose robot policies 29 ; 32 ; 12 . However, these approaches typically relied on textual scene descriptions 69 ; 64 ; 65 , external perception modules 22 , or predefined skill libraries 1 ; 14 , limiting their ability to interpret subtle visual cue in raw observations. To directly perceive fine-grained visual information, recent research has increasingly explored MLLM-based embodied agents, which integrate visual perception and language reasoning for vision-driven decision making 49 ; 42 ; 59 . These agents have demonstrated promising performance across a wide range of embodied tasks, including robot manipulation 17 ; 63 , household tasks 49 ; 45 ; 55 , and embodied navigation 43 ; 42 ; 57 in simulated and real-world environments 59 ; 67 . Nevertheless, as using MLLMs as step-by-step controllers over raw observations can be computationally inefficient for embodied control, particularly in long-horizon tasks 62 . Recent studies therefore leverage MLLMs as high-level planners, using them to predict abstract action plans, such as future waypoints