Ella: Embodied Social Agents with Lifelong Memory
Hongxin Zhang, Zheyuan Zhang, Zeyuan Wang, Zunzhe Zhang, Qinhong Zhou, Lixing Fang, Chuang Gan · 2025
Read Ella: Embodied Social Agents with Lifelong Memory by Hongxin Zhang, Zheyuan Zhang, Zeyuan Wang, Zunzhe Zhang, Qinhong Zhou, Lixing Fang, Chuang Gan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-24019.txt.
Introduction
Hongxin Zhang and co-authors introduce Ella, a software character built to live, learn, and socialise in a shared 3D world for days on end, in a 2025 paper. Most AI agents forget everything between tasks. Ella instead keeps a lifelong memory: one store organises facts by the names of people and things, and another keeps a diary of experiences placed in space and time. Foundation models supply the eyes and language.
In tests, fifteen such agents shared a world and went about daily life together. Ella used its memory to plan days, build relationships, and even influence and lead others toward shared goals. The agents were also probed with fresh control tests they had not seen. The work points toward game characters and virtual companions that genuinely remember a shared past.
Catalog record
| CREATOR | Hongxin Zhang, Zheyuan Zhang, Zeyuan Wang, Zunzhe Zhang, Qinhong Zhou, Lixing Fang, Chuang Gan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-24019
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ella: Embodied Social Agents with Lifelong Memory Hongxin Zhang, Zheyuan Zhang, Zeyuan Wang, Zunzhe Zhang, Qinhong Zhou, Lixing Fang, Chuang Gan Abstract We introduce Ella , an embodied social agent capable of lifelong learning within a community in a 3D open world, where agents accumulate experiences and acquire knowledge through everyday visual observations and social interactions. At the core of Ella ’s capabilities is a structured long-term multimodal memory system that stores, updates, and retrieves information effectively. It consists of a name-centric semantic memory for organizing acquired knowledge and a spatiotemporal episodic memory for capturing multimodal experiences. By integrating this lifelong memory system with foundation models, Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. We conduct capability-oriented evaluations in a dynamic 3D open world where 15 agents engage in social activities for days and are assessed with a suite of unseen controlled evaluations. Experimental results show that Ella can influence, lead, and cooperate with other agents well to achieve goals, showcasing its ability to learn effectively through observation and social interaction. Our findings highlight the transformative potential of combining structured memory systems with foundation models for advancing embodied intelligence. More videos can be found at https://umass-embodied-agi.github.io/Ella/ . 1 Introduction It’s a long-standing goal to create intelligent beings capable of survival in the human community ( gan2021threedworld ; li2023behavior ; puig2024habitat ) , which requires lifelong learning in an open and social world. The embodied agents must accumulate experiences, including visual observations and social interactions with other intelligent beings, such as conversations; and acquire knowledge from these multi-modal experiences, build new concepts of objects, agents, and events, and identify the connections among these concepts. With the rapid advancement of Foundation Models ( openai2023gpt4 ; ravi2024sam2segmentimages ; guo2025deepseek ) , a surge of powerful agents has emerged ( sumers2023cognitive ) . These range from agents operating solely in the text domain ( gur2023real ; shinn2024reflexion ) to multi-modal agents capable of controlling screens ( hong2024cogagent ) , playing games ( wang2023voyager ; wang2023describe ) , and even functioning as robots in the physical world ( ahn2022can ; huang2023voxposer ; du2023video ) . Despite these advancements, one crucial component remains underexplored in current agent research: long-term memory. Humans organize accumulated experiences in Episodic Memory ( tulving1972episodic ; tulving1983elements ; nuxoll2007extending ) and acquired knowledge in Semantic Memory ( lindes2016toward ) , enabling them to make long-term plans and exhibit higher-level cognitive capabilities ( laird2022introduction ; tenenbaum2011grow ) . In contrast, current work in embodied agents is limited to constrained spatial regions (primarily indoor spaces) and brief temporal scales (seconds for robotic manipulation or minutes for navigation tasks). For agents to thrive in an ever-evolving world, it is essential to develop a long-term memory system that supports learning new concepts and forming new relationships. In this direction, Generative Agents ( park2023generative ) introduced a textual temporal episodic memory, assuming oracle perception in a sandbox 2D environment. Similarly, Voyager ( wang2023voyager ) designed a single agent with long-term procedural memory, enabling it to acquire new skills in Minecraft through oracle perception and self-training. However, the challenge of constructing effective lifelong memory systems for embodied agents in an open and social world—where they must learn from visual observations and engage in social interactions with other intelligent beings, as illustrated in Figure 1(a) —remains largely unexplored. (a) (b) Figure 1 : (a) Embodied agents require lifelong learning to accumulate experiences and acquire knowledge through everyday visual observation and social interaction within a community in a 3D open world. (b) Ella self-evolves by growing episodic and semantic memory over time. In this work, we propose to build a robust long-term multi-modal memory system that can store, update, and retrieve information effectively. Borrowing the concepts from psychology and cognitive neuroscience ( tulving1972episodic ) , we construct the long-term memory in two forms: a name-centric semantic memory with a hierarchical scene graph and knowledge graph to organize acquired knowledge, and a spatiotemporal episodic memory to capture the agent’s multi-modal experiences. We present Ella , an embodied lifelong learning agent that can accumulate experiences and acquire knowledge effectively through visual perception and social interaction with other agents within a community in an open 3D world, by integrating this structured memory with foundation models. To plan robustly and behave consistently through days of social life, Ella adopts a planning-reaction framework where it first retrieves related context from memory to make a structured daily schedule, then updates the memory with new visual observations and social interactions, and make reactions to the new context, which could be revising the schedule, interacting with the environment, or engaging in social interactions. We simulate Ella and other baseline agents in Virtual Community ( vico ) , an open world simulation platform for multi-agent embodied AI, featuring large-scale community scenarios derived from the real world with realistic physics and renderings. Unlike traditional task-oriented evaluations for agents, assessing high-level cognitive capabilities in a lifelong setting is more critical ( crosby2019animal ) . To this end, we first simulate 15 agents for 9 hours (with a decision frequency of 1 second), representing their first day in the community. During this phase, agents must plan their day based on their unique characteristics and acclimate to the environment and other agents. Then we test the agents with unseen controlled evaluations: Influence Battle and Leadership Quest , where the agents work in groups to persuade others to attend their party at a specific location despite conflicting schedules or lead their group to prepare for an activity under resource constraints. Experimental results across three communities show that Ella demonstrates advanced cognitive abilities including social reasoning and leadership, showcasing its ability to learn effectively through visual observation and social interaction. In sum, our contribution includes: • We propose a structured long-term memory with name-centric semantic memory and spatiotemporal episodic memory to support lifelong learning in an open and social world. • We introduce Ella , the first embodied social agent that can self-evolve through visual observation and social interaction by integrating structured memory with foundation models. • We conduct capability-oriented experiments in a dynamic 3D open world with 15 agents for days, and demonstrate Ella ’s advanced cognitive abilities including social reasoning and leadership. 2 Related Work 2.1 Embodied Social Intelligence Social intelligence has been widely studied in embodied multi-agent environments ( vico ; lowe2017multi ; carroll2019utility ; amato2019modeling ; bard2020hanabi ; jain2020cordial ; puigwatch ; tsoi2020sean ; puig2023habitat ; wen2022multi ; szot2023adaptive ; zhang2023building ; li2019robust ) , while one branch focuses on simplified symbolic or game-like environments ( samvelyan2019starcraft ; suarez2019neural ; jaderberg2019human ; Baker2020Emergent ; niu2021multi ; sharon2015conflict ; yu2024mineland ) , often ignoring the challenges present in an open world, including perception and diverse personalities of agents. Specifically, generative agents ( park2023generative ) developed a unified temporal language memory, demonstrating the robust simulation of human-like agents within a symbolic community. Following this line of research, a series of works have explored socially intelligent agents within text-based sandbox environments li2023camel ; zhou2024sotopia ; liu2024training ; chen2024socialbench ; liu2024exploring ; liu2024interintent ; dai2024artificial . The other branch, including works on human-robot interaction ( gombolay2015decision ; goodrich2008human