AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue

Mohammad Al-Ratrout, Pavan Ravva, Shayla Sharmin, Aditya Raikwar, Ju Young Shin, and Leila Barmaki · 2026

Read AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue by Mohammad Al-Ratrout, Pavan Ravva, Shayla Sharmin, Aditya Raikwar, Ju Young Shin, and Leila Barmaki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25022.txt.

Introduction

Al-Ratrout and five colleagues present the Adaptive Friend Agent, an identity-aware memory system for shared voice assistants. When several people share one assistant, histories get mixed and preferences can leak across responses. They call this failure persona confusion. Their system pairs voice-based speaker ID with per-user memory stores. They also build PAT, a synthetic dataset of 58,289 chat turns across 133 profiles and 12 cases. Identity-aware routing lifts Persona Attribution Accuracy from about 36 percent to 61.3 percent. Human tests confirm higher felt personalization in routed responses.

Catalog record

CREATORMohammad Al-Ratrout, Pavan Ravva, Shayla Sharmin, Aditya Raikwar, Ju Young Shin, and Leila Barmaki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39854
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue Mohammad Al-Ratrout, Pavan Ravva, Shayla Sharmin, Aditya Raikwar, Ju Young Shin, and Leila Barmaki Abstract . When multiple people share a single voice assistant, the system conflates their histories: one resident’s preferences can leak into another’s responses, eroding utility and trust. We call this failure mode persona confusion , and we show it is a measurable problem in today’s single-user dialogue systems when deployed in shared environments. We present the Adaptive Friend Agent (AFA), a modular framework that combines voice-based speaker identification with per-user memory stores to enable identity-aware, personalized dialogue across multiple users. To support training and evaluation, we construct PAT (Personalized Agent chaT), a synthetic dataset of 58,289 persona-grounded dialogue turns spanning 133 user profiles and 12 real-world scenarios. We evaluate AFA across five LLM back-ends in a standard response-quality benchmark, with a LLaMA-2-70B model fine-tuned on PAT achieving the highest overall performance. To directly measure persona confusion prevention, we introduce an interleaved multi-user evaluation protocol with a novel metric, Persona Attribution Accuracy (PAA), demonstrating that identity-aware routing improves PAA from 35.7% to 61.3%. Human evaluation confirms annotators perceive significantly higher personalization in routing-enabled responses. Our results establish that identity-aware user routing is the critical component for preventing persona confusion in multi-user conversational systems. Link to our code. Figure 1. AFA system overview. The Adaptive Friend Agent (AFA) processes spoken audio from multiple residents sharing a single device. Voice embeddings identify who is speaking (Speaker ID), while ASR transcribes what was said. The resolved User ID routes the query to that user’s isolated memory store, containing their persona attributes and conversation history, which is injected into the LLM prompt to generate a personalized response. Takeaway: Identity-aware routing is the critical component for preventing persona confusion, improving Persona Attribution Accuracy (PAA) from 35.7% to 61.3% over a no-routing baseline. 1. Introduction When two residents share a voice assistant in a communal living environment, the system can confuse them: the grandmother asking about her medication may receive a response shaped by her grandson’s earlier gaming questions. We call this failure mode persona confusion , and it is not merely a technical error; it is a significant reliability risk. If a shared assistant attributes one resident’s social schedule, preferences, or history to another, the system’s utility and the user’s trust collapse. This risk is amplified in shared residential spaces such as smart homes, communal living facilities, and shared workspaces, where multiple people routinely interact with the same device. However, existing dialogue agents often assume a single generic user or forget past interactions. Even state-of-the-art chatbots can produce inconsistent answers when users refer back to information from previous sessions ( 19 ) . These inconsistencies underscore the need for explicit long-term memory and user modeling. Prior work on personalized dialogue, such as PersonaChat ( 20 ) or multi-session chat ( 19 ) , typically focuses on single-user personalization or fixed persona profiles, creating a gap in handling dynamic multi-user settings. Our proposed Adaptive Friend Agent (AFA) combines speaker identification with a memory-augmented large language model (LLM), allowing the assistant to retrieve and update each user’s personalized memory over time. This integration enables AFA to generate coherent, context-aware, and persona-consistent responses that reflect the evolving preferences of multiple residents across sessions. Our contributions are as follows: (1) We develop AFA, a conversational framework that integrates speaker identification, personalized memory retrieval, and LLM-based response generation to support multiple independent users within shared environments. (2) We construct PAT (Personalized Agent chaT), a large-scale synthetic dataset of 58,289 persona-grounded dialogue turns spanning 133 user profiles and 12 real-world scenarios. (3) We introduce a novel interleaved multi-user evaluation protocol and the Persona Attribution Accuracy (PAA) metric, which directly measures persona confusion prevention while controlling for topic bias, addressing a gap in existing personalization evaluations that rely on surface-level overlap metrics. Using this protocol alongside human evaluation, we demonstrate that identity-aware routing improves PAA from 35.7% to 61.3%, establishing routing as the critical component for multi-user personalization. 2. Related Work 2.1. Multi-User Personalization Recent work has highlighted the importance of long-term memory and personalization in dialogue systems. Beyond Goldfish Memory ( 19 ) introduced the Multi-Session Chat (MSC) dataset and showed that standard dialogue models struggle with long contexts spanning multiple sessions. They found that retrieval-augmented and summarization-based models outperform baselines on such data, indicating the importance of explicit memory. While MSC advanced long-term dialogue research, it is limited to repeated sessions between the same pair of speakers and does not address scenarios where multiple distinct users share a single system. This leaves persona confusion, the misattribution of one user’s information to another, unexplored. Our work addresses this gap by targeting multi-user personalization with identity-aware routing. Multi-user AI personalization proposed a multi-agent system workflow to reconcile multiple users’ preferences in group settings ( 7 ) . We adopt a complementary approach, focusing on a single agent that adapts to each user individually via speaker recognition and per-user memory, rather than coordinating among multiple agents. Moreover, MSC comprises human-human conversational data, while our PAT dataset captures human-AI interactions, which more closely reflects the deployment setting of a conversational assistant. Concurrent work by Fu et al. ( 4 ) introduces a persona-aware speech assistant for multi-party dialogue (PAChat) that unifies speech and speaker encoding within a fine-tuned LLM architecture. Our work differs in scope and methodology: (1) their system addresses identity-aware response generation within individual conversations, whereas AFA explicitly targets persistent per-user memory across multiple sessions with accumulating conversation history and profile evolution; (2) while their dataset contains multi-party group conversations in single sessions, our evaluation protocol explicitly tests multi-user robustness through interleaved sequences where distinct users alternate turns with the system, directly measuring persona confusion across user switches; and (3) we introduce the Persona Attribution Accuracy (PAA) metric, which directly measures persona confusion prevention by controlling for topic bias, complementing the standard NLG metrics used in prior evaluations. 2.2. Memory-Augmented Conversational Models There is growing interest in equipping LLM-based chatbots with external memory modules to handle long-term context. Notable examples include MemoryBank ( 22 ) , which stores and updates key memories of past dialogues, and MemoChat ( 12 ) , which refines LLM instructions to use self-composed “memos” for consistent long-range conversation. These methods demonstrate that explicit memory structures can greatly improve response consistency and persona retention over long dialogues. AFA’s personalized memory operates on this principle but is specialized for individual users: we maintain a separate memory store for each user and retrieve relevant context when they speak, allowing the assistant to adapt its responses based on that user’s history rather than treating all interactions generically. 2.3. Persona and Profile-Grounded Dialogue The role of persona in dialogue has been extensively explored since the introduction of PersonaChat ( 20 ) , which demonstrated that conditioning a chatbot on a fixed persona leads to more engaging and coherent interactions. Following this, many datasets such as FoCus ( 8 ) , MPChat ( 1 ) , and PEC ( 21 ) have investigated persona-based conversations in varying domains. However, most persona-chat research assumes a static persona profile known in advance and focuses exclusively on single-user settings. Our work similarly leverages

Read the complete text