RealTalk : A 21-Day Real-World Dataset for Long-Term Conversation

Dong-Ho Lee, Adyasha Maharana *, Jay Pujara, Xiang Ren, Francesco Barbieri · 2025

Read RealTalk : A 21-Day Real-World Dataset for Long-Term Conversation by Dong-Ho Lee, Adyasha Maharana *, Jay Pujara, Xiang Ren, Francesco Barbieri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13270.txt.

Introduction

Dong-Ho Lee and co-authors release three weeks of real human messaging as a yardstick for chatbots in a 2025 paper. Long-term conversation, remembering last week and staying in character, is where companion AI is weakest. Most research trains on synthetic chats written by models themselves, which flattens exactly the quirks that matter.

RealTalk gathers authentic app dialogues spanning twenty-one days. Comparing them with machine-written chats exposes gaps: real people swing through wider emotional ranges and hold their personas less steadily than scripts assume. Two benchmark tasks follow. In persona simulation, a model must continue a chat as a specific person; in memory probing, it must answer questions needing old context. Models struggled at both, though fine-tuning on a person's own chats improved impersonation. The corpus gives the field real ground to stand on.

Catalog record

CREATORDong-Ho Lee, Adyasha Maharana *, Jay Pujara, Xiang Ren, Francesco Barbieri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13270

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RealTalk : A 21-Day Real-World Dataset for Long-Term Conversation Dong-Ho Lee, Adyasha Maharana *, Jay Pujara, Xiang Ren, Francesco Barbieri Abstract Long-term, open-domain dialogue capabilities are essential for chatbots aiming to recall past interactions and demonstrate emotional intelligence (EI). Yet, most existing research relies on synthetic, LLM-generated data, leaving open questions about real-world conversational patterns. To address this gap, we introduce RealTalk , a 21-day corpus of authentic messaging app dialogues, providing a direct benchmark against genuine human interactions 1 1 1 https://github.com/danny911kr/REALTALK . We first conduct a dataset analysis, focusing on EI attributes and persona consistency to understand the unique challenges posed by real-world dialogues. By comparing with LLM-generated conversations, we highlight key differences, including diverse emotional expressions and variations in persona stability that synthetic dialogues often fail to capture. Building on these insights, we introduce two benchmark tasks: (1) persona simulation where a model continues a conversation on behalf of a specific user given prior dialogue context; and (2) memory probing where a model answers targeted questions requiring long-term memory of past interactions. Our findings reveal that models struggle to simulate a user solely from dialogue history, while fine-tuning on specific user chats improves persona emulation. Additionally, existing models face significant challenges in recalling and leveraging long-term context within real-world conversations. 1 Introduction Figure 1: A Motivation Example. LLM-simulated dialogues often exhibit excessive empathy, even when discussing negative topics, whereas real-world human dialogues demonstrate a broader emotional spectrum, incorporate reflective and grounding language, and progressively develop intimacy over time. Long-term open-domain dialogue capabilities are crucial for developing engaging chatbots that can remember previous interactions and respond with empathy, both of which are key aspects of emotional intelligence (EI) Goleman (1998) ; Mayer et al. (2008) . However, evaluating large language models (LLMs) in this context is challenging. A key obstacle lies in collecting real-world, human-to-human conversations featuring the same individuals over extended periods, ensuring consistent personas—an important factor for passing the Turing test Turing (2009) ; Vinyals and Le (2015) . To address this, recent efforts have turned to LLMs to simulate long-term dialogues between two human participants, resulting in synthetic conversation datasets Kim et al. (2023a) ; Jang et al. (2023a) ; Zhong et al. (2024) ; Du et al. (2024) ; Maharana et al. (2024) . Yet, questions remain about whether these simulated dialogues genuinely capture the nuances of real-world human interaction. In this study, we introduce RealTalk , a real-world, long-term dialogue dataset featuring pairs of individuals who initially connected through messaging apps. Over 21 days, 10 participants each engaged in two separate conversations with different partners, capturing evolving communication patterns. This resulted in a total of 10 unique conversations, each spanning approximately 21 sessions. These daily or near-daily interactions amounted to over 16,000 words per conversation (§ 3 ). The structure of these dialogues parallels the approach used in LoCoMo Maharana et al. (2024) , where LLMs engage in extended conversations, exchanging small talk, and the occasional image. By collecting human dialogues of comparable length, RealTalk enables direct comparisons between real-world and LLM-simulated conversations. Using our data, we analyze two key aspects of real-world long-term dialogues that distinguish them from LLM-simulated conversations. First, we examine emotional intelligence (EI) by comparing authentic human conversations with LLM-generated dialogues using a dialogue-level EI assessment (§ 4 - § 5.2 ). Our findings show that real conversations gradually build intimacy over time Altman (1973) ; Derks et al. (2008) , featuring a wide emotional range, whereas LLM-simulated chats often begin at a high intimacy level, predominantly maintain a positive tone, and display excessive empathy—even amid negative topics (See Figure 1 ). Next, we explore persona consistency , investigating whether individuals maintain a stable persona across multiple conversations and assessing how well models capture these shifts or continuities over time (§ 5.3 ). Our analysis reveals that human conversational styles naturally fluctuate based on context, reflecting variations in emotional intelligence. In contrast, LLMs—despite being prompted with distinct personas—exhibit only minimal differences, struggling to adapt and replicate nuanced persona shifts over extended interactions. Building on these insights, we introduce two benchmark tasks to evaluate model performance in long-term dialogues. (1) Persona simulation : Maintaining a consistent persona is key to fostering trust, engagement, and personalization in AI interactions. This task assesses how well an LLM can simulate an individual’s unique persona by continuing a conversation a dialogue given prior context. We evaluate the model’s ability to reproduce a user’s next message and match their EI attributes (§ 6.1 ); (2) Memory probing : Long-term memory is essential for AI to sustain coherent, context-aware conversations. This benchmark tests whether models can retain and apply long-term context by answering 728 human-annotated memory probing questions linked to the conversation (§ 6.2 ). Our experimental results show that models struggle to simulate a user solely from dialogue history. However, fine-tuning on a specific user’s chat history improves persona emulation. Additionally, existing models still face significant challenges in recalling and effectively leveraging long-term context in real-world conversations. To summarize, we introduce RealTalk , a large-scale, real-world dialogue dataset capturing 21 days of authentic, evolving conversations. Our analysis uncovers key differences in EI and persona consistency, showing that LLMs struggle to replicate human adaptability. To bridge this gap, we introduce two benchmarks—persona simulation and memory probing—to drive the development of more human-like, memory-aware AI. Dataset Dialogue Participants # Turns / \mathcal{C} # Session / \mathcal{C} # Tokens / \mathcal{C} Multimodal Collection MemoryBank Zhong et al. (2024) Human-AI 3.7 10 257.8 ✗ LLM-simulated LongMemEval Wu et al. (2024) Human-AI 9.8 50.2 1,572.3 ✗ LLM-simulated SODA Kim et al. (2023a) Human-Human 7.6 1 122.4 ✗ LLM-simulated Conversation Chronicles Jang et al. (2023b) Human-Human 58.5 5 1,054.7 ✗ LLM-simulated LoCoMo Maharana et al. (2024) Human-Human 588.2 27.2 13,377.2 ✓ LLM-simulated MPChat Ahn et al. (2023) Human-Human 2.8 1 53.3 ✓ Reddit MMDialog Feng et al. (2022) Human-Human 4.6 1 72.5 ✓ Social media Daily Dialog Li et al. (2017) Human-Human 7.9 1 114.7 ✗ Crowdsourcing MSC Xu et al. (2022) Human-Human 53.3 4 1,225.9 ✗ Crowdsourcing RealTalk Human-Human 894.4 21.9 17,109.8 ✓ Crowdsourcing Table 1: Comparison of data statistics across various datasets. Human-AI dialogues primarily address task-oriented interactions where humans aim to achieve specific goals through the dialogue. In contrast, Human-Human dialogues involve conversational exchanges such as chit-chat or other forms of social interaction. Unlike other works that simulate dialogues using models, RealTalk is entirely derived from real-world human interactions. 2 Related Work Long-term dialogues. Recent studies in long-term dialogue aim to improve model coherence and empathy through better memory recall of past interactions (See Table 1 for more details). Early work focused on collecting human-human dialogues from online sources ( e.g., Reddit) or via crowd-sourcing to evaluate model in multi-session, open-domain conversations Xu et al. (2022) ; Feng et al. (2022) ; Ahn et al. (2023) ; Li et al. (2017) , and collecting human-AI dialogues to analyze user interactions and align models with user expectations Zheng et al. (2023) ; Köpf et al. (2024) ; Zhao et al. (2024b) . With LLMs now capable of processing longer contexts, recent studies simulate extensive human-human dialogues Kim et al. (2023a) ; Jang et al. (2023b) ; Maharana et al.

Read the complete text