Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue
Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu · 2026
Read Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue by Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01223.txt.
Introduction
Lin and co-authors build a test for reflective memory in dialogue. Long chats ask models to join scattered cues over time. Past tests check recall of plain facts only. RefMem-Bench holds 26,000 graded question pairs. It spans eight reflective skills and three task forms. Models must infer latent sense from cues spread through a chat. Surface search alone will not pass it.
The team also gives REMIND, a staged aid. It finds proof tied to the question first. Then it grounds key cues and adds level-based training. A staged alignment step moves that skill into fact-based paths. Tests show the bench is hard for current models. REMIND lifts both answer scores and memory recall.
Catalog record
| CREATOR | Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu Abstract Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations. To address this gap, we introduce RefMem-Bench , a benchmark for reflective memory in long-horizon dialogue. RefMem-Bench contains 26K annotated QA instances with eight reflective-memory dimensions and three task formats, requiring models to move beyond surface-level retrieval and infer latent meanings from evidence distributed across interaction histories. To enhance reflective memory capability, we propose RE flective M emory IND uction ( REMIND ), a hierarchical framework that treats reflective memory as progressive meaning construction. REMIND couples question-conditioned evidence retrieval, salience-aware grounding, and abstraction-level supervision, and uses Progressive Reflective Alignment to distill high-level reflective reasoning into the factual inference pathway. Experiments show RefMem-Bench poses a substantial challenge to current models, while REMIND consistently improves both answer accuracy and memory recall through progressive evidence perception, grounding, and abstraction. 1 Introduction As Large Language Models (LLMs) are applied to increasingly long and multimodal contexts, memory has emerged as a key factor in their practical performance ( 23 ; 49 ) . A strong model must do more than process the immediate input by instead incrementally accumulating user experiences, tracking evolving states, and capturing recurring behavioral patterns across extended turns and sessions ( 34 ) . Recent advances in retrieval-augmented generation, memory-augmented modeling, and long-context training have significantly improved the ability of LLMs to access vast interaction histories ( 33 ; 6 ) . These developments move LLMs closer to becoming persistent companions that can leverage a wealth of prior experiences to produce responses that are both personalized and contextually grounded. However, existing evaluation paradigms are still largely centered on explicit memory recall, measuring whether a model can retrieve information that has been directly stated in the interaction history. Benchmarks such as LongBench ( 4 ) , LoCoMo ( 29 ) , and LongMemEval ( 39 ) are representative examples of this setting. As illustrated in Figure 1 (a), these tasks typically require models to identify surface-level details from prior context, such as a specific name or a visually observable object. While such abilities are important, they capture only a limited notion of memory. From a schema-based view of memory ( 2 ) , effective remembering involves not only retaining details, but also synthesizing distributed signals into structured, higher-level representations. In long-horizon dialogue, this means that models must go beyond explicit retrieval and integrate scattered historical cues to infer latent states, behavioral patterns, or personal traits , as illustrated in Figure 1 (b). Yet current benchmarks provide little systematic evaluation of this higher-level synthesis, creating the need for a dedicated benchmark for reflective memory that is essential to building truly intelligent long-term agents. To bridge this gap, we introduce RefMem-Bench , a benchmark for reflective memory in long-horizon dialogue. RefMem-Bench contains 26K annotated question-answer instances covering 1K sessions and 71K dialogue turns. Unlike benchmarks centered on explicit recall, it spans eight reflective-memory dimensions and three task formats, requiring models to integrate distributed evidence across extended interaction histories to infer latent user states, behavioral regularities, temporal evolution, and implicit constraints. All instances are evidence-anchored and human-verified to support reliable evaluation of memory reasoning. To improve performance on this capability, we further propose REMIND ( RE flective M emory IND uction), a hierarchical framework that treats reflective memory as meaning construction rather than evidence retrieval alone. REMIND organizes reasoning through a three-level Cognitive Pyramid , consisting of factual, attentional, and reflective states, and is trained with Progressive Reflective Alignment to distill high-level reflective reasoning into the factual inference pathway, enabling efficient test-time inference without explicit multi-stage reasoning. To summarize, our work provides three key contributions: 1) We formalize reflective memory as a distinct target for long-term LLM evaluation, introduce RefMem-Bench as a large-scale multimodal benchmark with evidence-anchored and human-verified annotation; 2) we propose REMIND as a unified framework that integrates factual retrieval, salience grounding, and abstraction induction in a hierarchical training objective; and 3) extensive experiments show that RefMem-Bench is highly challenging for current models, while REMIND consistently improves both answer accuracy and memory recall across task formats and backbone settings. Figure 1: Example of factual memory and reflective memory. 2 Related Work Long-Term Dialogue Benchmarks. Recent long-term dialogue benchmarks have substantially advanced the evaluation of memory in sustained interactions, yet most remain centered on explicit recall. Representative examples include LoCoMo ( 29 ) and LongMemEval ( 39 ) , which evaluate memory through long-horizon question answering, summarization, temporal reasoning, and knowledge updates. REALTALK ( 21 ) , PersonaMem ( 15 ) , and LifeBench ( 7 ) move toward more realistic long-span interactions and evolving user profiles, while MADial ( 13 ) , MMRC ( 42 ) , and MemGallery ( 5 ) extend evaluation to memory-aware generation and multimodal conversation. In contrast, our work focuses on reflective memory: whether models can integrate temporally scattered and cross-modal cues into coherent latent interpretations. Long-Term Memory Methods. Existing long-term memory methods for LLMs mainly improve interaction over long horizons by introducing external memory structures, latent memory mechanisms, or adaptive retrieval pipelines. Early systems such as MemoryBank ( 49 ) and MemGPT ( 30 ) show that explicit memory storage can help preserve salient user information beyond the immediate context window, while model-level approaches such as LongMem ( 34 ) and MemoryLLM ( 35 ) explore cached context and latent memory pools for scalable long-range retention. More recent systems, including Mem0 ( 8 ) , MemoryOS ( 17 ) , A-Mem ( 41 ) , and M+ ( 36 ) , further improve scalability, structured storage, retrieval quality, and adaptive memory management. In contrast to the above memory systems, we treat long-term memory as hierarchical meaning construction, emphasizing how models transform retrieved traces into abstraction-level reasoning. Figure 2: Overview of RefMem-Bench. (a) Workflow of dataset construction. (b) Comparison between RefMem-Bench and representative dialogue datasets and memory benchmarks. Statistics marked with ∗ are derived from values reported in the original papers. 3 Benchmark Dataset Curation. RefMem-Bench is built through a semi-automatic pipeline over long-horizon conversations collected from REALTALK ( 21 ) , DialSim ( 18 ) , and LoCoMo ( 29 ) , with complementary text-centric and image-centric branches (Figure 2 ). In the text branch, we segment multi-turn dialogues into fine-grained topical units, cluster them with Agglomerative Clustering ( 16 ) , and refine large clusters via K-Means ( 28 ) , producing questions for Temporal Dynamics , Pattern Induction , Belief Modeling , Experience-Grounded Planning , and Implicit Boundary Modeling . In the image branch, we collect speaker-level image histories from REALTALK, encode them with CLIP ( 31 ) , and cluster them to construct questions for Cross-Modal Latent Inference , Personalized Visual Baselines , and Topic-Scoped Visual Prototypes . All generated instances are reviewed by three expert annotators to verify answers and evidence, correct LLM-induced errors, and filter out ambiguous or low-quality questions, with inconsistent cases resolved by an adjudicator following prior annotation protocols ( 39 ) . More details are