On Memory Construction and Retrieval for Personalized Conversational Agents
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Jianfeng Gao · 2025
Read On Memory Construction and Retrieval for Personalized Conversational Agents by Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Jianfeng Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05589.txt.
Introduction
Zhuoshi Pan and co-authors improved how chat agents remember long conversations in 2025. Personal agents must recollect past talks to stay coherent. Banks built at the turn level, session level, or by summary each lose accuracy or meaning when searched.
Compressing prompts, they found, acts as a denoiser and lifts retrieval accuracy at every granularity. Their method, SeCom, cuts talks into topic-coherent segments with a segmentation model, then cleans each memory unit by compression. On two long-conversation benchmarks it beat rival methods by a wide margin. The segmentation model also topped three dialogue segmentation data sets. The work treats memory shape, not just memory size, as the key design choice.
Catalog record
| CREATOR | Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Jianfeng Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05589
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Memory Construction and Retrieval for Personalized Conversational Agents Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Xufang Luo, Hao Cheng, Dongsheng Li, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Jianfeng Gao Abstract To deliver coherent and personalized experiences in long-term conversations, existing approaches typically perform retrieval augmented response generation by constructing memory banks from conversation history at either the turn-level, session-level, or through summarization techniques. In this paper, we present two key findings: (1) The granularity of memory unit matters: Turn-level, session-level, and summarization-based methods each exhibit limitations in both memory retrieval accuracy and the semantic quality of the retrieved content. (2) Prompt compression methods, such as LLMLingua-2 , can effectively serve as a denoising mechanism, enhancing memory retrieval accuracy across different granularities. Building on these insights, we propose SeCom , a method that constructs the memory bank at segment level by introducing a conversation Se gmentation model that partitions long-term conversations into topically coherent segments, while applying Com pression based denoising on memory units to enhance memory retrieval. Experimental results show that SeCom exhibits a significant performance advantage over baselines on long-term conversation benchmarks LOCOMO and Long-MT-Bench+ . Additionally, the proposed conversation segmentation method demonstrates superior performance on dialogue segmentation datasets such as DialSeg711 , TIAGE , and SuperDialSeg . 4 4 footnotetext: Project page: https://aka.ms/secom 1 Introduction Large language models (LLMs) have developed rapidly in recent years and have been widely used in conversational agents. In contrast to traditional dialogue systems, which typically focus on short conversations within specific domains ( Dinan et al., 2019 ) , LLM-powered conversational agents engage in significantly more interaction turns across a broader range of topics in open-domain conversations ( Kim et al., 2023 ; Zhou et al., 2023 ) . Such long-term, open-domain conversations over multiple sessions present significant challenges, as they require the system to retain past events and user preferences to deliver coherent and personalized responses ( Chen et al., 2024 ) . Some methods maintain context by concatenating all historical utterances or their summarized versions ( LangChain Team, 2023a ; Wang et al., 2023 ) . However, these strategies can result in excessively long contexts that include irrelevant information, which may not be relevant to the user’s current request. As noted by Maharana et al. (2024) , LLMs struggle with understanding lengthy conversations and grasping long-range temporal and causal dynamics, particularly when the dialogues contain irrelevant information ( Jiang et al., 2023c ) . Some other works focus on retrieving query-related conversation history to enhance response generation ( Yuan et al., 2023 ; Alonso et al., 2024 ; Kim et al., 2024 ; Maharana et al., 2024 ) . These approaches typically construct a memory bank from the conversation history at either the turn-level ( Yuan et al., 2023 ) or session-level ( Wang et al., 2023 ) . Chen et al. (2024) , Li et al. (2024) and Zhong et al. (2024) further leverage summarization techniques to build memory units, which are then retrieved as context for response generation. Building on these works, a key question arises: Which level of memory granularity—turn-level, session-level, or their summarized forms—yields the highest effectiveness? Moreover, is there a novel memory structure that could outperform these three formats? In this paper, we first systematically investigate the impact of different memory granularities on conversational agents within the paradigm of retrieval augmented response generation ( Lewis et al., 2020 ; Ye et al., 2024 ) . Our findings indicate that turn-level, session-level, and summarization-based methods all exhibit limitations in terms of the accuracy of the retrieval module as well as the semantics of the retrieved content, which ultimately lead to sub-optimal responses, as depicted in Figure 1 , Figure 2 , and Table 1 . Figure 1: Illustration of retrieval augmented response generation with different memory granularities. Turn-level memory is too fine-grained, leading to fragmentary and incomplete context. Session-level memory is too coarse-grained, containing too much irrelevant information. Summary based methods suffer from information loss that occurs during summarization. Ours (segment-level memory) can better capture topically coherent units in long conversations, striking a balance between including more relevant, coherent information while excluding irrelevant content. Bullseye \odot indicates the retrieved memory units at turn level or segment level under the same context budget. [0.xx]: similarity between target query and history content. Turn-level retrieval errors: false negative , false positive . (a) Response quality as a function of chunk size, given a total budget of 50 turns to retrieve as context. (b) Retrieval DCG obtained with different memory granularities using BM25 based retriever. (c) Retrieval DCG obtained with different memory granularities using MPNet based retriever. Figure 2: The impact of memory granularity on the response quality (a) and retrieval accuracy (b, c). Specifically, users often interact with agents over multiple turns to achieve their goals, causing relevant information to be dispersed across multiple interactions. This dispersion can pose a great challenge to the retrieval of turn-level memory units as some of the history conversation turns may not explicitly contain or relate to keywords mentioned in the current request ( e.g. , Turn-5 in Figure 1 ). As a result, the retrieved contexts ( e.g. , Turn-3 and Turn-6 in Figure 1 ) can be fragmentary and fail to encompass the complete request-related information flow, leading to responses that may lack coherence or omit essential information. On the other hand, a single conversation session may cover multiple topics, especially when users do not initiate a new chat session upon switching topics. Therefore, constructing memory units at the session level risks including irrelevant content ( e.g. , definition of the prosecutor’s fallacy and reasons of World War II in Figure 1 ). Such extraneous content in the session-level memory unit may not only distract the retrieval module but also disrupt the language model’s comprehension of the context, causing the agent to produce responses that are off-topic or include unnecessary details. Long conversations are naturally composed of coherent discourse units. To capture this structure, we introduce a conversation segmentation model that partitions long-term conversations into topically coherent segments, constructing the memory bank at the segment level. During response generation, we directly concatenate the retrieved segment-level memory units as the context as in Yuan et al. (2023) ; Kim et al. (2024) , bypassing summarization to avoid the information loss that often occurs when converting dialogues into summaries ( Maharana et al., 2024 ) . Furthermore, inspired by the notion that natural language tends to be inherently redundant ( Shannon, 1951 ; Jiang et al., 2023b ; Pan et al., 2024 ) , we hypothesize that such redundancy can act as noise for retrieval systems, complicating the extraction of key information ( Grangier et al., 2003 ; Ma et al., 2021 ) . Therefore, we propose removing such redundancy from memory units prior to retrieval by leveraging prompt compression methods such as LLMLingua-2 ( Pan et al., 2024 ) . Figure 3 shows the results obtained with a BM25 based retriever and an MPNet based retriever ( Song et al., 2020 ) on Long-MT-Bench+ . As demonstrated in Figure 3a and Figure 3b , LLMLingua-2 consistently improves retrieval recall given different retrieval budgets K ( i.e. , the number