HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents

Shuqi Cao, Jingyi He, Fei Tan · 2026

Read HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents by Shuqi Cao, Jingyi He, Fei Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18349.txt.

Introduction

Cao, He, and Tan present HiGMem, a two-level memory system for long-term conversational agents. Pure vector retrieval returns bloated evidence sets, adding similar turns without improving recall. HiGMem lets the model read event summaries first as semantic anchors, then open only the turns worth reading. On the LoCoMo10 benchmark it tops four of five question categories, and adversarial F1 reaches 0.78, up from A-Mem, while retrieving ten times fewer turns.

Catalog record

CREATORShuqi Cao, Jingyi He, Fei Tan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32102
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents Shuqi Cao, Jingyi He, Fei Tan Abstract Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem . 1 Introduction As LLMs evolve into autonomous agents, memory systems have become essential for long-term autonomy 16 ; 13 , transforming historical data into actionable insights to maintain persona consistency and leverage past interactions for contextually-aware responses. Effective memory allows an agent to ground its responses in historical context, transforming raw interaction data into a coherent understanding of its world and relationships. However, existing memory systems, including hierarchical ones, still rely predominantly on vector similarity for retrieval 7 ; 15 . While effective at casting a wide net, this approach tends to produce bloated evidence sets: once the most relevant memories have been recalled, adding more superficially similar fragments yields diminishing gains in recall but steadily erodes retrieval precision, inflates the context fed to the answer stage, and makes the evidence set harder to inspect and manage 8 ; 4 . The root cause is that vector similarity alone cannot judge whether a memory is truly worth reading as it lacks a mechanism for reasoning over different levels of abstraction to actively assess which fine-grained details actually contribute to answering a query. Consequently, developing retrieval strategies that can deliver concise, high-precision evidence sets remains a pivotal research objective. To address these challenges, we introduce HiGMem, a hierarchical and LLM-guided memory system for long-term conversational agents. HiGMem tackles the core limitation of vector-only retrieval through two complementary designs. First, a two-level event-turn architecture organizes memories by abstraction level, from high-level thematic summaries to granular conversational details, providing the structural foundation for multi-granularity reasoning. Second, an LLM-guided Hierarchical Retrieval strategy leverages event summaries as semantic anchors, enabling the LLM to actively predict which fine-grained turns are worth reading rather than passively returning all similar fragments. Empirically, on the LoCoMo10 benchmark 5 , HiGMem achieves the best F1 on four of five question categories while retrieving an order of magnitude fewer turns than the strongest baseline, demonstrating that hierarchical LLM-guided retrieval can provide a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. Our contributions are twofold: (i) We propose a retrieval paradigm for long-term conversational memory that combines hierarchical abstraction with LLM-guided filtering, enabling the system to deliver concise and reliable evidence sets through reasoning while avoiding the prohibitive cost of applying LLM reasoning to every stored memory. (ii) We implement this paradigm as HiGMem, an open-source memory system evaluated on the LoCoMo10 benchmark. HiGMem achieves the best F1 on four of five question categories while retrieving an order of magnitude fewer turns, providing a reproducible baseline for future research on long-term conversational memory. 2 Related Work Long-term conversational agents depend on both strong multi-turn dialogue modeling and effective memory retrieval over long interaction histories. Recent training-time methods such as ReSURE ( 1 ) and daDPO ( 17 ) improve multi-turn conversational ability through more reliable supervision and distribution-aware distillation. Our work focuses on the memory side of this broader problem: retrieval-time evidence localization for long-term conversational agents. Memory-augmented LLM agents such as MemoryBank ( 18 ) , MemGPT ( 7 ) , and A-Mem ( 15 ) extend long-horizon interaction by storing past experiences as external memories and retrieving them at inference time. However, systems that rely solely on vector-similarity retrieval tend to return broad evidence sets once the memory store grows over extended conversations. Once the most relevant evidence has been recalled, adding more semantically similar memories often brings diminishing retrieval precision, inflating context cost, and making evidence harder to inspect and manage. Recent work has explored more structured memory organization. RAPTOR ( 11 ) uses recursive summaries for multi-granularity retrieval, and concurrent work such as H-MEM ( 12 ) introduces hierarchical memory routing. These studies highlight the value of structure, but prior evidence remains limited on whether long-term conversational memory systems can maintain high retrieval recall while passing only a compact set of fine-grained evidence to the answer model. Figure 1: The overall architecture of HiGMem. 3 The HiGMem System HiGMem is designed to address a central trade-off in long-term conversational memory retrieval: the system should recover sufficient evidence for a query, but should do so with a compact and precise evidence set rather than a large pool of semantically similar memories. We therefore pursue three objectives simultaneously: high evidence recall, high retrieval precision, and manageable token cost. To achieve this, HiGMem combines a two-level event-turn memory architecture with LLM-guided retrieval. The event layer provides concise semantic anchors that allow the LLM to inspect memory at low cost, while the turn layer preserves the fine-grained dialogue evidence required for downstream generation. Based on these event summaries, the LLM predicts which related turns are worth reading, producing a smaller and more reliable evidence set than retrieval based solely on vector similarity. For clarity, in the remainder of this section we use Turn node to denote HiGMem’s structured memory unit built from a dialogue turn and its metadata, and Event node to denote the higher-level unit that groups related Turn nodes. 3.1 Hierarchical Memory Architecture HiGMem features a streamlined two-level hierarchical architecture that organizes information across different granularities of abstraction. A critical component of this design is the establishment of robust bidirectional links between layers, ensuring both bidirectional interoperability and data provenance traceability. At the foundation lies the turn layer, which captures raw interaction details at the finest granularity. Here, each Turn node represents a raw dialogue turn enriched with LLM-generated metadata (e.g., keywords, tags, timestamp and context), serving as the primary evidence base. Conversely, the event layer operates at the top level, grouping relevant Turn nodes into coherent narrative units. This layer includes a short summary and a structured fact sheet. The fact sheet orchestrates the timeline and specific Event nodes by maintaining explicit index links to the corresponding Turn nodes in the bottom layer. 3.2 Memory Construction Mechanism HiGMem supports real-time, automated memory updates. When new dialogue turn D_{t} is ingested, the system executes an

Read the complete text