MemX: A Local-First Long-Term Memory System for AI Assistants

Lizheng Sun · 2026

Read MemX: A Local-First Long-Term Memory System for AI Assistants by Lizheng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16171.txt.

Introduction

Sun presents MemX, a long-term memory system for AI assistants that runs on the user's own machine. It is built in Rust on libSQL, with an embedding API for search. Its pipeline mixes vector recall, keyword recall, rank fusion, and re-ranking. A low-confidence rule blocks false recalls when no answer is stored. On custom Chinese test suites, MemX reached 91.3% Hit@1. On LongMemEval, fact-level storage doubled session-level scores.

Catalog record

CREATORLizheng Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46229
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16171

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MemX: A Local-First Long-Term Memory System for AI Assistants Lizheng Sun Abstract We present MemX , a local-first long-term memory system for AI assistants with stability-oriented retrieval design. MemX is implemented in Rust on top of libSQL and an OpenAI-compatible embedding API, providing persistent, searchable, and explainable memory for conversational agents. Its retrieval pipeline applies vector recall, keyword recall, Reciprocal Rank Fusion (RRF), four-factor re-ranking, and a low-confidence rejection rule that suppresses spurious recalls when no answer exists in the memory store. We evaluate MemX on two axes. First, two custom Chinese-language benchmark suites (43 queries, ≤ 1,014 {\leq}1{,}014 records) validate pipeline design: Hit@1 = 91.3 % \text{Hit@1}{=}91.3\% on a default scenario and 100 % 100\% under high confusion, with conservative miss-query suppression. Second, the LongMemEval benchmark (500 queries, up to 220,349 records) quantifies system boundaries across four ability types and three storage granularities. At fact-level granularity the system reaches Hit@5 = 51.6 % \text{Hit@5}{=}51.6\% and MRR = 0.380 \text{MRR}{=}0.380 , doubling session-level performance, while temporal and multi-session reasoning remain challenging ( ≤ 43.6 % {\leq}43.6\% Hit@5). FTS5 full-text indexing reduces keyword search latency by 1,100 × 1{,}100{\times} at 100k-record scale, keeping end-to-end search under 90 ms. Unlike Mem0 and related work that targets end-to-end agent benchmarks, MemX focuses on a narrower, reproducible baseline: local-first deployment, structural simplicity, explainable retrieval, and stability-oriented design. Keywords: long-term memory, local AI assistant, hybrid retrieval, reciprocal rank fusion, rejection rule, benchmark 1 Introduction Large language models have achieved strong single-turn comprehension, yet they remain stateless across sessions. Without a persistent memory layer, an AI assistant cannot reliably retain user preferences, project conventions, operational procedures, incident resolutions, or domain-specific constraints. This gap is especially pronounced for local AI assistants, where users expect the system to accumulate knowledge over time, avoid repetitive questioning, and refrain from fabricating answers when no relevant memory exists. The challenge of equipping LLMs with long-term memory has been widely recognized as a key research direction [ 20 ] . While retrieval-augmented generation (RAG) [ 10 ] has become the dominant paradigm for grounding LLM outputs in external knowledge, most RAG systems target document corpora rather than the incremental, personalized memories that accumulate during daily assistant interactions. Recent work has begun to address this gap through diverse architectural choices. Mem0 [ 2 ] frames long-term memory as an integral component of agent infrastructure, covering memory extraction, update, retrieval, and downstream task performance. MemGPT [ 14 ] introduces an operating-system metaphor for managing LLM memory hierarchies. Generative Agents [ 15 ] demonstrate how persistent memory enables believable social simulation. MemLLM [ 13 ] takes a complementary fine-tuning approach, augmenting the model itself with an explicit read–write memory module. A recent survey by 5 highlights the growing diversity of RAG architectures, yet notes that most systems assume access to a pre-existing knowledge base rather than incrementally constructed personal memories. Effective memory retrieval requires combining semantic and lexical signals. Dense vector search, typically organized through approximate nearest-neighbor structures such as HNSW [ 12 ] , captures meaning-level similarity but can miss keyword-specific cues. Sparse lexical methods, from classical BM25 [ 11 ] to lightweight full-text indexes such as SQLite FTS5, complement dense retrieval by matching exact terms. Recent embedding models such as BGE-M3 [ 1 ] natively support both dense and sparse representations, underscoring the importance of hybrid retrieval. Reciprocal Rank Fusion (RRF) [ 3 ] provides a simple, parameter-light method for combining ranked lists from heterogeneous retrievers, making it well-suited to systems that pair vector and full-text indexes. Most existing memory-augmented systems assume cloud-hosted deployment with ample compute and centralized storage, and their evaluations emphasize end-to-end agent task completion rather than retrieval quality in isolation. However, the local-first software paradigm [ 9 ] argues that users should retain full ownership of their data, with the ability to operate offline and without depending on third-party services. For a personal AI assistant, local-first deployment offers concrete advantages: lower latency, stronger privacy guarantees, and independence from external infrastructure changes. Yet a narrower but critical gap remains: under these constraints, how should a memory system balance recall against the risk of spurious results when no relevant memory exists? This gap motivates the following research question: In a single-user, locally-deployed AI assistant scenario, can a structurally simple, explainable memory system achieve stable recall of relevant memories while minimizing spurious recalls on unanswerable queries? We answer this question with MemX, a system built on four design principles (local-first deployment, structural simplicity, real-embedding evaluation, and stability-over-recall) and validate it on two custom benchmark suites with live embedding APIs as well as the external LongMemEval benchmark [ 18 ] (500 queries, up to 220k records). Contributions. 1. A complete implementation of a local long-term memory system featuring hybrid retrieval with explicit access/retrieval tracking separation. 2. A reproducible benchmark framework that directly invokes internal search functions with real embeddings, supporting scenario extension, threshold sweeping, and structured JSON reporting. 3. Empirical evidence on two custom scenarios (43 queries) characterizing the pipeline’s strengths (stable single-topic recall, conservative miss handling) and boundaries (multi-topic coverage gaps, high-confusion false recalls). 4. A storage-granularity study on LongMemEval [ 18 ] (500 queries, up to 220k records) showing that fact-level chunking doubles retrieval quality over session-level storage, and quantifying per-ability-type performance boundaries. 2 Related Work Long-term memory for LLM agents. The need for persistent memory in LLM-based systems has been widely recognized [ 20 ] . Mem0 [ 2 ] proposes a production-oriented memory layer with extraction, update, and retrieval stages, evaluated on end-to-end agent tasks. MemGPT [ 14 ] draws an analogy to virtual memory, allowing an LLM to page information in and out of its context window. MemLLM [ 13 ] fine-tunes models to use an explicit read-write memory. MemX differs from these systems in scope: rather than targeting general agent benchmarks, we focus on a local, single-user deployment with reproducible retrieval evaluation. Retrieval-augmented generation. RAG [ 10 ] established the paradigm of conditioning generation on retrieved passages. Subsequent surveys [ 5 ] document the rapid diversification of RAG architectures. MemX can be viewed as a specialized RAG store whose retrieval pipeline is optimized for stability rather than maximum recall. Vector and hybrid retrieval. FAISS [ 8 ] and HNSW [ 12 ] are foundational approximate nearest-neighbor methods. Hybrid approaches that combine dense and sparse retrieval—fused via Reciprocal Rank Fusion [ 3 ] —have shown consistent benefits in information retrieval [ 11 , 1 ] . MemX adopts RRF as the fusion step and adds a four-factor re-ranking stage tailored to memory retrieval. Graph-enhanced memory. GraphRAG [ 4 ] and related efforts [ 6 ] enrich retrieval with graph structure. Mem0’s later work also explores graph-augmented memory where contradiction, supersession, and causality links participate in retrieval. MemX maintains a link table ( memory_links ) supporting seven relation types, but does not yet integrate multi-hop traversal or graph-aware re-ranking into the search pipeline; we note this as a planned extension (Section 6 ). Local-first software. 9 articulate the principles of local-first software: data ownership, offline capability, and low-latency access. MemX embodies these principles by storing all data in a single libSQL [ 17 ] file on the user’s machine. 3 System Design 3.1 Architecture Overview Figure 1 illustrates the

Read the complete text