NextMem: Towards Latent Factual Memory for LLM-based Agents
Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua · 2026
Read NextMem: Towards Latent Factual Memory for LLM-based Agents by Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15634.txt.
Introduction
Zhang and co-authors present NextMem, a latent memory system for language model agents. Text memory loads the context and costs a lot to index. Trained-in memory can forget old facts and costs a lot to update. NextMem uses an autoregressive autoencoder to build compact latent memory. Training runs in two stages, with alignment first and gradual substitution next. Quantization cuts storage needs. The authors report strong gains in retrieval, robustness, and extensibility.
Catalog record
| CREATOR | Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64871 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15634
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NextMem: Towards Latent Factual Memory for LLM-based Agents Zeyu Zhang, Rui Li, Xiaoyan Zhao, Yang Zhang, Wenjie Wang, Xu Chen, Tat-Seng Chua Abstract Memory is critical for LLM-based agents to preserve past observations for future decision-making, where factual memory serves as its foundational part. However, existing approaches to constructing factual memory face several limitations. Textual methods impose heavy context and indexing burdens, while parametric methods suffer from catastrophic forgetting and high costs. To address these challenges, we introduce NextMem, a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory while ensuring accurate reconstruction. For better optimization, we propose a two-stage training process, including autoregressive reconstruction alignment and progressive latent substitution. We also incorporate quantization to reduce storage overhead. Extensive experiments demonstrate that NextMem achieves superior performance, and excels in retrieval, robustness, and extensibility properties. We release our code and model checkpoints at https://github.com/nuster1128/NextMem . 1 Introduction In recent years, LLM-based agents have emerged as a new AI paradigm in many fields ( 26 ; 30 ) , such as personal assistants ( 11 ) and academic research ( 37 ) . Memory is among their most critical components, which is responsible for retaining past information to support future decision-making ( 38 ) . Although the information is typically stored as multiple levels of memory ( 12 ) , factual memory still remains as their foundation, preserving details of observed facts. Compared to other task-oriented memories, such as preference ( 22 ) and experience ( 42 ) , which commonly require task-specific extraction from the original information, factual memory emphasizes its lossless preservation, as we present their comparison in Figure 1 . Figure 1 : Comparison between task-oriented and factual memory. Previous research typically represents memory in two forms, including texts and parameters ( 38 ) . Textual memory is commonly utilized as a context for prompting LLMs, which can be stored and retrieved by databases ( 18 ; 43 ) . However, it increases context length and indexing overhead when storing and using a large amount of detailed facts. Other studies incorporate information by modifying the parameters of LLMs ( 6 ; 35 ) . However, they often suffer from catastrophic forgetting and high costs to store detailed facts accurately. Both paradigms have obvious limitations in managing factual memory. Recent studies in LLM reasoning propose to leverage latent embeddings to represent intermediate reasoning steps ( 7 ; 31 ) , as well as compresses task instructions into latent embeddings for optimization ( 15 ; 29 ) . These studies show the potential of latent representations in storing information. Nevertheless, they primarily focus on improving reasoning efficiency or task adaptation, rather than preserving factual information. Motivated by these studies, we intend to leverage latent representations to address the limitations in managing factual memory. Our primary goals are two-fold: (1) Efficiently transform textual memory into shorter latent representations compatible with LLMs. (2) Latent representations can be accurately and efficiently reconstructed into the original memory. We emphasize reconstruction because factual memory requires lossless preservation rather than partial extraction. Therefore, unlike extraction or indexing approaches, our encoding and decoding processes should be reversible. In this study, we propose a simple yet effective latent memory framework, named NextMem . We design an autoregressive autoencoder built upon LLMs for efficient memory encoding and decoding. Our training process has two critical stages, including autoregressive reconstruction alignment and progressive latent substitution, with further quantization to reduce storage cost while preserving accuracy. Additionally, we conduct extensive experiments to verify model effectiveness and reveal key properties and insights. Our experiments also indicate that the latent memory encoded by NextMem exhibits superior retrieval properties, robustness, and extensibility. To facilitate future research, we have released our code and model checkpoints at https://github.com/nuster1128/NextMem . Our contributions are summarized as follows: \bullet We introduce a simple yet effective framework for latent factual memory, with autoregressive reconstruction alignment and progressive latent substitution. \bullet We integrate quantization methods into the latent memory of our framework, which reduces storage overhead while maintaining competitive performance. \bullet We validate our approach through extensive experiments and provide further insights. We provide the research community with open-source code and model checkpoints. 2 Preliminaries 2.1 Memory in LLM-based Agents Unlike vanilla LLMs that generate single-turn responses, LLM-based agents interact iteratively with their environments. We formalize this process as a Markov Decision Process (MDP) ( 23 ) . Specifically, let S and A denote the state and action spaces. The environment is featured by a transition function E:S\times A\rightarrow S with rewards. At each timestep t , the agent selects an action a^{t}=f(s^{t}) based on the current state s^{t}\in S via a policy f . Then, the state will be updated by the environment s^{t+1}=E(s^{t},a^{t}) with a reward r^{t} , which can be further observed by agents to take the next action a^{t+1} . This process repeats until task completion, and the objective is to optimize the policy f to maximize the cumulative reward \sum r^{t} . Memory is fundamental to agent decision-making. To accommodate diverse memory representations, we conceptualize the memory procedure as an encoding-decoding process rather than the traditional paradigm of storage, retrieval, and utilization ( 40 ) . Under this framework, an agent encodes historical information into memory, which is subsequently decoded to augment LLM inference. Specifically, let M^{t} denote the memory state at timestep t . The encoding process is formalized as: M^{t}=\text{Encode}(s^{t},M^{t-1}), which is executed upon observing the current state s^{t} . The decoding process is integrated into agent’s policy function: f(s^{t})=\text{LLM}\circ\text{Decode}(s^{t},M^{t}), where function LLM represents LLM inference with core capabilities, such as reasoning, inference, and tool calling. Importantly, information encoded into M^{t} at step t may not be utilized until a later timestep, reflecting the asynchronous feature of encoding and decoding in agentic workflows. 2.2 Representation of Agent Memory Memory can be manifested in different representations, and our encoding-decoding framework provides a unified perspective for these memory forms as follows. Textual Memory. Textual memory stores information in text format and leverages in-context learning to integrate relevant data into LLM inference ( 38 ) . This approach often employs indexing structures for query-based retrieval. The encoding process is formulated as: \displaystyle m^{t}=\text{LLM}(\theta;p_{\text{extract}}\|s^{t})\in\mathcal{V}^{n}, \displaystyle M^{t}=M^{t-1}\oplus\{(m^{t},\text{Index}(m^{t}))\}, where p_{\text{extract}} denotes the prompt used to extract memory from observations, \| represents concatenation, and \oplus signifies a structural merge. Here, we use \text{LLM}(\theta;\cdot) to represent LLM inference parameterized by \theta over a vocabulary \mathcal{V} with sequence length n , and use \text{Index}(\cdot) to denote the approach of indexing. Then, the decoding process can be defined as: \displaystyle q^{t}=\text{LLM}(\theta;p_{\text{intent}}\|s^{t})\in\mathcal{V}^{n}, \displaystyle c^{t}=\text{Retrieval}(\text{Index}(q^{t}),M^{t}), \displaystyle f(s^{t})=\text{LLM}(\theta;p_{\text{instruct}}\|s^{t}\|c^{t}), where p_{\text{intent}} is the prompt for generating the retrieval intent, p_{\text{instruct}} is the task-specific instruction prompt, and \text{Retrieval}(\cdot) denotes the operation of fetching relevant context c^{t} based on the established indexes. Parametric Memory. Parametric memory incorporates new information by modifying model parameters, which are subsequently integrated with the base parameters at the inference stage. The encoding process can be formalized as: \displaystyle\Delta\theta^{t}=h_{\text{meta}}(s^{t},\theta^{0}\oplus\Delta\theta^{t-1})\in\mathbb{R}^{d}, \displaystyle M^{t}:=\theta^{0}\oplus\Delta\theta^{t}, where \theta^{0} denotes the base parameters of the LLM. The function h_{\text{meta}}(\cdot) is designed to predict parameter modifications within the space \mathbb{R}^{d} based on the current observation s^{t} . The decoding process is defined as: \displaystyle f(s^{t})=\text{LLM}(\theta^{0}+\Delta\theta^{t};p_{\text{instruct}}\|s^{t}), where \Delta\theta^{t} represents the cumulative