Memento: Fine-tuning LLM Agents without Fine-tuning LLMs

Huichi Zhou*, Yihang Chen*, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang†, Jun Wang† · 2025

Read Memento: Fine-tuning LLM Agents without Fine-tuning LLMs by Huichi Zhou*, Yihang Chen*, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang†, Jun Wang† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16153.txt.

Introduction

Huichi Zhou and ten co-authors teach AI agents to learn from memory without retraining in a 2025 paper. Retraining a large language model is costly and slow. Fixed reflection scripts are cheap but rigid. The team wants agents that adapt on the job at low cost.

Their agent, Memento, stores past cases in an episodic memory bank. A small learned policy picks which cases guide each new task. Feedback from the world updates the memory over time. The base language model itself stays frozen throughout. On a broad agent test set, Memento ranked first in its class. Memory from past cases added clear gains on new task types. The method suits deep research tasks that change often.

Catalog record

CREATORHuichi Zhou*, Yihang Chen*, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang†, Jun Wang†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16153

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs Huichi Zhou*, Yihang Chen*, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang†, Jun Wang† Abstract Abstract In this paper, we introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either rigid, relying on static, handcrafted reflection workflows, or computationally intensive, requiring gradient updates of LLM model parameters. In contrast, our method enables low-cost continual adaptation via memory-based online reinforcement learning. We formalise this as a Memory-augmented Markov Decision Process (M-MDP), equipped with a neural case-selection policy to guide action decisions. Past experiences are stored in an episodic memory, either differentiable or non-parametric. The policy is continually updated based on environmental feedback through a memory rewriting mechanism, whereas policy improvement is achieved through efficient memory reading (retrieval). We instantiate our agent model in the deep research setting, namely Memento , which attains top-1 on GAIA validation ( 87.88 % 87.88\% Pass@ 3 3 ) and 79.40 % 79.40\% on the test set. It reaches 66.6 % 66.6\% F1 and 80.4 % 80.4\% PM on the DeepResearcher dataset, outperforming the state-of-the-art training-based method, while case-based memory adds 4.7 % 4.7\% to 9.6 % 9.6\% absolute points on out-of-distribution tasks. Our approach offers a scalable and efficient pathway for developing generalist LLM agents capable of continuous, real-time learning without gradient updates, advancing machine learning towards open-ended skill acquisition and deep research scenarios. The code is available at https://github.com/Agent-on-the-Fly/Memento . (a) Memento vs. Baselines on GAIA validation and test sets. (b) Ablation study of Memento across benchmarks. (c) Continual learning curves across memory designs. (d) Memento ’s accuracy improvement on OOD datasets. Figure 1 : Overview of Memento evaluation across baselines, benchmarks, memory designs and generalisation. 1 Introduction A Large Language Model (LLM) agent refers to a system that leverages one or more LLMs to autonomously perform complex tasks through interaction, reasoning, and decision making, often with access to external tools, memory, or environments ( Christianos et al. 2023 , Yang et al. 2025 ) . Unlike passive LLMs that respond to prompts in isolation, LLM agents operate proactively and iteratively, guided by explicit goals. They are increasingly deployed as autonomous problem solvers ( Choudhary et al. 2021 , Wei et al. 2022 , Yao et al. 2023 ) spanning various domains. Notable examples include deep research agents ( OpenAI 2025 , Google 2025 , ByteDance 2025 ) , tool-enhanced execution systems ( Li et al. 2025c , Zheng et al. 2025 , Qian et al. 2025 ) , and code generation agents ( Cui et al. 2021 , Guo et al. 2024 , Grosnit et al. 2024 , Guo et al. 2025 ) , all of which demonstrate strong capabilities in complex scientific and engineering tasks. Despite recent progress, current LLM agents typically follow two prevailing paradigms, each exhibiting fundamental limitations. The first approach builds specialised frameworks with fixed workflows and hard-coded reasoning, which work well for narrow tasks but lack flexibility. After deployment, such agents are static: they neither incorporate online information nor adapt to novel situations. The second paradigm focuses on updating the LLM itself through parameter tuning of underlying LLMs – via supervised fine-tuning or reinforcement learning – which allows for more flexible behaviour ( Christianos et al. 2023 , Shi et al. 2025 ) but comes at a high computational cost. These approaches are inefficient for continuous adaptation and online learning, impractical for agents deployed in open-ended scenarios. This observation raises a central research challenge towards generalist agents: How can we build LLM agents that learn continuously from a changing environment without the prohibitive cost of fine-tuning the underlying LLMs? Inspired by human memory mechanisms, we address this challenge by proposing a memory-based learning framework that enables continual adaptation without modifying the underlying LLMs. We observe that humans’ performance steadily improves because each experience is (i) encoded as an episodic trace ( Pritzel et al. 2017 ) , (ii) distilled into abstract rules during sleep-dependent consolidation ( Squire et al. 2015 ) , (iii) selectively reinforced by dopamine-driven credit assignment ( Glimcher 2011 ) , and (iv) retrieved through case- or analogy-based reasoning when similar problems arise ( Ashley 1992 ) . Thus, instead of fine-tuning the base model, LLM agents leverage an external memory to store past trajectories – including successes and failures labels – and draw from similar past experiences to guide decision making. This approach aligns with the principles of case-based reasoning (CBR) ( Aamodt and Plaza 1994 , Guo et al. 2024 , Guo et al. 2025 ) , a psychologically grounded learning strategy supported by evidence that humans often solve problems by recalling analogous past situations ( Anderson 2013 , Ross 1989 ) . For example, in a deep research scenario, deep research agents that have previously succeeded on a web-based task can leverage their experience to solve never-seen and structurally similar tasks ( Wiratunga et al. 2024 ) . Our method offers a novel path to continual learning for deep research agents – efficient, generalizable, and inspired by how humans learn. To this end, we introduce Memento , a non-parametric, learn-on-the-fly framework for CBR ( Smyth and McClave 2001 , Hatalis et al. 2025 ) , instantiated as a planner–executor architecture grounded in a memory-based Markov Decision Process (MDP). Memento comprises three principal components: (i) a planner, (ii) a tool - enabled executor, and (iii) a growing Case Bank that stores past trajectories as episodic memory. Instead of relying solely on the LLM’s parametric memory, which is fixed after training, online case-based reasoning in Memento is implemented by storing rich episodic traces. Our experiments are conducted on 4 benchmarks, where GAIA ( Mialon et al. 2023 ) for long-horizon tool use, DeepResearcher ( Zheng et al. 2025 ) for real-time web research, SimpleQA ( Wei et al. 2024 ) for factual precision, and HLE ( Phan et al. 2025 ) for long-tail academic reasoning. We use a planner–executor architecture with GPT- 4.1 as the planner and o 4 -mini as the default executor (o 3 for GAIA), instrumented with tools, namely Memento . We achieve top-1 on GAIA validation ( 87.88\% Pass@ 3 ) and 79.40\% on the private test leaderboard, and it reaches 66.6\% F 1 and 80.4\% PM on the DeepResearcher dataset, outperforming the state-of-the-art training-based system, while case-based memory adds 4.7 to 9.6 absolute points on out-of-distribution tasks and yields 95.0\% PM on SimpleQA. To our knowledge, we are the first to cast case-based continual learning for LLM agents, achieving the top-level performance on the GAIA benchmark, thereby providing a principled framework for continual adaptation of Deep Research agents. 2 Related Work We first review methods that equip LLMs with continual-learning capabilities. Then, we discuss approaches that augment agents with external tools and multi-agent coordination. Lastly, we introduce agent memory mechanisms, characterising design choices in representation, retrieval, and decay, and their implications for continual learning. 2.1 Continual-learning in LLM Agent Systems Continual-learning strategies for LLM agents can be categorised into two camps. Parametric approaches ( Zhu et al. 2025b , Zhu et al. 2025a ) update the

Read the complete text