What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
Xinjie He, Zhiyuan Lin, Su Liu, Jialun Wu, Qiyang Xie, Weikai Zhou, Shuai Xiao, [0.5em] Columbia University · 2026
Read What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA by Xinjie He, Zhiyuan Lin, Su Liu, Jialun Wu, Qiyang Xie, Weikai Zhou, Shuai Xiao, [0.5em] Columbia University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23067.txt.
Introduction
He and six coauthors study how training data composition shapes RL memory agents for multi-session dialogue, holding architecture, algorithm, and hyperparameters fixed. Three curricula are compared: in-domain LoCoMo, mixed LoCoMo plus LongMemEval, and out-of-domain LongMemEval only. The mixed curriculum wins F1 on both test sets. Narrow out-of-domain training transfers temporal reasoning skill despite weak aggregate scores. Per-type gaps dwarf aggregate gaps, so single-number comparisons underreport curriculum effects. Two lessons cover filtering format noise and using continuous rewards at small group sizes.
Catalog record
| CREATOR | Xinjie He, Zhiyuan Lin, Su Liu, Jialun Wu, Qiyang Xie, Weikai Zhou, Shuai Xiao, [0.5em] Columbia University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32009 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23067
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA Xinjie He, Zhiyuan Lin, Su Liu, Jialun Wu, Qiyang Xie, Weikai Zhou, Shuai Xiao, [0.5em] Columbia University Abstract Reinforcement learning (RL) has emerged as a viable recipe for training LLM agents to reason over external memory banks in multi-session dialogue. Existing work trains exclusively on a single benchmark, leaving open how the composition of training data shapes the skills a memory agent acquires. We present a controlled empirical study that holds architecture, RL algorithm, and all hyperparameters fixed and varies only the training curriculum across three conditions: in-domain (LoCoMo), mixed-benchmark (LoCoMo + LongMemEval), and out-of-domain (LongMemEval only). Across two benchmarks and ten question types, curriculum composition acts as a fine-grained lever on specialization rather than a uniform scaling factor on performance. The mixed curriculum yields the strongest overall F1 on both evaluation sets. Training on a narrow out-of-domain set transfers a targeted skill — temporal reasoning — despite weak aggregate performance. And per-type differences substantially exceed aggregate differences, indicating that single-number benchmark comparisons systematically underreport curriculum effects. We further report two practical lessons from adapting GRPO to a single-GPU regime: cross-benchmark mixing requires filtering format-specific noise from memory banks to preserve training signal, and binary exact-match reward produces no learning signal at the small group sizes ( G = 4 G=4 ) required on one GPU, motivating continuous reward functions in this regime. 1 Introduction Large language models operate within fixed context windows, with no persistent memory across interactions. This limitation is acute in multi-session dialogue, where users expect the system to recall preferences, events, and relationships from prior conversations. Recent work addresses this by augmenting LLMs with external memory banks — structured stores that persist across sessions and support retrieval at inference time [ 2 , 27 , 10 , 18 ] . A key challenge is learning to use retrieved memories well: selecting relevant entries from a noisy candidate set, reasoning across them, and producing concise answers. Heuristic pipelines rely on fixed retrieval rules; RL-based approaches [ 28 ] instead let the agent discover such selection and reasoning patterns through outcome-driven training, achieving competitive results with small supervision budgets. The framing that has dominated this line of work is training on a single benchmark and reporting aggregate scores. Yet memory-augmented QA is a multi-skill problem: different question types exercise different combinations of retrieval precision, multi-hop composition, temporal ordering, and knowledge-update tracking. When a single benchmark exercises only a subset of these skills, single-benchmark training implicitly selects which capabilities the RL signal reinforces. This raises two questions the existing literature leaves open. First, does exposing the RL signal to a broader source of question types — mixing benchmarks — bias the policy toward a more general memory skill set, or does it dilute the in-domain skill without cross-benchmark gains? Second, when only a small out-of-domain set is available, does RL surface a targeted capability, or does it fail entirely? We refer to the resulting per-question-type profile as the policy’s specialization : systematic variation in per-type strengths induced by training curriculum composition. To answer these, we fix architecture (Qwen-2.5-7B with LoRA), RL algorithm (GRPO), and all optimization hyperparameters, and vary only the training curriculum. Config A replicates the single-benchmark baseline of prior RL-memory work with 152 LoCoMo QA pairs. Config B mixes LoCoMo with 60 LongMemEval pairs (212 total). Config C trains on the 60 LongMemEval pairs alone. All three are evaluated on both LoCoMo (1,307 test questions, 4 types) and LongMemEval (415 test questions, 6 types). Our contributions are as follows. (i) We present a controlled curriculum study for RL-based memory agents in which architecture, algorithm, and hyperparameters are held fixed and only the training source varies. Under this design, per-question-type differences substantially exceed aggregate differences, so curriculum composition acts as a lever on specialization rather than on overall accuracy (Section 4.2). (ii) We draw practical guidance from this design space: mixing benchmarks yields the strongest generalist in our setting, a narrow out-of-domain set can induce a targeted behavior (temporal reasoning), and the transition between targeted specialization and more stable aggregate gains appears between roughly 60 and 150 training examples in our setup (Section 5.3). (iii) We document two engineering findings that affect the reproducibility of GRPO-based memory training on a single GPU: cross-benchmark data requires filtering format-specific noise from the memory bank (Section 5.1), and binary exact-match reward collapses to zero advantage at G=4 , strongly motivating continuous reward functions when the large-group regime is not available (Section 5.2). Figure 1: Experimental design. Three curricula, identical training recipe, evaluated on two benchmarks with per-type F1 breakdown. 2 Related Work 2.1 Memory-Augmented LLM Agents The challenge of equipping LLMs with persistent memory has motivated several architectural approaches. Early agent frameworks such as Reflexion [ 22 ] demonstrated the value of persistent state for multi-step reasoning, though their memory policies are largely handcrafted. Recent memory systems build on this foundation: MemGPT [ 14 ] treats the LLM’s context window as a virtual memory with OS-inspired paging between a primary and secondary store. Mem0 [ 2 ] provides a modular memory system with explicit CRUD operations. A-Mem [ 27 ] introduces dynamic agentic memory with structured entries. LangMem [ 10 ] chains memory entries across sessions. Zep [ 18 ] employs a temporally-aware knowledge graph for agent memory, benchmarking directly against MemGPT on Deep Memory Retrieval and LongMemEval. These systems rely on heuristic memory management policies. Recent work has begun applying RL to memory-augmented agents. Memory-R1 [ 28 ] trains both a Memory Manager (for CRUD operations) and an Answer Agent (for memory-grounded QA) using GRPO [ 21 ] , achieving strong results with 152 training examples. Our work focuses on the Answer Agent component — the agent that selects and reasons over retrieved memories to answer questions — and studies how training data composition affects its learned skills. We use heuristic memory construction and focus our RL training on the answer generation policy, isolating the effect of curriculum composition from memory management quality. 2.2 Benchmarks for Long-Term Memory LoCoMo [ 12 ] (ACL 2024) provides multi-session dialogues averaging 26,000 tokens with approximately 200 questions per dialogue spanning single-hop, multi-hop, open-domain, and temporal reasoning. It features rich multi-party conversations between named speakers. LongMemEval [ 25 ] (ICLR 2025) offers 500 questions across six categories: single-session-user, single-session-assistant, single-session-preference, multi-session, temporal-reasoning, and knowledge-update. Each question is paired with approximately 40 haystack sessions in user-assistant chat format, testing precise retrieval from large conversation histories. Both benchmarks exercise retrieval-augmented generation [ 11 ] over dialogue, relying on dense retrievers [ 9 , 19 ] to surface relevant context. These benchmarks test complementary skills: LoCoMo emphasizes reasoning over rich dialogues, while LongMemEval emphasizes precise retrieval and temporal reasoning. This complementarity motivates our mixed-curriculum approach. 2.3 Curriculum Learning for RL-Based LLM Training Curriculum learning — structuring training data to improve learning — is a long-standing idea [ 1 , 5 ] that has been revisited for LLM post-training. Recent work in this area has focused on the difficulty axis : difficulty-based curricula [ 15 , 8 ] schedule examples from easy to hard, and distribution-level curricula [ 24 ] reweight data