Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use
Nicholas Roth, Christopher Hidey, Lucas Spangher, William F. Arnold, Chang Ye, Nick Masiewicki, Jinoo Baek, Peter Grabowski, Eugene Ie · 2025
Read Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use by Nicholas Roth, Christopher Hidey, Lucas Spangher, William F. Arnold, Chang Ye, Nick Masiewicki, Jinoo Baek, Peter Grabowski, Eugene Ie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22931.txt.
Introduction
Nicholas Roth and co-authors split a tool-use aide into two minds: a big plan mind and a small rote mind. One-mind aides slip in set ways: bent call fields, lost fields, made-up fields, and stale plans in fast yards. Their 2025 split gives the big model the plan and the live read of user words. The small model holds tool shapes and outs by rote, a fast, cheap post.
Races show plan truth and slip grit both climb in the split form. A trade stands bare: live learn in the big mind versus set rote in the small. Split minds may be the road to aides that hold up in wild yards.
Catalog record
| CREATOR | Nicholas Roth, Christopher Hidey, Lucas Spangher, William F. Arnold, Chang Ye, Nick Masiewicki, Jinoo Baek, Peter Grabowski, Eugene Ie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22931
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Factored Agents: Decoupling In-Context Learning and Memorization for Robust Tool Use Nicholas Roth, Christopher Hidey, Lucas Spangher, William F. Arnold, Chang Ye, Nick Masiewicki, Jinoo Baek, Peter Grabowski, Eugene Ie Abstract In this paper, we propose a novel factored agent architecture designed to overcome the limitations of traditional single-agent systems in agentic AI. Our approach decomposes the agent into two specialized components: (1) a large language model (LLM) that serves as a high level planner and in-context learner, which may use dynamically available information in user prompts, (2) a smaller language model which acts as a memorizer of tool format and output. This decoupling addresses prevalent issues in monolithic designs, including malformed, missing, and hallucinated API fields, as well as suboptimal planning in dynamic environments. Empirical evaluations demonstrate that our factored architecture significantly improves planning accuracy and error resilience, while elucidating the inherent trade-off between in-context learning and static memorization. These findings suggest that a factored approach is a promising pathway for developing more robust and adaptable agentic AI systems. 1 Introduction Recent advances in Artificial Intelligence (AI) have brought forth the promise of “Agentic AI” systems endowed with a degree of autonomy and goal-directed behavior that could fundamentally transform computing and human–computer interactions. Agentic AI can be broadly defined as a class of models that can competently and reliably interact with external APIs, which they may use to interact with tools and systems. For our purposes, we note that they may synthesize both learned knowledge and contextual cues to achieve their objectives. Such systems hold the potential to not only perform complex computations but also to adapt dynamically to new tasks and environments ( Acharya et al. 2025 ) . In its most common instantiation, agentic AI is generally proposed as a single-agent design. This baseline approach involves a monolithic Large Language Model (LLM) responsible for all aspects of planning, memory, and action execution. Such a system may be effective in a static environment when the format of the APIs that interact with external systems are included in its training or fine-tuning set. However, in a large, dynamic, and real-world system, a single agent is ill-suited. Specifically, the single-agent framework is susceptible to several types of errors: those at the small scale of producing correct structured API output, and those at the larger scale of deciding what to do. In the small scale, we note that many of the following errors may come from parsing errors : 1. Malformed Fields: the LLM may misspell or misunderstand the input type required from it 2. Missing Fields: the LLM may not understand the complete set of arguments necessary for a given API call 3. Hallucinated Fields: the LLM may imagine additional fields that don’t exist – a long noted trait of LLMs ( Goddard 2023 ) . Such agents may also exhibit general planning errors , in which they suggest an incorrect order or set of actions to partake in, thereby limiting their effectiveness. As we will argue further on, a critical tension arises between skills related to in-context learning (ICL) and memorization: we will argue that the parsing errors may be addressed through an agent skilled in memorizing, whereas the planning errors may be addressed with an agent skilled in in-context learning, and improvement in one area necessarily leads to decline in the other. To address these challenges, we propose a novel factored agent architecture that decouples the roles of memory and context adaptation. Our design partitions the agent into two specialized components: 1. A larger in-context learner that dynamically assimilates new information from the prompt and plans an appropriate tool-use prompt. 2. A small language model which acts as a memorizer for tool APIs that retains and retrieves long-term knowledge. Our factored agent thereby mitigates the drawbacks of the traditional single-agent approach while retaining its core benefits. The contributions of this paper are as follows: • We introduce a factored agent architecture that distinctly separates memorization and in-context learning. • We provide a detailed critique of current single-agent frameworks, highlighting issues such as malformed, missing, and hallucinated fields, as well as general planning errors. • We analyze the trade-offs between in-context learning and static memorization, offering insights into their respective advantages and limitations. • We demonstrate through empirical evaluation that our proposed architecture significantly improves planning accuracy and error resilience in real-world tasks. In the subsequent sections, we elaborate on the design and implementation of our factored agent, present comprehensive experimental results, and discuss the broader implications of our work for the future of agentic AI. 2 Background Literature Agentic AI is a rapidly evolving field that seeks to create intelligent agents capable of performing tasks requiring both in-context learning (ICL) and memorization. A central challenge in this endeavor is that within modern single agents, there appears to be a trade off between these skills. Recent work, such as Li et al. 2023a , observes that as an agent becomes more of a memorizer, its ICL capabilities diminish, and vice versa; Biderman et al. 2024 report this finding as well in Low Rank Adaptor models. Regrettably for tool use models, memorization tends to increase in larger models ( Satvaty et al. 2024 ; Kiyomaru et al. 2024 ) and seems to occur in current tool use agents ( Li et al. 2023b ) . This suggests that a single agent may not be optimal for all tasks. To address this, researchers have explored various approaches. TinyAgents ( Erdogan et al. 2024 ) fine-tune small language models for specific tasks, but this approach may not scale well to a large number of tasks. ToolFormer ( Schick et al. 2023 ) is a transformer specifically trained to generate API calls, but it may hallucinate in novel situations. Qin et al. 2024 trained ToolLLaMA on over 16,000 APIs (composing the ToolBench dataset) and evaluated on a subset. Gorilla ( Patil et al. 2023 ) is a more general agent that can interact with APIs by including API documentation in a Retrieval Augmented Generation (RAG) approach, and maintains a continually expanding database of API calls. However, both approaches may fail when performing supervised fine-tuning (SFT) on the same model that may also need planning and ICL abilities. A promising direction is to combine the strengths of memorization and ICL. ReAct ( Yao et al. 2023 ) is an example of an agent that reasons and acts in the world. ReAct focuses on general orchestration of two agents, with the “reasoning” agent being a base reasoner, and the “acting” agent being the tool agent, but they often rely on external tools and knowledge bases. Other, older approaches have factored agent architectures: Wen et al. 2016 and Wu et al. 2019 are notable examples. By carefully balancing memorization and ICL, we can create more versatile and intelligent agents. 3 System Design Figure 1 : Representation of the factored agents model with a flow of information between users, with hand-offs between various models. Please see Figure 1 for a diagram of the model setup and a flow of intermediate hand-offs between agents in our factored approach. In an abstracted example, let’s assume that the user may desire to adopt a cat in San Francisco. They would