Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning
Yuval Shemla 1 , Ayal Yakobe 2 , Tanmay Agarwal 2 , Dhaval Patel 3 , Kaoutar El Maghraoui 2 · 2026
Read Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning by Yuval Shemla 1 , Ayal Yakobe 2 , Tanmay Agarwal 2 , Dhaval Patel 3 , Kaoutar El Maghraoui 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17774.txt.
Introduction
Yuval Shemla and four colleagues test whether small language models can absorb tool knowledge into their weights instead of reading full tool descriptions with every prompt. Fine-tuning Gemma and Qwen3 models with 8-bit QLoRA on about 1,700 tool-use examples, they find description-free planning beats an informed but untuned baseline while cutting input length by 82.6%. The best Gemma run scores 0.65 on AT-F1 against 0.47 for the baseline, though the smaller Qwen model runs faster with less memory and forgets more general knowledge.
Catalog record
| CREATOR | Yuval Shemla 1 , Ayal Yakobe 2 , Tanmay Agarwal 2 , Dhaval Patel 3 , Kaoutar El Maghraoui 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38738 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17774
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning Yuval Shemla 1 , Ayal Yakobe 2 , Tanmay Agarwal 2 , Dhaval Patel 3 , Kaoutar El Maghraoui 2 Abstract Tool-using language agents commonly include full tool schemas in every prompt, even when the available tools are fixed across many queries. This repeated schema context increases input length and can make small models unreliable planners. We study whether small language models can internalize a fixed tool catalog through supervised parameter-efficient fine-tuning, enabling structured tool planning without explicit tool descriptions at inference time. Using AssetOpsBench, an industrial asset-operations benchmark with MCP-style tools, we fine-tune Gemma 4 E4B and Qwen3-4B with 8-bit QLoRA on approximately 1,700 tool-use examples. Under description-free inference, the fine-tuned models outperform an unfine-tuned schema-informed baseline that receives the full tool catalog, while reducing prompt length by 94.7%. The best Gemma run achieves an AT-F1 of 0.65 and an overall LLM-judge score of 3.88, compared with 0.47 and 2.88 for the informed baseline. These results suggest that, for fixed tool catalogs, supervised QLoRA fine-tuning can shift tool knowledge from prompt context into adapter weights, reducing repeated schema overhead while maintaining or improving tool-planning quality. I Introduction Large language models (LLMs) are increasingly used as planning components in tool-using agents, where they decompose user requests, select tools, construct arguments, and coordinate multi-step workflows. In many current tool-use pipelines, including MCP-style systems [ 1 ] , the model receives a serialized tool catalog in the prompt so that it can infer which tools exist and how they should be called. This schema-informed approach is flexible: new tools can be added by updating the prompt. However, it also repeats the same catalog across queries, creating substantial input overhead. This paper studies a narrower but practically important regime: fixed-catalog tool planning . In many enterprise and industrial deployments, the available tool catalog changes slowly relative to the number of user queries. In such settings, repeatedly prompting with full tool descriptions may be unnecessary. We ask whether this stable catalog knowledge can instead be amortized into a small model through supervised fine-tuning, allowing the model to generate structured tool-use plans without receiving tool descriptions at inference time. This setting targets two related inefficiencies. First, small language models often require extensive tool-context prompting to produce valid plans. Second, repeated schema prompting can dominate the input budget. In our AssetOpsBench setting [ 14 ] , the serialized tool catalog accounts for approximately 2,200 of the 2,400 prompt tokens, while the user query itself is short. Description-free inference removes this catalog section entirely, reducing the prompt to approximately 128 tokens and asking the fine-tuned model to recover server routing, tool selection, argument keys, and dependency structure from its learned parameters. I-A Contributions We propose and evaluate description-free tool planning: a fixed-catalog setting in which the model must generate structured tool-use plans without receiving tool descriptions at inference time. Our contributions are: 1. We formalize static-catalog description-free tool planning , where a model must select MCP servers, tools, arguments, and dependencies for a fixed catalog while omitting the serialized tool catalog from the inference prompt. 2. We develop a catalog-grounded supervised fine-tuning protocol in which a teacher model has access to full tool schemas during data construction, while the student model is trained and evaluated with the tool catalog removed. 3. We show that 8-bit QLoRA fine-tuning on approximately 1,700 examples enables \sim 4B parameter models to outperform an unfine-tuned schema-informed baseline on AssetOpsBench, despite receiving no tool descriptions at inference time. 4. We evaluate whether the resulting specialization preserves broader model capability, showing that LoRA rank induces a quality–retention trade-off between tool-planning performance and general multiple-choice accuracy. II Related Work II-A Tool-Use and Function-Calling Evaluation Tool-use evaluation has evolved from measuring whether models can call individual APIs to testing tool selection, argument construction, dependency tracking, and multi-step execution. API-Bank [ 11 ] introduced a runnable benchmark for tool-augmented LLMs with 73 API tools. BFCL [ 15 ] evaluates function calling across serial and parallel invocations using AST-based evaluation. MCP-Bench [ 19 ] extends this to MCP-style agents, evaluating multi-step tasks that require cross-tool coordination across live MCP servers. Our work does not propose a new benchmark; instead, we study a stricter inference condition within a fixed-domain benchmark: the model must plan without receiving tool descriptions at inference time. II-B Schema-Informed and Retrieval-Augmented Tool Learning A dominant approach to tool use is to expose tool information at inference time through prompt schemas or retrieved documentation. Gorilla [ 16 ] fine-tunes models for API invocation with retrieval-aware training. ToolLLM [ 17 ] constructs ToolBench over 16,464 real-world APIs with a neural API retriever. ToolACE [ 12 ] shows that high-quality synthetic function-calling data with automatic verification can train strong small models. These approaches are well suited to broad or changing API ecosystems. Our setting makes the opposite trade-off: we assume a fixed, mature tool catalog and ask whether its schema information can be amortized into model parameters. II-C Tool and Skill Internalization Recent work has begun to move tool or skill knowledge from prompt context into model parameters. ToolGen [ 18 ] represents each tool as a unique vocabulary token, integrating tool knowledge into the language model’s generation process and enabling unified tool retrieval and calling across over 47,000 tools. SKILL0 [ 13 ] studies skill internalization for embodied agents by providing skill context during RL training and progressively withdrawing it until the agent operates without runtime skill retrieval. Our work is motivated by the same observation, prompt-time tool context is expensive, but uses a different approach: ordinary supervised fine-tuning with QLoRA on a fixed industrial catalog, without special tool tokens, vocabulary expansion, or reinforcement learning. II-D Prompt Compression and Schema Removal Prompt-compression methods such as LLMLingua [ 10 ] reduce inference cost by removing less important prompt tokens. Description-free tool planning can be viewed as an extreme fixed-catalog variant of prompt compression: rather than compressing the tool catalog, we remove the entire serialized schema from the inference prompt. The missing catalog information must therefore be recovered from the fine-tuned model weights rather than from a shorter prompt. Unlike standard knowledge distillation [ 8 ] , we do not require teacher model inference at test time. II-E Parameter-Efficient Fine-Tuning and Retention LoRA [ 9 ] freezes the pretrained model and trains low-rank adapter updates, while QLoRA [ 3 ] further reduces memory by quantizing the frozen base model. Although adapter-based fine-tuning generally reduces catastrophic forgetting compared with full fine-tuning [ 6 ] , adapter updates can still bias the effective model toward the fine-tuning distribution and degrade prior capabilities. This is especially relevant for tool planning, where the training distribution consists of structured plans rather than general natural language. We therefore evaluate both tool-planning quality and general multiple-choice retention after fine-tuning. II-F Industrial Tool-Planning Benchmarks We use AssetOpsBench [ 14 ] because it provides a bounded but realistic industrial tool-planning environment with domain-specific MCP servers, simulated telemetry, human-authored scenarios, and multi-step workflows. This differs from broad benchmarks such as ToolBench [ 17 ] and MCP-Bench [ 19 ] , which evaluate large-scale tool discovery or cross-domain MCP usage. Our goal is not broad generalization to unseen tools, but