Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning
Polaris Jhandi, Owais Kazi, Shreyas Subramanian, Neel Sendas · 2025
Read Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning by Polaris Jhandi, Owais Kazi, Shreyas Subramanian, Neel Sendas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15943.txt.
Introduction
Researchers show that a fine tuned small language model can beat large models at agentic tool calling in a 2025 paper. Large models are costly for routine enterprise work, which motivates targeted small models. The team fine tunes the OPT-350M model, released by Meta AI in 2022, with supervised fine tuning for tool use tasks.
After one epoch of training, the small model reaches a 77.55 percent pass rate on the ToolBench evaluation. That score far exceeds ChatGPT with chain of thought at 26.00 percent, ToolLLaMA DFS at 30.18 percent, and ToolLLaMA CoT at 16.27 percent in their comparison. The authors argue thoughtful targeted training lowers adoption barriers for production generative AI.
Catalog record
| CREATOR | Polaris Jhandi, Owais Kazi, Shreyas Subramanian, Neel Sendas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15943
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuning Polaris Jhandi, Owais Kazi, Shreyas Subramanian, Neel Sendas Abstract As organizations scale adoption of generative AI, model cost optimization and operational efficiency have emerged as critical factors determining sustainability and accessibility. While Large Language Models (LLMs) demonstrate impressive capabilities across diverse tasks, their extensive computational requirements make them cost-prohibitive for routine enterprise use. This limitation motivates the exploration of Small Language Models (SLMs), which can deliver comparable performance in targeted applications while drastically reducing infrastructure overhead (Irugalbandara et al., 2023). In this work, we investigate the feasibility of replacing LLM-driven workflows with optimized SLMs. We trained a domain-adapted SLM to execute representative tasks traditionally handled by LLMs, such as document summarization, query answering, and structured data interpretation. As part of the experiment, we investigated the fine-tuning of facebook/opt-350m model (single epoch only) using the Hugging Face TRL (Transformer Reinforcement Learning), specifically the Supervised Fine-Tuning (SFT) trainer. The OPT-350M model was released by Meta AI in 2022 as part of the OPT (Open Pretrained Transformer) family of models. Similar studies demonstrate that even models at the 350M parameter scale can meaningfully contribute to instruction-tuning pipelines (Mekala et al., 2024). Experimental results demonstrated that our fine-tuned SLM achieves exceptional performance with a 77.55% pass rate on ToolBench evaluation, significantly outperforming all baseline models including ChatGPT-CoT (26.00%), ToolLLaMA-DFS (30.18%), and ToolLLaMA-CoT (16.27%). These benchmarks, first introduced in ToolLLM (Qin et al., 2023) and later stabilized by follow-up efforts (Zhang et al., 2024), have become the standard for evaluating tool-augmented reasoning. Recent work has also extended ToolBench traces to preference-based optimization (Zeng et al., 2024) and designed alternative multi-API corpora for tool-use robustness (Liu et al., 2024). These findings emphasize that thoughtful design and targeted training of SLMs can significantly lower barriers to adoption, enabling cost-effective, large-scale integration of generative AI into production systems. Introduction Running a state-of-the-art LLM at production scale entails substantial infrastructure investment, ongoing operational costs, and often reliance on closed APIs, which introduces additional risks related to data privacy, latency, and robustness. For organizations seeking to embed generative AI deeply into mission-critical operations, such constraints present formidable barriers. This motivates a pivot toward approaches that optimize not only model performance but also economic and operational feasibility. In this paper, we investigate whether SLMs can achieve comparable results to LLMs in agentic tool calling through targeted fine-tuning with practical amounts of human supervision. This question arises from the broader context of generative AI adoption at scale, where the balance between capability and cost has become increasingly critical. Despite significant advances in tool-augmented language models, a critical gap remains in understanding the trade-offs between model size, training efficiency, and task performance. Most existing work focuses on either scaling up model parameters or improving training techniques, but few studies systematically investigate whether small models can achieve competitive performance through targeted optimization. Furthermore, while previous work has shown promising results for SLMs in specific domains, comprehensive evaluation across diverse tool manipulation scenarios has been limited. Our systematic evaluation across six ToolBench categories provides robust evidence for SLM effectiveness in agentic tool calling. Our central hypothesis is that SLMs, when carefully trained and aligned, can achieve near-equivalent performance to LLMs on focused tasks such as tool calling and structured automation. We fine-tuned the facebook/opt-350m model using the ToolBench dataset, conducting training on Amazon SageMaker with Hugging Face TRL library integration. The rest of this paper is structured as follows: Section 2 reviews related work in tool-augmented language models and small model optimization. Section 3 discusses operational challenges with LLMs. Section 4 presents our SLM approach. Section 5 describes the ToolBench evaluation framework. Section 6 presents experimental results demonstrating our SLM’s 77.55% pass rate. Section 7 concludes with implications for scalable AI deployment. Related work Tool-Augmented Language Models The integration of external tools with language models has emerged as a critical research direction for enhancing AI capabilities beyond pure text generation. Early work by Schick et al. (2023) introduced the concept of Toolformer, which teaches language models to use external APIs through self-supervised learning. This foundational work demonstrated that models could learn to invoke calculators, search engines, and other tools to improve their problem-solving capabilities. Building on this foundation, ReAct (Yao et al., 2023) introduced a paradigm combining reasoning and acting, where models alternate between generating thoughts and taking actions. This approach showed significant improvements in multi-step reasoning tasks and became a standard framework for tool-augmented AI systems. The ReAct framework’s success led to widespread adoption in various applications, from web browsing agents to code generation systems. More recently, ToolLLM (Qin et al., 2023) scaled tool integration to over 16,000 real-world APIs, creating comprehensive benchmarks for evaluating tool manipulation capabilities. Their work established ToolBench as a standard evaluation framework and demonstrated that fine-tuned models could achieve competitive performance with proprietary systems like GPT-4 in tool calling scenarios. Concurrent work by Patil et al. (2023) explored Gorilla, focusing on API call generation and demonstrating that smaller, specialized models could outperform larger general-purpose models in specific domains. This work provided early evidence supporting our hypothesis that targeted training can overcome parameter limitations. Small Language Models and Efficiency The pursuit of efficient language models has gained momentum as deployment costs and environmental concerns have grown. Touvron et al. (2023) introduced LLaMA, demonstrating that smaller models trained on high-quality data could match or exceed the performance of much larger counterparts. This work challenged the prevailing assumption that model size directly correlates with capability. Subsequent research by Taori et al. (2023) with Alpaca showed that instruction-following capabilities could be effectively transferred to smaller models through careful fine-tuning on high-quality instruction datasets. Their work demonstrated that a 7B parameter model could achieve performance comparable to much larger systems on many tasks. The concept of knowledge distillation, pioneered by Hinton et al. (2015) and later adapted for language models by Sanh et al. (2019) with DistilBERT, provided theoretical foundations for our approach. These works established that smaller models could capture the essential capabilities of larger teachers through appropriate training strategies. Recent work by Zhou et al. (2023) on parameter-efficient fine-tuning methods like LoRA (Low-Rank Adaptation) has shown that targeted modifications to small subsets of model parameters can achieve significant performance improvements. This research direction aligns with our focus on efficient training approaches for specialized tasks. Supervised Fine-tuning and Domain Adaptation Supervised Fine-tuning (SFT) has emerged as a crucial technique for adapting pre-trained language models to specific domains and tasks. Ouyang et al. (2022) demonstrated the effectiveness of SFT in their work on InstructGPT, showing how human feedback and supervised training could significantly improve model alignment and task performance. The Hugging Face TRL (Transformer Reinforcement Learning) library, developed by von Werra et al. (2022), has democratized access to advanced fine-tuning techniques. Their SFTTrainer implementation provides robust infrastructure for supervised fine-tuning, handling dataset management, training loops, and evaluation metrics seamlessly. Domain-specific adaptation has been extensively studied across various fields. Rogers et al. (2020) provided comprehensive analysis of BERT adaptations, showing that domain-specific fine-tuning often outperforms general-purpose models. Similar findings have been reported in specialized domains like biomedical text processing (Lee et al., 2020) and legal document analysis (Chalkidis et al., 2020).