Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, Karthik Vaidhyanathan · 2025
Read Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling by Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, Karthik Vaidhyanathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19277.txt.
Introduction
Ishan Kavathekar and co-authors ask whether small language models can handle function calling, the skill that lets an assistant turn a request into structured commands for software. Their 2025 empirical study tests small models across domains with zero-shot prompts, few-shot examples, and fine-tuning, then runs the survivors on an actual edge device to measure latency and memory.
Fine-tuning won clearly, few-shot helped, and zero-shot lagged. The persistent weakness was format discipline: small models often knew the right call but wrote it in the wrong shape. Prompt injection attacks barely dented them. The team releases the fine-tuned models for reuse. The stakes are practical: if a small model can call functions reliably on-device, assistants escape the data center's cost and connectivity demands. This study maps how close that escape is, and names the formatting gap as the wall still standing.
Catalog record
| CREATOR | Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, Karthik Vaidhyanathan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19277
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, Karthik Vaidhyanathan Abstract . Function calling is a complex task with widespread applications in domains such as information retrieval, software engineering and automation. For example, a query to book the shortest flight from New York to London on January 15 requires identifying the correct parameters to generate accurate function calls. Large Language Models (LLMs) can automate this process but are computationally expensive and impractical in resource-constrained settings. In contrast, Small Language Models (SLMs) can operate efficiently, offering faster response times, and lower computational demands, making them potential candidates for function calling on edge devices. In this exploratory empirical study, we evaluate the efficacy of SLMs in generating function calls across diverse domains using zero-shot, few-shot, and fine-tuning approaches, both with and without prompt injection, while also providing the finetuned models to facilitate future applications. Furthermore, we analyze the model responses across a range of metrics, capturing various aspects of function call generation. Additionally, we perform experiments on an edge device to evaluate their performance in terms of latency and memory usage, providing useful insights into their practical applicability. Our findings show that while SLMs improve from zero-shot to few-shot and perform best with fine-tuning, they struggle significantly with adhering to the given output format. Prompt injection experiments further indicate that the models are generally robust and exhibit only a slight decline in performance. While SLMs demonstrate potential for the function call generation task, our results also highlight areas that need further refinement for real-time functioning. Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Trkiye DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence CCS: General and reference Empirical studies 1. Introduction Advancements in LLMs have enabled strong performance across a wide range of tasks ( Kojima et al., 2022 ; Shanahan, 2024 ; Zhao et al., 2023 ) . They are increasingly being adopted in software engineering for tasks such as code generation ( Lu et al., 2021 ; Ahmad et al., 2021 ) , debugging ( Tian et al., 2024 ) , and API integration ( Zhang et al., 2023 ) , as well as in systems for task automation. One critical application for LLMs is function calling and execution. LLMs often rely on extensive cloud-based infrastructure, raising concerns about privacy, latency, and high computational overhead ( Yan et al., 2024a ; Chen et al., 2025 ) . As a result, SLMs are emerging as a powerful alternative for real-world applications that can efficiently perform specific tasks locally, making them ideal for environments where responsiveness and data security are critical ( Wang et al., 2024b ; Nguyen et al., 2024 ) . SLMs can potentially serve as a bridge between user inputs and backend systems in function calling applications, by converting natural language queries into structured function calls ( Wang et al., 2024b ) . Function calling, though less common than NLP tasks like summarization or translation, is crucial for interpreting and executing user commands. It enables SLMs to perform real-world actions, making it essential in industries requiring timely, accurate responses, such as healthcare, finance, automotive systems, telecom, and smart home automation. However, this intersection of function calling with SLMs is relatively an unexplored and untapped space. To this end, we conduct an exploratory empirical study on the capability of SLMs to generate function calls. We systematically select 5 SLMs based on a coding capabilities evaluation benchmark and on the number of parameters present in the model and evaluate them on a comprehensive dataset of 60,000 samples spanning diverse functional domains such as technology, entertainment, finance, and more. We evaluate the SLMs using three inference strategies: zero-shot, few-shot, and finetuning and assess their performance across different metrics which capture multiple dimensions of function call generation. Our metrics assess syntactic correctness, semantic accuracy, and the models’ ability to generate structured outputs across various scenarios as well. Additionally, we conduct prompt injection experiments to assess the models’ robustness to minor perturbations in the user prompt. Furthermore, we conduct experiments on an edge device to examine the models’ practical applicability in constrained computational environments and analyze the relationship between the memory footprint of the model and latency. In summary, our contributions include: (i) an analysis of SLMs with various prompting techniques, including prompt injection, for function calling, (ii) finetuned SLMs with performance analysis, highlighting areas for improvement, and (iii) experiments on edge devices to assess efficiency. Our findings indicate that certain SLMs demonstrate promising capabilities across various approaches. Additionally, our analysis reveals interesting insights that open new avenues for future research. We provide the finetuned models to support further research and applications. The replication package for our study is here 1 1 1 https://github.com/Raghav010/Small-Models-Big-Tasks . Figure 1. Overview of the study design illustrating different components of the research methodology. 2. Background 2.1. Small Language Models (SLMs) SLMs are generative models capable of understanding, processing, and generating text corresponding to the user prompt. These models operate on a much smaller scale compared to LLMs. In contrast to billions of parameters of LLMs, SLMs use significantly less parameters ranging from millions to a few billion. This lower memory and computational requirement leads to the model’s improved efficiency, accessibility, better customizability, and faster inference time. These characteristics make SLMs a better alternative for domain-specific, constrained, and low resource setting compared to LLMs. 2.2. Zero-shot, Few-shot, and Finetuning approaches for inference Zero-shot approach involves leveraging the pre-training of the model and prompting it directly without any other task information. In contrast, few-shot approach involves providing one or more examples of task-answer responses in the prompt itself. The model uses these examples to understand the task and generate the desired output ( Song et al., 2023 ) . Since these examples are provided into the model’s context, few-shot approach is also known as in-context learning. The number of examples to be provided in the prompt differs according to the task. This choice is also driven by the length of one example and the context size of the model. Zero-shot and few-shot inferencing do not require additional training of the model, making them computationally efficient techniques. Finetuning refers to the retraining of the model on task-specific data to improve it’s performance on the task. However, finetuning is computationally more expensive compared to zero-shot and few-shot approaches due to the need for additional training, which involves updating model parameters, and increased memory usage. To address this, techniques like LoRA ( Hu et al., 2021 ) , prompt tuning ( Lester et al., 2021 ) and prefix tuning ( Li and Liang, 2021 ) are employed. These methods reduce the computational cost significantly, while giving almost the same performance. 2.3. GGUF models GGUF ( Gerganov, 2023 ) is a storage format used for storing and running quantized LLMs efficiently on compute constrained devices. Quantization is a compression process that converts the weights of a model from high-precision representation to low-data-precision representation to reduce memory usage and improve computational efficiency ( Dettmers et al., 2022 ) . This format supports models with varying sizes and quantization levels,