Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation
Yi-Chang Chen, Po-Chun Hsu, Chan-Jan Hsu, Da-shan Shiu · 2024
Read Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation by Yi-Chang Chen, Po-Chun Hsu, Chan-Jan Hsu, Da-shan Shiu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01130.txt.
Introduction
Yi-Chang Chen, Po-Chun Hsu, Chan-Jan Hsu, and Da-shan Shiu improve how language models call external functions in a 2024 paper. Function calling lets a model use tools on request, and doing it well in several languages is hard. The team tested prompt formats for describing functions and ways to blend tool data with general instruction data.
Mixing in instruction data improved both calling accuracy and the model's judgment about when a tool is relevant. A new Decision Token, paired with synthetic examples where no tool applies, sharpened that judgment further. Step-by-step reasoning helped as well. For other languages, their translation pipeline lifted performance markedly, with the clearest gains in Traditional Chinese. The authors frame the tricks as cheap wins for agent builders.
Catalog record
| CREATOR | Yi-Chang Chen, Po-Chun Hsu, Chan-Jan Hsu, Da-shan Shiu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01130
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation Yi-Chang Chen, Po-Chun Hsu, Chan-Jan Hsu, Da-shan Shiu Abstract Large language models (LLMs) have significantly advanced autonomous agents, particularly in zero-shot tool usage, also known as function calling. This research delves into enhancing the function-calling capabilities of LLMs by exploring different approaches, including prompt formats for integrating function descriptions, blending function-calling and instruction-following data, introducing a novel Decision Token for conditional prompts, leveraging chain-of-thought reasoning, and overcoming multilingual challenges with a translation pipeline. Our key findings and contributions are as follows: (1) Instruction-following data improves both function-calling accuracy and relevance detection. (2) The use of the newly proposed Decision Token, combined with synthetic non-function-call data, enhances relevance detection. (3) A tailored translation pipeline effectively overcomes multilingual limitations, demonstrating significant improvements in Traditional Chinese. These insights highlight the potential for improved function-calling capabilities and multilingual applications in LLMs. 1 Introduction The field of autonomous agents has seen remarkable advancements in recent years, largely driven by the capabilities of large language models (LLMs). These models have significantly enhanced the performance of autonomous agents across a variety of tasks Huang et al. 2024 ; Qin et al. 2024 ; Qu et al. 2024 . A critical ability for these agents is zero-shot tool usage, also known as function calling. This capability allows LLMs to access up-to-date information from the internet or in-house databases and leverage third-party services, enabling integration with various systems. Such capabilities open up numerous potential applications, including electronic design automation Zhong et al. 2023 , financial reporting Theuma and Shareghi 2024 , and travel planning Hao et al. 2024 . Despite the progress made through tuning-based methods Grattafiori et al. 2024 ; Liu et al. 2024a ; Liu et al. 2024b for enabling function-calling capabilities, there remains a gap in research regarding the format variance of prompts, the combination of function-calling data with instruction-following data, and multilingual limitations. This work aims to address these gaps by investigating the following aspects: Prompt Formats: We explore two strategies for incorporating function descriptions into prompts: (1) introducing a dedicated role for presenting function descriptions, and (2) embedding function descriptions within the system role alongside usage instructions. We aim to determine the impact of these formats on function-calling performance. Data Integration: We examine the combination of function-calling data with instruction-following data to assess its impact on both instruction-following and function-calling capabilities. Our findings indicate that the use of instruction-following data significantly enhances function-calling accuracy and relevance detection. Decision Token: We propose a novel Decision Token for conditional prompts, designed to improve relevance detection and facilitate the creation of synthetic non-function-call data for fine-tuning. Our results show that the inclusion of the Decision Token and non-function-call data enhances function-calling relevance detection. Chain-of-Thought (CoT) Reasoning: We incorporate CoT reasoning through a synthetic data pipeline that constructs reasoning descriptions from sequences of conversations and function calls. Multilingual Translation: We address the multilingual limitations of current function-calling models by introducing a translation pipeline specifically tailored to overcome the challenges of direct translation methods. Our Traditional Chinese experiments confirm this approach’s effectiveness. In summary, this research provides valuable insights into enhancing LLMs’ function-calling capabilities and highlights the potential for multilingual applications. The following sections detail our methodology, experiments, and results, demonstrating the effectiveness of our proposed strategies. Figure 1: An illustration of prompt templates used for function calling and instruction following in LLMs. During training, LLMs are given conditional prompts (shown on the left) and tasked with generating corresponding text completions (shown on the right). When a function call is required, the model generates structured function calls in the form of a list of functions, where each function is specified with its arguments in the format func_name(arg1=value1, ...) . Special tokens, including <s> , <|im_start|> , <|im_end|> , <|answer|> , and <|use_tool|> , are each represented by a single token after tokenization. For more details, refer to Section 3.1 . 2 Related Work Integrating function-calling capabilities into LLMs significantly broadens their problem-solving abilities by enabling interactions with external tools and APIs. Studies have shown that API-integrated LLMs can perform tasks such as programming assistance Gao et al. 2022 , real-time information retrieval Schick et al. 2023 , complex mathematical computations He-Yueya et al. 2023 , and internet utilization Komeili et al. 2021 ; Gur et al. 2024 . This allows LLMs to access up-to-date information and leverage third-party services, facilitating integration with various systems across advanced applications like electronic design automation Zhong et al. 2023 , financial reporting Theuma and Shareghi 2024 , and travel planning Hao et al. 2024 . To enable such function-calling capabilities, researchers have explored two main categories of methods. The first involves sophisticated prompting techniques. Frameworks like ReACT Yao et al. 2022 and its successors Xu et al. 2023 ; Shinn et al. 2023 ; Yang et al. 2023b ; Crouse et al. 2024 ; Wang et al. 2024 combine reasoning and acting within prompts to guide model responses. More closely related to our work, the second category focuses on training models to generate function calls through fine-tuning. Fine-tuned models such as Gorilla Patil et al. 2023 , ToolAlpaca Tang et al. 2023 , ToolLlama Qin et al. 2024 , and the Hermes 3 series by Nous-Research Teknium et al. 2024 enhance function-calling capabilities by relying on synthetic data generated by proprietary models like GPT-4 or ChatGPT. Open-source initiatives like NexusRaven-V2 Nexusflow.ai 2023 and IBM’s Granite-20B-FunctionCalling Abdelaziz et al. 2024 aim to develop function-calling models suitable for commercial use without relying on proprietary data. Moreover, many works involve self-supervision to further enhance performance across diverse domains Schick et al. 2023 ; Parisi et al. 2022 ; Yang et al. 2023a ; Liu et al. 2024a . Among the works in the second category, some fine-tuned models and datasets have been openly released. For instance, ToolAlpaca Tang et al. 2023 and ToolLLM Qin et al. 2024 have made available their synthetic data or data generation pipelines. ToolACE Liu et al. 2024a has released both the fine-tuned Llama model and the self-instruction dataset. Additionally, the Gorilla team developed a comprehensive benchmark to evaluate LLMs’ function-calling capabilities Yan et al. 2024 . Notably, ToolACE Liu et al. 2024a demonstrated that diversified function-calling sample data helps models learn better function-calling abilities. However, there is a lack of comprehensive analysis on how variations in prompt and meta-information design, as well as the impact of non-function-calling-related instruction tuning data, affect the effectiveness of function-calling capabilities. Existing studies tend to adopt specific prompt templates without extensively investigating the impact of different designs, indicating a need for further research in this area. 3 Methodology 3.1 Prompt Templates for Function Calling and Instruction Following We employ a tuning-based approach to enable both function-calling and instruction-following capabilities in our LLMs. This involves fine-tuning pretrained base models using prompt templates based on the Chat Markup Language (ChatML), a widely adopted format introduced by OpenAI. Two main strategies for incorporating function descriptions into prompts are explored: (1) introducing a dedicated role, such as tools , to represent function descriptions in JSON format (Figure 1 (b)); and (2) embedding function descriptions alongside usage instructions within the system role (Figure 1 (c)). In the