RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs

Vibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma Cheng Chen , Tracy Holloway King , Michael Friedrich Adobe Inc.345 Park AvenueSan Jose, CA, 95110 · 2025

Read RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs by Vibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma Cheng Chen , Tracy Holloway King , Michael Friedrich Adobe Inc.345 Park AvenueSan Jose, CA, 95110 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10495.txt.

Introduction

Vibha Belavadi and seven co-authors make made up train data for language models that call soft ware tasks. In tools to make art and text, user asks must map to API calls. But rules on what is kept pried and lack of logs bar train on real chats. Past made up tools lack the range and hard points of real use. Models tuned on them score less than best.

Their build, RouteNator, routes builds through field stores. Those stores are facts on what is held, graphs of facts with form, and models for text and for sight. The route aims to match how real user asks spread out. On a full set of real asks, models tuned with the made up data won gains. They won on task sort truth and on pick of traits next to old ways. The team claims new top marks for task calls.

Catalog record

CREATORVibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma Cheng Chen , Tracy Holloway King , Michael Friedrich Adobe Inc.345 Park AvenueSan Jose, CA, 95110
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs Vibha Belavadi, Tushar Vatsa, Dewang Sultania, Suhas Suresha, Ishita Verma Cheng Chen , Tracy Holloway King , Michael Friedrich Adobe Inc.345 Park AvenueSan Jose, CA, 95110 Abstract This paper addresses fine-tuning Large Language Models (LLMs) for function calling tasks when real user interaction data is unavailable. In digital content creation tools, where users express their needs through natural language queries that must be mapped to API calls, the lack of real-world task-specific data and privacy constraints for training on it necessitate synthetic data generation. Existing approaches to synthetic data generation fall short in diversity and complexity, failing to replicate real-world data distributions and leading to suboptimal performance after LLM fine-tuning. We present a novel router-based architecture that leverages domain resources like content metadata and structured knowledge graphs, along with text-to-text and vision-to-text language models to generate high-quality synthetic training data. Our architecture’s flexible routing mechanism enables synthetic data generation that matches observed real-world distributions, addressing a fundamental limitation of traditional approaches. Evaluation on a comprehensive set of real user queries demonstrates significant improvements in both function classification accuracy and API parameter selection. Models fine-tuned with our synthetic data consistently outperform traditional approaches, establishing new benchmarks for function calling tasks. † † ∗ These authors contributed equally to this work. 1 Introduction Digital content creation platforms increasingly rely on natural language interfaces to make complex design tools accessible to non-technical users. A critical challenge lies in accurately translating user queries into appropriate function calls Schick et al. 2023 for instance, when a user requests “Find me an image of an elephant with the background being Taj Mahal”, the system must orchestrate multiple API calls for searching, background removal, and compositing. In this paper, we specifically address the challenge of training models to classify user intent into two distinct categories: queries that can be fulfilled through search API operations versus those requiring generation through Generative-AI-powered APIs. Given a user query, our model determines: (1) whether to route the request to “Search” or “Generate” operations based on user intent, (2) the appropriate Content Type parameter selection (e.g. Photo, Template, Background, Video), and (3) prompt optimization specific to each API type—simplifying “Search” queries while preserving detailed specifications for “Generate” API requests. The examples below in Listing 1 demonstrate how our model processes and classifies different types of user queries: Listing 1: Example function calls for user queries ⬇ example1 = { ’ input ’: ’ Find me an image of an elephant ’, ’ output ’: { ’ function ’: ’ Search ’, ’ content_type ’: ’ Photo ’, ’ extracted_prompt ’: ’ elephant ’ } }, example2 = { ’ input ’: ’ Create a birthday invitation for my nephew whose birthday is on January 21’, ’ output ’: { ’ function ’: ’ Generate ’, ’ content_type ’: ’ Template ’, ’ extracted_prompt ’: ’ invitation for nephew ’ s birthday on January 21’ } } Figure 1: Data generation architecture overview integrating metadata, knowledge graph, and visual content. A “Weighted Router” directs text and image inputs to different prompt categories: length-based, API-based, and media type. They are processed by Text-to-Text and Vision-to-Text LLMs to generate synthetic data for downstream tasks. While existing function calling models Patil et al. 2023 show promise, their performance on specialized domains remains suboptimal and privacy restrictions on production data create training challenges. To address these limitations, we present two key contributions: 1. A methodology for incorporating structured domain knowledge into synthetic data generation that leverages: (a) Techniques for extracting generalizable patterns from content metadata; (b) Methods for utilizing domain-specific knowledge graphs to generate contextually relevant queries. 2. A novel router-based architecture for synthetic data generation featuring: (a) Multiple specialized LLM prompt templates as distinct routes; (b) A weighted routing mechanism using population-level statistics; (c) Integration of multi-modal language models to increase data diversity. Our approach (Figure 1 ) improves downstream model performance while producing balanced training data across content types, with well-distributed keywords and diverse sentence structures that better align with real-world user interactions. 2 Related Work The challenge of generating high-quality synthetic training data for language models has been explored through various approaches. Prior work in generating high-quality synthetic training data for language models spans three key categories: Instruction-tuning approaches have shown significant promise, starting with Self-Instruct’s Wang et al. 2023 175-seed task framework. The field expanded through WizardLM’s EvolInstruct Xu et al. 2023 , Unnatural Instructions Honovich et al. 2022 , FLAN Wei et al. 2022 , FLAN-T5 Chung et al. 2024 , Alpaca Taori et al. 2023 , Prompt-Breeder Fernando et al. 2023 , and Template-based Generation Gholami and Omar 2023 . Multimodal synthetic data generation advanced through Visual Instruction Tuning Liu et al. 2023 , MiniGPT-4 Zhu et al. 2023 , and InternVL Chen et al. 2024 , incorporating visual and textual information for enhanced data generation. Function calling approaches, exemplified by Gorilla Patil et al. 2023 building on Self-Instruct Wang et al. 2023 , addressed API parameter matching challenges, though lacking domain-specific knowledge integration. Our work differs from previous approaches in several key aspects. (1) We focus specifically on generating synthetic data for function calling while maintaining real-world query distributions. Unlike general instruction tuning approaches, we target the unique challenges of function calling data generation which include precise parameter matching requirements, maintaining real-world API usage distributions and handling complex nested function calls. (2) We introduce a novel router-based architecture that combines multiple generation strategies. We extend existing router-based approaches by adding weighted probabilistic sampling and by using population-level statistics to guide routing decisions. We also combine text-to-text and vision-to-text generation paths. (3) We incorporate domain-specific knowledge while respecting privacy constraints by not directly referring to the real-world datasets. (4) We leverage multiple modalities (text and images) to increase the diversity and quality of generated data, particularly for visual content-related APIs. We introduce novel evaluation metrics measuring the qualitative alignment of the synthetic data with real-world data covering content-type alignment, diversity in data types generated, word length variability, and positional variance of key terms within sentences. 3 Methodology and Experiments This section details how our data generation approach evolved. 3.1 Template-based Heuristic Generation Figure 2: Knowledge Graph of concepts linked by edges Our initial approach employed rule-based templates that combined content metadata with domain-specific Knowledge Graph (KG) relationships Kumar et al. 2023 between different aspects of digital content creation. Consider a snapshot of the Knowledge Graph sub-graph (Figure 2 ). This sub-graph consists of interconnected nodes representing User Intents ("Birthday", "Diwali", "Baby shower"), Design Types ("Card", "Invite"), Scene Objects ("Cake"), and associated Actions ("Cut a cake"). Each edge between the nodes represents a relationship between them. These connections were created using the historical and semantic relationships seen between different entities like User Intents, Design Types, Scene Objects and Actions. They enable the generation of semantically coherent queries by following established relationships between concepts. We create synthetic “Search” API data by generating random prompts combining the related entities (e.g. Intent and Design Type) with search synonyms e.g. "find me", "search for", "look for", "search", "show me". Listing 2: Examples of Image and Template metadata ⬇ image_asset_metadata = { ...

Read the complete text