APIGen-MT : A gentic PI peline for M ulti- T urn Data Gen eration via Simulated Agent-Human Interplay

Akshara Prabhakar Zuxin Liu Ming Zhu Jianguo Zhang Tulika Awalgaonkar · 2025

Read APIGen-MT : A gentic PI peline for M ulti- T urn Data Gen eration via Simulated Agent-Human Interplay by Akshara Prabhakar Zuxin Liu Ming Zhu Jianguo Zhang Tulika Awalgaonkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03601.txt.

Introduction

Akshara Prabhakar and co-authors build a factory for fake but verifiable conversations that train AI agents. Agents learn multi-turn work from example dialogues, and real ones are scarce and costly. Their 2025 framework, APIGen-MT, works in two phases. First, a committee of language models drafts and reviews detailed task blueprints, complete with the correct actions.

Simulated humans then act out those blueprints against the agent, producing full interaction trajectories. Models trained on the output — a family spanning small to large — beat flagship commercial systems on agent benchmarks, with the small models oddly strongest in long multi-turn work. The pipeline makes correctness checkable by construction. Agent builders get synthetic data with a ground truth spine.

Catalog record

CREATORAkshara Prabhakar Zuxin Liu Ming Zhu Jianguo Zhang Tulika Awalgaonkar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

APIGen-MT : A gentic PI peline for M ulti- T urn Data Gen eration via Simulated Agent-Human Interplay Thanks: Corresponding Authors Akshara Prabhakar Zuxin Liu Ming Zhu Jianguo Zhang Tulika Awalgaonkar Abstract Training effective AI agents for multi-turn interactions requires high-quality data that captures realistic human-agent dynamics, yet such data is scarce and expensive to collect manually. We introduce APIGen-MT , a two-phase framework that generates verifiable and diverse multi-turn agent data. In the first phase, our agentic pipeline produces detailed task blueprints with ground-truth actions, leveraging a committee of LLM reviewers and iterative feedback loops. These blueprints are then transformed into complete interaction trajectories through simulated human-agent interplay. We train a family of models—the xLAM-2-fc-r series with sizes ranging from 1B to 70B parameters. Our models outperform frontier models such as GPT-4o and Claude 3.5 on τ \tau -bench and BFCL benchmarks, with the smaller models surpassing their larger counterparts, particularly in multi-turn settings, while maintaining superior consistency across multiple trials. Comprehensive experiments demonstrate that our verified blueprint-to-details approach yields high-quality training data, enabling the development of more reliable, efficient, and capable agents. We open-source 5K synthetic data trajectories and the trained xLAM-2-fc-r models to advance research in AI agents. Model https://huggingface.co/Salesforce/xLAM-2 Dataset https://huggingface.co/Salesforce/APIGen-MT-5k Website https://apigen-mt.github.io 1 Introduction The growth of Large Language Model (LLM) agents has been accelerating at an unprecedented rate, driven by advancements in AI capabilities and increasing demand across various industries [ 21 , 1 , 10 , 5 , 30 , 54 , 23 , 7 , 25 ] . Their role has evolved beyond simple conversational chatbots to AI agents capable of executing real-world tasks, such as managing financial transactions, scheduling appointments, and handling customer service requests. These applications demand not only linguistic fluency but also precise execution, reliability, and adherence to domain-specific policies. Realistic enterprise use cases involve having an assistant (also referred to as agent in this document) that is capable of fluently conversing with humans of different personalities, incrementally understanding their intent, extracting the background details needed, accurately invoke APIs, and operate over a complex business logic structure. Figure 1 : Comparative performance of larger xLAM-2-fc-r models (8B-70B, trained with APIGen-MT data) against state-of-the-art baselines on function-calling (BFCL v3 [ 45 ] ) and agentic ( \tau -bench [ 49 ] ) capabilities. Despite their potential, building robust and reliable AI agents presents significant challenges [ 49 ] . Recent benchmarks reveal that even advanced LLMs struggle with multi-turn interactions, particularly when required to perform complex function calls, track long-term dependencies, or request missing information [ 50 , 28 , 46 , 45 , 19 ] . Although framework design and prompt engineering have shown promise, the underlying model capabilities remain the primary bottleneck, largely due to two fundamental obstacles: (1) the scarcity of high-quality agent interaction data in public pretraining corpora, and (2) the prohibitive cost and time required to manually collect and label such data, especially for domain-specific applications requiring specialized knowledge. Several approaches have attempted to address these challenges. APIGen [ 26 ] introduced techniques for generating single-turn function calling data, while [ 41 ] explored methods for knowledge distillation in agent training. However, these approaches primarily focus on single-turn interactions, failing to capture the complexity of real-world agent usage, where multiple turns are often required. Other efforts like [ 51 , 50 , 11 ] , while incorporating multi-turn aspects, lack human-agent interplay–crucial for realistic data generation. The verification and synthesis of high-quality multi-turn trajectories containing both linguistic diversity and grounded actions remains largely unsolved, creating a significant barrier to advancing agent capabilities. To address these limitations, we introduce APIGen-MT , an agentic data synthesis pipeline for generating high-quality multi-turn agent data. It operates in two main steps: first, a data agent generates a detailed and verified task "blueprint", and second, this blueprint guides the generation of realistic multi-turn interactions through simulated agent-human interplay ( Subsection 4.2 ). The blueprint generation includes sampling relevant APIs, policies, domain data, and user personas to create grounded general tasks configurations, and using reverse task recombination ( SubSubsection 4.1.3 ) to enhance complexity. These blueprints are validated through format/execution checks and an LLM committee review using a reflection-based mechanism [ 39 ] . Subsequently, the validated blueprint seeds a simulated interaction between a human LM and an agent (e.g., gpt-4o ), producing a complete interaction trajectory with dialogue, actions, and environment feedback for training. The main contributions of our work are summarized as follows: • We propose APIGen-MT , an agentic data synthesis pipeline that leverages environment execution feedback and a review committee to ensure the high-quality of generated multi-turn agent data. • We develop a two-phase framework that first creates detailed task blueprints with verifiable groundtruth actions, then transforms these blueprints into realistic multi-turn conversational agent trajectories with tool-usage through simulated human-agent interplay. • We train a series of models across multiple architectures and scales (Llama 3.1/3.2 and Qwen 2.5 at 1B to 70B parameters), demonstrating superior performance on two popular agentic benchmarks: \tau -bench and BFCL, surpassing many frontier models including gpt-4o ( Figure 1 ). • We open-source 5K high-quality synthetic data ( APIGen-MT-5k ) and trained models, i.e., the xLAM-2-fc-r series, to advance research in AI agent space. 2 Related Work Tool-Use Agents. Tool-use capabilities enhance LLMs by enabling interaction with external tools, extending their reasoning and functionality [ 44 , 33 , 24 ] . Function-calling frameworks allow LLMs to parse queries, select tools, and interpret results, but often require predefined tools, limiting adaptability [ 26 , 43 ] . Efforts were made to address this by creating reusable tools from scratch on the fly [ 9 ] , built upon by ToolMaker [ 44 ] which leverages tools from existing code repositories. Others compose workflows or learn from demonstrations [ 33 , 36 ] . Recently, several works have adopted specialized approaches for agent training—critique-informed planning [ 13 ] , fine-tuning on selective steps [ 48 ] , teasing apart reasoning from format following (Agent-FLAN) [ 12 ] , and autonomously invoking tools without explicit post-training (ToRL) [ 22 ] . Interactive Conversational Benchmarks. Evaluating LLM agents in multi-turn settings requires specialized benchmarks. MultiChallenge [ 40 ] and ToolDial [ 38 ] assess agents on context maintenance and tool-augmented dialogue. InterCode [ 47 ] and CRMArena [ 19 ] evaluate iterative problem-solving and customer management. ToolSandbox [ 28 ] provides a stateful, interactive benchmark for tool use. User simulations have become essential in these benchmarks, offering systematic, realistic interactions [ 49 , 28 , 31 ] . Our work complements these efforts by generating synthetic multi-turn conversations to train and evaluate agents in such realistic settings. Synthetic Data Generation. The scarcity of high-quality training data drives synthetic data generation. Multi-agent frameworks like MAG-V [ 37 ] , AgentInstruct [ 29 ] , MATRIX [ 42 ] , and IntellAgent [ 20 ] create realistic datasets by simulating agent interactions. Other approaches utilize instruction composition [ 18 , 11 ] , intermediate graphs [ 6 ] and multi-turn planning to produce complex dialogues [ 55 ] . Related to our effort in generating multi-turn training data, BUTTON [ 11 ] generates synthetic compositional instruction tuning

Read the complete text