SynAE : A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti · 2026
Read SynAE : A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations by Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22564.txt.
Introduction
Wang and team launch SynAE, testing how well fake benchmarks for tool agents copy and augment real data paths. Tool agents are tested on static run traces, but in-house sets are sparse. Teams more and more use fake sets, raising the problem of sizing the fake-real link. SynAE tests truth, match, and range: prompts and replies; tool calls; final outputs; and downstream tests. On recent benchmarks, SynAE detects shifts in truth, match, range. It shows no single metric covers fake data quality, driving multi-axis checks.
Catalog record
| CREATOR | Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70940 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22564
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SynAE : A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations Shuaiqi Wang, Aadyaa Maddi, Zinan Lin, Giulia Fanti Abstract Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces , including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE , an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. A demo of SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html , with code at https://github.com/wsqwsq/SynAE . 1 Introduction Agent evaluation and testing is a nascent but critical component of pre-deployment processes for production agentic workflows [ 26 , 2 ] . Today, tool-calling agent evaluations are often (but not always) run on static baseline datasets 1 1 1 These evaluation datasets are also commonly referred to as benchmarks ; we use both terms interchangeably in this paper. consisting of a trace generated from agent interactions; these typically include user inputs, tool calls, intermediate interactions with the agent, and a final output [ 44 , 22 ] . Such datasets are often collected from real-user interactions with an environment and/or synthesized by measuring scripted interactions with a dynamic environment. While the design of proper agent evaluations is its own active research area [ 12 , 1 , 48 ] , common evaluation metrics measure whether the agent selects the correct tools and produces the desired final output [ 44 , 22 , 2 ] . In many practical situations, existing baseline datasets cannot be directly used or are insufficient for agent evaluation, either because they contain sensitive user data (e.g., emails, travel details) subject to privacy restrictions [ 31 , 3 ] or because they are too small for comprehensive testing [ 26 ] . Practitioners therefore increasingly use synthetic datasets as replacements for or augmentations to real execution traces in evaluation pipelines ( Figure 1(a) ) [ 28 , 32 , 9 ] . Synthetic data may be generated by directly synthesizing trajectories or by synthesizing inputs to an interactive environment. (a) Today, agents are commonly evaluated on synthetic datasets modeled after a real dataset of agent trajectories, e.g., for privacy or augmentation reasons. However, it is often unclear whether these synthetic datasets are representative of the real dataset. Our evaluation framework, SynAE , measures the semantic and structural similarity between real and synthetic datasets of execution trajectories, and the validity and diversity of the synthetic data. (b) The SynAE evaluation pipeline takes as input real and synthetic datasets (task completion trajectories), along with optional metric configurations that may specify data-specific prior knowledge and one or more LLM agents. The LLM agents are used only to compare downstream performance between real and synthetic input datasets. The framework evaluates synthetic data along three dimensions: validity, fidelity, and diversity. Figure 1 : The SynAE framework evaluates the quality of synthetic data used in agent evaluations. Although synthetic datasets are increasingly used for agent evaluation, typical workflows lack systematic quality checks against the real baseline data. Indeed, the current literature on testing agents with synthetic data provides almost no quantitative methods for evaluating the quality of such synthetic data, leaving operators with little visibility into evaluation gaps. In this work, we develop a comprehensive evaluation framework, SynAE , to assess how well synthetic trajectories replicate and augment the characteristics of real data trajectories, including task instructions and responses, and the associated reference tool calls and outputs. As illustrated in Figure 1(b) , SynAE takes as input a real baseline dataset, a synthetic dataset, and optional metric configurations that may specify data-specific prior knowledge and one or more LLM agents; the input agent(s) need not be the same as the one being evaluated—each agent input to SynAE is used purely to compare downstream performance between real and synthetic input datasets. SynAE evaluates synthetic data along three properties: (1) validity, (2) fidelity, and (3) diversity. For validity , we evaluate whether synthetic tool calls and outputs successfully fulfill the given instructions, using LLM-as-a-judge by default or rule-based checkers when available. We measure fidelity by assessing how similar the real and the synthetic data are; specifically, we compute metrics on both real and synthetic data and evaluate their similarity. We quantify the diversity of synthetic data with entropy-based metrics for various representations of the dataset. For each property, SynAE computes a suite of metrics to evaluate (a subset of): (1) task instructions and responses, (2) tool calls, (3) final outputs, and (4) downstream evaluation. We demonstrate the utility of SynAE with three recent agent benchmarks playing the role of the “real dataset” in Fig. 1(a) : T1 [ 4 ] , BFCL [ 27 ] , and ACP [ 13 ] . We then construct synthetic datasets using NVIDIA NeMo [ 23 ] , an industry-standard synthetic data tool, as well as custom synthetic data generation methods designed to simulate common real-world pitfalls, such as degraded data fidelity and limited diversity. Our experiments show that SynAE captures fine-grained variations in the validity, fidelity, and diversity of synthetic task-completion trajectories. They further suggest that no single metric is sufficient to fully characterize synthetic data quality, underscoring the need for SynAE . Overall, we view SynAE as a plug-in component for agentic workflows, allowing operators to automatically evaluate the quality of synthetic benchmark datasets. 1.1 Related Work We provide a brief overview of related works, and provide a detailed discussion in Appendix A . Several prior works evaluate synthetic benchmarks for LLMs using factors such as task difficulty, realism, and whether model rankings are preserved [ 7 , 18 , 38 , 19 ] . However, these works mainly focus on standard NLP settings rather than agent benchmarks, where evaluation is more difficult due to multi-step decision making, interaction with tools, and the need to attribute failures to different parts of the agent pipeline. Recent evaluations on synthetic benchmarks for tool-calling agents has largely focused on single-turn settings or instruction-level quality [ 30 , 9 , 48 , 1 , 26 ] , without systematically evaluating associated reference tool calls, outputs, or multi-step dependencies. Multi-turn tool-calling agents are commonly evaluated using end-to-end task success on state-based checkers [ 42 , 16 , 46 , 41 , 37 , 11 ] , response-based criteria [ 27 , 14 ,