Data Swarms : Optimizable Generation of Synthetic Evaluation Data

Shangbin Feng Yike Wang Weijia Shi Yulia Tsvetkov · 2025

Read Data Swarms : Optimizable Generation of Synthetic Evaluation Data by Shangbin Feng Yike Wang Weijia Shi Yulia Tsvetkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00741.txt.

Introduction

Shangbin Feng, Yike Wang, Weijia Shi, and Yulia Tsvetkov published this paper in 2025. It improves the synthetic test data used to grade language models. Hand-made benchmarks are costly and leak into training data, so generated tests are attractive, if their quality can be steered.

Data Swarms treats test generation as an optimization problem. A swarm of generator models starts from existing data, each scored on goals like difficulty and diversity. A particle swarm search breeds better generators by letting them share what works. An adversarial variant pits generators against a swarm of test-taking models, so the tests harden as the takers improve, and both sides evolve. Across five goals, the method beat eight rival generation approaches. It balanced several goals at once and produced tests that graded new models it had never seen. The authors offer the swarms as a renewable source of demanding evaluations.

Catalog record

CREATORShangbin Feng Yike Wang Weijia Shi Yulia Tsvetkov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data Swarms : Optimizable Generation of Synthetic Evaluation Data Shangbin Feng Yike Wang Weijia Shi Yulia Tsvetkov Abstract We propose Data Swarms , an algorithm to optimize the generation of synthetic evaluation data and advance quantitative desiderata of LLM evaluation. We first train a swarm of initial data generators using existing data, and define various evaluation objectives to reflect the desired properties of evaluation (e.g., generate more difficult problems for the evaluated models) and quantitatively evaluate data generators. We then employ particle swarm optimization to optimize the swarm of data generators, where they collaboratively search through the model parameter space to find new generators that advance these objectives. We further extend it to Adversarial Swarms , where the data generator swarm generates harder data while the test taker model swarm learns from such data, co-evolving dynamically for better data and models simultaneously. Extensive experiments demonstrate that Data Swarms outperforms eight data generation baselines across five evaluation objectives, while Adversarial Swarms produce more robust learning of synthetic data and stronger generalization. Further analysis reveals that Data Swarms successfully optimizes compositions of multiple evaluation objectives and generalizes to new off-the-shelf LLMs, unseen at optimization time. 1 Introduction With new LLMs emerging daily—each trained with different training recipes, domain expertise, and alignment strategies—rigorous evaluation is more critical than ever to reveal the strengths and weaknesses of LLMs, and guide model selection for diverse applications. The most common evaluation approach relies on static data, ranging from single-skill datasets [ 55 , 15 ] to comprehensive multi-task benchmarks [ 28 , 45 ] . As state-of-the-art models become saturated on these static datasets [ 12 ] with contamination concerns [ 13 ] , new evaluation methods focus on synthetic data generation for incorporating temporal updates [ 21 ] , adaptively probing model weakness [ 3 ] , leveraging privileged information [ 25 ] , and more. However, existing approaches for generating synthetic evaluation data are largely heuristic, relying on intuitive, trial-and-error engineering decisions, such as prompt design [ 48 ] , agent setup [ 24 ] , and selectively reporting successful outcomes as “best practices and lessons” [ 30 ] . It is important to systematize this process, and that the generated evaluation data achieves evaluation objectives [ 25 ] , for example, generating difficult evaluation data that exposes weaknesses, or better separating a pool of models to offer statistically significant signals about their strengths and weaknesses. While heuristic-based approaches could achieve these goals, they hardly generalize to new model capabilities and evaluation domains without another round of manual trial-and-error. This calls for a general, automated approach to synthetic evaluation data generation with well-defined evaluation objectives. In this work, we take an optimization view towards evaluation data . We propose Data Swarms , optimizing a swarm of data generator models towards quantitative objectives with swarm intelligence. Particle swarm optimization (PSO) [ 22 ] is an algorithm that optimizes a swarm of continuous representations for a utility function, guided by particle- and swarm-level utility signals (§ 2 ). Guided by its initial success in multi-LLM collaboration [ 9 , 10 ] , we employ PSO to optimize data-generator LMs. Specifically, given a static seed dataset, we train an initial swarm of data generators with self-instruct [ 48 ] . For each iteration, we generate evaluation instances with each data generator, evaluate test taker models on them, calculate the evaluation objective metrics (e.g., difficult : one minus the max performance of test takers), and employ PSO to optimize the data generator swarm guided by the metrics. This iterative process produces optimized data generators and new evaluation data characterized by specific evaluation objectives, and we propose a suite of five objectives— difficult, separate, novel, consistent, personalized —as evaluation data characteristics to optimize towards. We also propose Adversarial Swarms , an extended algorithm in which a data generator swarm and a test taker swarm co-evolve in an adversarial setting with two steps. In data-step , the data swarm takes a PSO optimization step as previously described with the current snapshot of test takers, aiming to generate harder data; In model-step , the test taker swarm takes a PSO optimization step guided by their performance on this iteration’s generated data. By alternating between the data and model steps, Adversarial Swarms jointly enhances both synthetic data generation and model capabilities. Extensive experiments with four evaluation domains and eight baselines demonstrate that Data Swarms advances the five evaluation objectives, outperforming baselines across four data domains. Adversarial Swarms further improves model performance on a held-out set, outperforming fine-tuning with either static or generated data by 8.2% on average. Empirical analyses reveal that Data Swarms generalizes to unseen test takers, works for tasks with verifiable ground truths, and successfully optimizes compositions of evaluation objectives. In sum, Data Swarms uniquely offers an optimization angle to quantitatively measure and improve synthetic evaluation data—enabling the creation of harder dynamic benchmarks, supporting the evaluation of emerging model capabilities, and reducing risks of data contamination and model exposure. Figure 1: Overview of Data Swarms . (Left) Given a data generator, we sample evaluation problems, evaluate test taker LLMs on them, and calculate objectives. (§ 2.2 ) (Center) We optimize multiple data generators with particle swarm optimization to adapt them to evaluation objectives. Specifically, models are impacted by the personal/global best/worst checkpoints to update velocity and explore model weight space for optimization. (Right) In Adversarial Swarms , data generators and test takers co-evolve adversarially to produce better data and models simultaneously. 2 Methodology We propose Data Swarms , an algorithm to optimize synthetic evaluation data generation for diverse evaluation objectives (e.g., generating more difficult evaluation examples). Given seed data \mathcal{D} and a pool of test taker models \{\mathbf{m}_{i}\}_{i=1}^{n} , we train an initial swarm of data generators (§ 2.1 ), define a suite of evaluation objectives (§ 2.2 ), and optimize the data generators with particle swarm intelligence (PSO) towards these objectives (§ 2.3 ). Data Swarms transcends the heuristics-driven status quo in synthetic data generation by introducing an optimization-based framework that enables the creation of challenging evaluation data, consistent evaluation, and novel insights into model behavior. 2.1 Training Initial Data Generators While existing datasets are often offered as a monolithic resource, there is inherent diversity within any dataset [ 33 ] : different instances test different variations and mixtures of model capabilities, forming clusters and taxonomies of sub-evaluations [ 56 ] . Given seed data \mathcal{D} in an existing dataset, we propose to train a swarm of data generators reflecting different sub-evaluations within \mathcal{D} through clustering. By having a pool of multiple data generators, PSO could enable their collaborative search in the model weight space to optimize synthetic data generation (§ 2.3 ). Concretely, we first cluster \mathcal{D} into N subsets (\mathcal{D}_{1},\cdots,\mathcal{D}_{N})=\mathrm{cluster}(\mathcal{D},N) with K-means over average-pooled query embeddings. For each \mathcal{D}_{i} , we employ Self-Instruct [ 48 ] to train a data generator \mathbf{x}_{i} to model this distinct cluster of queries: specifically, we randomly sample (\mathbf{d}_{1},\cdots,\mathbf{d}_{2k})\sim\mathcal{D}_{i} and employ the following format: “You are an expert in generating synthetic evaluation data, specifically about <domain>. You are given a set of k examples. Please follow the pattern and generate k more examples. Examples: \mathbf{d}_{1},\cdots,\mathbf{d}_{k},\underline{\mathbf{d}_{k+1},\cdots,\mathbf{d}_{2k}} ” . We then fine-tune an off-the-shelf

Read the complete text