Less is More: Adaptive Coverage for Synthetic Training Data
Sasan Tavakkol, Max Springer, MohammadHossein Bateni, Vincent Cohen-Addad, Neslihan Bulut, MohammadTaghi Hajiaghayi · 2025
Read Less is More: Adaptive Coverage for Synthetic Training Data by Sasan Tavakkol, Max Springer, MohammadHossein Bateni, Vincent Cohen-Addad, Neslihan Bulut, MohammadTaghi Hajiaghayi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14508.txt.
Introduction
Six authors, led by Sasan Tavakkol, wrote this 2025 paper on picking the best subset of synthetic training data. Large language models can now write labeled training sets on demand. That power helps when a classifier must be built fast, for fast-moving trends or new forms of online abuse.
Past work asked whether synthetic data can match human-labeled data. This team asks a new question: which subset of a synthetic set should be kept? Their sampling method rests on the maximum coverage problem, a classic way to pick items that cover the most ground. Classifiers trained on the picked subset beat those trained on the full set. Less data gave better results, as the title says. The approach also cuts training cost while it raises accuracy.
Catalog record
| CREATOR | Sasan Tavakkol, Max Springer, MohammadHossein Bateni, Vincent Cohen-Addad, Neslihan Bulut, MohammadTaghi Hajiaghayi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14508
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Less is More: Adaptive Coverage for Synthetic Training Data Sasan Tavakkol, Max Springer, MohammadHossein Bateni, Vincent Cohen-Addad, Neslihan Bulut, MohammadTaghi Hajiaghayi Abstract Synthetic training data generation with Large Language Models (LLMs) like Google’s Gemma and OpenAI’s GPT offer a promising solution to the challenge of obtaining large, labeled datasets for training classifiers. When rapid model deployment is critical, such as in classifying emerging social media trends or combating new forms of online abuse tied to current events, the ability to generate training data is invaluable. While prior research has examined the comparability of synthetic data to human-labeled data, this study introduces a novel sampling algorithm, based on the maximum coverage problem, to select a representative subset from a synthetically generated dataset. Our results demonstrate that training a classifier on this contextually sampled subset achieves superior performance compared to training on the entire dataset. This “less is more” approach not only improves model accuracy but also reduces the volume of data required, leading to potentially more efficient model fine-tuning. 1 Introduction In recent years, the remarkable advancement in large language models (LLMs) such as OpenAI’s GPT [ 1 ] or Google’s Gemma [ 51 ] have dramatically expanded the capability to generate extensive synthetic textual data. Such synthetic data promises substantial utility for training machine learning models, especially in domains where human-labeled data are prohibitively costly, inaccessible due to privacy or ethical constraints, or impractical to acquire at scale [ 4 , 11 ] . Consequently, synthetic data generation has quickly become an appealing alternative for tuning models for various downstream tasks, including text classification, sentiment analysis, relation extraction, and information retrieval [ 36 ] . However, the mere abundance of synthetic data does not guarantee superior model performance. Increasing evidence demonstrates that naively utilizing large synthetic datasets introduces critical pitfalls: notably, redundancy and imbalance [ 15 , 31 , 34 ] . LLM-generated samples frequently exhibit redundancy by over-representing certain common patterns or phrases, potentially saturating datasets with semantically repetitive information. Consider hate speech detection, where nuanced distinctions between offensive, sarcastic, or context-dependent language are crucial: when prompted to generate training examples, an LLM may produce many straightforwardly toxic utterances, yet underrepresent borderline, coded, or indirect forms of harm [ 15 , 21 ] . Such skewed representation not only dilutes the informative value of synthetic datasets but actively harms model generalization and robustness by obscuring valuable minority cases. Consequently, models trained on these synthetic corpora risk becoming overly specialized on frequent cases, compromising predictive accuracy on more nuanced real-world scenarios. Motivated by these gaps, we propose A daptive C overage S ampling (ACS), a novel method that effectively curates synthetic text datasets by carefully balancing redundancy, representational diversity, and computational efficiency. ACS uniquely frames synthetic data downsampling as a structured maximum-coverage optimization problem defined over a graph representation of the data. Specifically, synthetic text samples are first embedded into a latent semantic space, forming nodes within a complete graph where edges represent semantic similarity. Our approach leverages a binary search to systematically determine the optimal similarity threshold for edge pruning, thus inducing a sparser subgraph. Subsequently, a greedy maximum-coverage approximation algorithm selects the subset of k samples maximizing representational coverage, where coverage is defined as the proportion of the dataset “covered” by the selected subset and its similarity-neighbors. Figure 1: Overview of the ACS pipeline. (1) Prompt an LLM to generate a large pool of synthetic samples under user-specified constraints. (2) Samples are embedded into a semantic space and connected into a complete, weighted similarity graph. (3) Perform a binary search over edge-weight threshold to induce a subgraph. (4) Greedy max-cover procedure then iteratively selects the k nodes (highlighted in dark blue) that together cover desired fraction of the remaining graph (uncovered nodes depicted as white). (5) Selected subset is returned for downstream model training. A key strength of ACS lies in its use of a theoretically grounded binary search procedure to tune the pruning threshold, automating the trade-off between dataset compactness and semantic coverage. This allows the method to systematically filter out repetitive or redundant samples that might otherwise hinder model performance. We evaluate ACS across several NLP tasks—including sentiment classification, relation extraction, and named entity recognition—and find that models fine-tuned on ACS-selected subsets match or outperform those trained on full synthetic datasets typically with just 10–30% of the original corpus. These results highlight the promise of principled data selection in synthetic data regimes: by identifying compact yet diverse training sets, ACS improves generalization while significantly reducing training compute cost. In doing so, our approach moves beyond heuristic-driven methods, offering a scalable and theoretically informed path toward more effective use of synthetic data. 2 Related Work Large Language Models. LLMs, built upon the transformer architecture introduced by [ 53 ] , have transformed language processing, achieving unprecedented performance across a broad spectrum of tasks including language modeling, translation, classification, and question-answering [ 3 , 40 , 50 ] . These models leverage massive-scale pretraining on extensive datasets to encode rich linguistic and factual knowledge, enabling fluent and contextually relevant text generation [ 51 ] . Consequently, the sophistication of LLM-generated content increasingly blurs the line between synthetic and authentic human-written text [ 22 , 43 , 49 , 56 ] . This indistinguishability raises an intriguing question: under what conditions and to what extent can LLM-generated data replace or complement human-annotated examples for training machine learning models? Synthetic Training Data Generation. High-quality datasets crucially underpin the performance and generalization capabilities of modern machine learning systems. However, acquiring diverse and representative labeled data from human annotators is frequently costly, labor intensive, and fraught with privacy or ethical challenges [ 27 , 16 , 25 , 45 ] . Moreover, human-generated annotations inherently carry biases or inconsistencies, potentially limiting their effectiveness in certain contexts. To overcome these limitations, synthetic data generation has emerged as a promising alternative, aimed at artificially populating underrepresented data regions and mitigating biases or gaps in existing datasets [ 15 , 31 ] . To address data scarcity in specialized or emerging domains, researchers frequently employ data augmentation techniques to enhance model robustness and accuracy [ 10 , 54 ] . Moreover, semi-supervised learning [ 37 ] , multi-task learning [ 17 ] , unsupervised pretraining [ 9 , 41 ] , and few-shot learning [ 8 , 23 ] constitute alternative frameworks for learning from limited labeled examples. However, while effective in certain contexts, these approaches typically presume access to at least some high-quality human-generated examples as seed data, limiting their broader applicability. Leveraging LLMs for Synthetic Data. LLMs offer a compelling approach to synthetic data generation due to their fluency, versatility, and capacity to mimic diverse linguistic styles and content structures [ 11 ] . Recent studies have demonstrated promising outcomes leveraging prompt-based methods (zero and few-shot) for generating training data for NLP tasks [ 34 ] . The effectiveness of synthetic datasets produced by these models depends critically on task characteristics, including the complexity of label spaces [ 11 ] , the inherent subjectivity or ambiguity of the task [ 30 ] , and crucially, the diversity and representativeness of generated samples [ 21 ] . Though the models are promising, these factors