MetaSynth : Meta–Prompting–Driven Agentic Scaffolds for Diverse Synthetic Data Generation

Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, Graham Horwood · 2025

Read MetaSynth : Meta–Prompting–Driven Agentic Scaffolds for Diverse Synthetic Data Generation by Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, Graham Horwood (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12563.txt.

Introduction

Haris Riaz and co-authors present MetaSynth, a way to generate training data for language models that is varied enough to teach new specialties. Small modern models are raised largely on synthetic text written by bigger models. That text tends to be monotonous, which limits what students can learn from it, especially in narrow domains.

MetaSynth has one model direct a team of expert models, each prompted to write from its own angle. Diversity is designed in rather than hoped for. With only twenty-five million tokens of this data, the team adapted a strong open model to finance and to biomedicine. General ability survived the specialization. The recipe offers a cheaper path to domain-tuned models.

Catalog record

CREATORHaris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, Graham Horwood
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MetaSynth : Meta–Prompting–Driven Agentic Scaffolds for Diverse Synthetic Data Generation Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, Graham Horwood Abstract Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data generated using larger Language models. Questions remain about leveraging synthetic data for other use cases, such as adapting LLMs to specific domains. A key limitation of synthetic data is low diversity , which negatively impacts its downstream applicability for improving other models. To address this, we propose MetaSynth , a method for generating synthetic data that enhances diversity through meta-prompting, where a language model orchestrates multiple “expert” LLM agents to collaboratively generate data. Using only 25 million tokens of synthetic data generated with MetaSynth , we successfully adapt a well-trained LLM (Mistral-7B-v0.3) to two specialized domains–Finance and Biomedicine–without compromising the capabilities of the resulting model in general tasks. In addition, we evaluate the diversity of our synthetic data using seven automated metrics, and find that it approaches the diversity of LLM pre-training corpora. Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in Biomedicine. The same model shows degraded performance when trained on data generated using a template prompt, even when the template includes prior generations and varying In-Context exemplars of real data. Our findings suggest that a few million tokens of diverse synthetic data without mixing any real data, is sufficient for effective domain adaptation when using MetaSynth. † † footnotetext: This work was done while Haris Riaz was an intern at Amazon Web Services. * * footnotetext: Equal contribution. 1 Introduction Human generated public text data cannot sustain the continued scaling and expansion of large language models (LLMs). It has been argued by Villalobos et al. (2024) that the available stock of public human text data will be fully utilized by 2028 if current LLM development trends continue, or earlier if LLMs are trained on more data than is compute optimal. This is evidenced in e.g., Llama 3 Grattafiori et al. (2024) which uses one order of magnitude more data compared to only two year old estimates of compute optimal large language models Hoffmann et al. (2022) . As a potential remedy, synthetic data generated with LLMs has shown remarkable potential to alleviate the impending issue of data scarcity for future model scaling. However, low diversity is a key issue in any type of synthetic data. In this work, we hypothesize that there are two prominent reasons that affect the diversity of data synthesized by LLMs: a) the choice of seed instances used to initialize data generation and b) the prompts used, which commonly follow predefined templates, where variation in the prompt is mainly introduced via placeholders whose content is populated dynamically. Examples of data generation methods which use template-like prompts with variation include: Self-prompting Li et al. (2024) , Attrprompt Yu et al. (2023) , CLINGEN Xu et al. (2025) and Explore-Instruct Wan et al. (2023) , among others. We contend that this variation yields limited diversity. For instance, generating a collection of domain-specific (e.g., financial) texts with similar prompts results in repetitive sentence structures—many texts begin with lexical patterns such as ‘‘In today’s ever-changing financial landscape’’ or ‘‘as the financial world evolves’’ —and often contain recurring phrases, and generic buzzwords. Recently, Suzgun and Kalai (2024) find that Meta-prompting Zhang et al. (2024) approaches – where an LLM itself writes the prompts to solve a problem – can elicit more diverse and creative outputs, significantly improving problem-solving capabilities for mathematical and algorithmic reasoning tasks, largely due to the feedback, self-verification, chain-of-thought Wei et al. (2023) , and planning dynamics inherent in these approaches. It has also been shown that using an optimized meta-prompt can improve the quality and downstream effectiveness of LLM generated synthetic data Kim et al. (2024) . We argue that a key use case for synthetic data arises when abundant real data exists in the form of pre-training corpora, but one wishes to effectively tailor an LLM to a specific domain using only a small amount of carefully generated synthetic data Arannil et al. (2024) . In this work, we investigate data-efficient domain adaptation through meta-prompting, where a language model is instructed to act as a supervisor that writes specialized prompts for other models to collaboratively generate diverse data. Our contributions are as follows: (1) We propose MetaSynth , a method to create diverse synthetic documents for continual pre-training (CPT) by leveraging a meta language model (which we refer to as meta-LM) and Conditional Instance Generation – where the meta-LM categorizes, and keeps track of each generated instance in memory, to ensure distinctness between them. (2) We propose MetaSynth- Instruct , which can generate and iteratively evolve complex instructions for instruction pre-training . Notably, this evolution is entirely driven by prompts written by the meta-LM itself. Furthermore, unlike other instruction-pretraining approaches e.g., ( Cheng et al. (2024a) , Cheng et al. (2024b) ) our instructions are purely evolved from contexts synthesized by MetaSynth i.e., without using any human-written text (section 3.3 ). (3) MetaSynth- Instruct can also synthesize training data for fine-tuning encoder models such as BERT Devlin et al. (2019) , RoBERTa Liu et al. (2019) etc. We observe that encoders fine-tuned on this data can outpeform those fine-tuned on data generated with template-based prompting (section 6 ). (4) We generate synthetic documents for continual pre-training by prompting an LLM with its prior outputs (memory) where the prompt follows a predefined template containing in-context exemplars of real data. However, when these synthetic documents are mixed with real data in a 1:1 ratio over 25M tokens, we find that it does not improve the Mistral-7B base model, and even leads to slight performance degradation across two domains. In contrast, using 25M tokens of diverse synthetic data from MetaSynth yields substantial improvements to the base model across various ratios of mixing real and synthetic data. Experiments on ten datasets in Finance and Biomedicine—evaluating nine mixing ratios following Cheng et al. (2024b) – indicate that mixing real data with synthetic data is not needed if synthetic data is diverse . (section 5 ). (5) We systematically measure the diversity of LLM generated synthetic data across multiple dimensions using seven automated metrics, including the Task2Vec diversity coefficient Lee et al. (2023) , which encapsulates formal notions of data diversity, among others (see Section 4 ). We find that our approach significantly improves the diversity of generated data relative to template-based prompting (section 4 ). We argue that model degradation and “model collapse” (as discussed, inter alia, in Shumailov et al. (2024) ; Seddik et al. (2024) ; Gerstgrasser et al. (2024) ) can be avoided even when training solely on a small amount of synthetic data if it is sufficiently diverse. We present our method below, which we view through the lens of inference-time compute scaling, to ensure diversity in synthetic data. Figure 1: Demonstration of an example MetaSynth agentic workflow for synthesizing a financial document. A meta-LM orchestrates various expert agents that iteratively refine and generate diverse documents conditioned on an initial set of seed documents and previously synthesized documents. Refer to Section 2.2 for a detailed description of the workflow. Algorithm

Read the complete text