SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms

Alex Havrilla, Edward Hughes, Mikayel Samvelyan, Jacob Abernethy · 2025

Read SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms by Alex Havrilla, Edward Hughes, Mikayel Samvelyan, Jacob Abernethy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06499.txt.

Introduction

Alex Havrilla, Edward Hughes, Mikayel Samvelyan, and Jacob Abernethy present SPARQ, a way to grow training data for AI reasoning, in a 2025 paper. Teaching models math needs many good practice problems. Most methods copy a bigger model's work or start from human-written problems, which limits scale.

SPARQ lets one model invent its own problems and rate each by how often it can solve them. That solve rate stands in for difficulty. Starting from a small seed set, the method produced more than twenty million problem and solution pairs. After filtering for the right difficulty, training the same model on its own output raised scores by up to twenty-four percent. Tests also show quality and variety matter more than sheer volume.

Catalog record

CREATORAlex Havrilla, Edward Hughes, Mikayel Samvelyan, Jacob Abernethy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06499

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms Alex Havrilla, Edward Hughes, Mikayel Samvelyan, Jacob Abernethy Abstract Large language model (LLM) driven synthetic data generation has emerged as a powerful method for improving model reasoning capabilities. However, most methods either distill large state-of-the-art models into small students or use natural ground-truth problem statements to guarantee problem statement quality. This limits the scalability of these approaches to more complex and diverse problem domains. To address this, we present Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms ( SPARQ ), a novel approach for generating high-quality and diverse synthetic math problem and solution pairs using only a single model ( Gemma-2-9b ) by measuring a problem’s solve-rate : a proxy for problem difficulty. Starting from a seed dataset of 7.5K samples, we generate over 20 million new problem-solution pairs. We show that filtering the generated data by difficulty and then fine-tuning the same model on the resulting data improves relative model performance by up to 24%. Additionally, we conduct ablations studying the impact of synthetic data quantity, quality and diversity on model generalization. We find that higher quality, as measured by problem difficulty, facilitates better in-distribution performance. Further, while generating diverse synthetic data does not as strongly benefit in-distribution performance, filtering for more diverse data facilitates more robust OOD generalization. We also confirm the existence of model and data scaling laws for synthetically generated problems, which positively benefit downstream model generalization. † † footnotetext: ∗ Work done during an internship at Google Research. Correspondence to [email protected] . Google Research 1 Google DeepMind 2 Georgia Tech 3 1 Introduction The quantity of high-quality problem statements is one of the most impactful factors affecting the downstream performance of both supervised fine-tuned (SFT) and RL fine-tuned models ( Toshniwal et al. 2024b ; Singh et al. 2024 ) . However, the vast majority of synthetic data generation approaches either (1) generate only new solutions to a fixed problem set ( Toshniwal et al. 2024b ; Toshniwal et al. 2024a ; Havrilla et al. 2024b ) or (2) generate new problems using a large state-of-the-art oracle without carefully filtering the resulting problem data for quality and correctness ( Yue et al. 2023 ; Yu et al. 2024 ) . These approaches sidestep a key difficulty in problem generation: the lack of a ground truth signal which can be used to filter out illogical/invalid problems. As a result, these approaches are not scalable to generating new problem-solution pairs whose difficulty exceeds the capabilities of existing SOTA models. Additionally, the restriction to natural data limits problem-solution diversity, especially in the increasingly complex domains in which models are applied. In this work we present Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms ( SPARQ ), a novel synthetic data generation algorithm producing high-quality and diverse problem-solution pairs using a single student model S {}_{\theta_{2}} ( Gemma-2-9b ( Team et al. 2024 ) ) and seed dataset \mathcal{D} . The key idea behind SPARQ is to use monte-carlo rollouts from S {}_{\theta_{2}} to estimate the solve-rate (i.e., "difficulty") of a problem-solution pair (Q,A) for S {}_{\theta_{2}} . We then use the solve-rate as a proxy for problem quality, allowing us to filter out low-quality generations which are either too hard/impossible or too easy. We demonstrate that generating and filtering data based on this quality score significantly improves the pass@1 accuracy of the MATH ( Hendrycks et al. 2021 ) SFT Gemma-2-9b student from 38% to 47%. An ablation confirms that increased average train problem quality correlates well with improved downstream performance. We then investigate the role of problem diversity in problem generation and downstream generalization using SPARQ . Problem diversity is measured first by annotating problem-solution pairs (Q,A) with the skill/technique set used in solving Q . For example, Q may require a combination of the pigeonhole principle and algebra in the correct solution. Representing problems with their most relevant skills allows us to measure the overall diversity of a set of problems by examining the coverage (number of unique skills) and redundancy (i.e., number of repeated skills). Given a fixed sample budget N , we find increasing problem diversity does not improve in-distribution generalization relative to a randomly selected training data baseline. However, we find models trained on more diverse synthetic data generalize better to out-of-distribution ( OOD ) tasks as the amount of compute used to solve the task increases. In summary, we make the following contributions: 1. We introduce the solve-rate of a problem Q with respect to S {}_{\theta_{2}} as an effective way of measuring and filtering by synthetic problem quality. 2. We present SPARQ , a new approach for synthetic problem generation that directly optimizes for the data quality and diversity. Training on the resulting data leads to an absolute downstream improvement over the baseline of 9%. 3. We carefully ablate the impact of quality and diversity of synthetic data generated with SPARQ on model generalization, revealing increased quality to correlate well with better in-distribution generalization and diversity to benefit OOD generalization. 2 Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms Figure 1: Diagram of the synthetic training data generation pipeline with SPARQ . Phase 1: Data generation begins with \mathcal{D}_{\text{seed}} initializing the working set. Samples (Q,A) are iteratively selected from the working set \mathcal{W} via the selection distribution C and mutated via the generator G {}_{\theta_{1}} . All new mutations are stored in the archive \mathcal{A} . The quality of new samples (Q^{\prime},A^{\prime}) is assessed via the difficulty student model S {}_{\theta_{2}} has solving for A^{\prime} given Q^{\prime} . The working set update function U then updates \mathcal{W} with new sample (Q^{\prime},A^{\prime}) . Phase 2: After data generation, data filtration proceeds by removing all low-quality samples from the archive \mathcal{A} . Each remaining question Q is then paired with its successful verifications S_{k} forming synthetic training tuples (Q,S_{k}) . Setup and notation Let \theta_{1}\in\mathbb{R}^{d_{1}},\theta_{2}\in\mathbb{R}^{d_{2}} be optionally distinct model parameters. Suppose we are given a generating language model G {}_{\theta_{1}} and student model S {}_{\theta_{2}} . Fix a tokenized vocabulary \mathcal{V}=\{1,...,V\}\subseteq\mathbb{N} for some V\in\mathbb{N} . Let the task \tau be a distribution of problem-solution pairs (Q,A)\in\mathcal{V}^{L_{Q}}\times\mathcal{V}^{L_{A}} where L_{Q},L_{A}\in\mathbb{N} are the maximum context lengths of problems and solutions respectively. Let \mathcal{D} be a seed dataset of n\in\mathbb{N} problem-solution pairs (Q_{i},A_{i})\sim\tau . A high-level synthetic data generation recipe Given \mathcal{D} we initialize a working set set \mathcal{W}^{(0)} of mutation candidates meant to hold high-quality samples for future mutation. We also initialize an archive \mathcal{A}^{(0)} which accumulates all generated data regardless of quality. Our synthetic data generation algorithm then proceeds in two phases: data generation and data filtration. The data generation phase proceeds iteratively at round t by sampling a batch of problem-solution pairs (Q_{i}^{(t)},A_{i}^{(t)})\sim\textbf{C} of size b\in\mathbb{N} from the working set \mathcal{W}^{(t)} using the mutation selection distribution \textbf{C}^{(t)}\in\Delta(\mathcal{W}^{(t)}) . Mutations ({Q^{\prime}}_{i}^{(t)},{A^{\prime}}_{i}^{(t)})\sim G {}_{\theta_{1}} (\cdot|(Q_{i},A_{i})) are then sampled using the generator G {}_{\theta_{1}} . We call (Q_{i}^{(t)},A_{i}^{(t)}) the parent and ({Q^{\prime}}_{i}^{(t)},{A^{\prime}}_{i}^{(t)}) the child or mutation of (Q_{i}^{(t)},A_{i}^{(t)}) . We then measure the Quality of the new pairs ({Q^{\prime}}_{i}^{(t)},{A^{\prime}}_{i}^{(t)}) and update the working set \mathcal{W}^{(t)}\to\mathcal{W}^{(t+1)} via the update function \textbf{U}^{(t)} . The update function \textbf{U}^{(t)}\left(\mathcal{W}^{(t)},\{({Q^{\prime}}_{1}^{(t)},{A^{\prime}}_{1}^{(t)}),...,({Q^{\prime}}_{b}^{(t)},{A^{\prime}}_{b}^{(t)})\}\right)\to\mathcal{W}^{(t+1)} inserts the high-quality subset of new mutations into \mathcal{W}^{(t)} and removes excess low-quality

Read the complete text