Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis

Jiayu Fu, Mourad Heddaya, Chenhao Tan, [0.3em] Department of Computer Science, University of Chicago, Chicago, IL 60637, USA[0.3em] {fujiayu, mourad, chenhao}@uchicago.edu · 2026

Read Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis by Jiayu Fu, Mourad Heddaya, Chenhao Tan, [0.3em] Department of Computer Science, University of Chicago, Chicago, IL 60637, USA[0.3em] {fujiayu, mourad, chenhao}@uchicago.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04386.txt.

Introduction

Fu, Heddaya and Tan build a pipeline that auto-generates hard math benchmarks from hypotheses about model errors. Most math benchmarks need heavy manual effort and scale poorly. Prior auto-generation rarely targets specific weak concepts and skills. The new pipeline uses AI hypotheses to pinpoint weak concepts and skills, then makes fresh problems for those weaknesses. Problems from the best hypotheses cut Llama-3.3-70B-Instruct accuracy to as low as 45 percent, against 77 percent on the original MATH benchmark. The pipeline also adapts beyond math to probe other capabilities.

Catalog record

CREATORJiayu Fu, Mourad Heddaya, Chenhao Tan, [0.3em] Department of Computer Science, University of Chicago, Chicago, IL 60637, USA[0.3em] {fujiayu, mourad, chenhao}@uchicago.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99299
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis Jiayu Fu, Mourad Heddaya, Chenhao Tan, [0.3em] Department of Computer Science, University of Chicago, Chicago, IL 60637, USA[0.3em] {fujiayu, mourad, chenhao}@uchicago.edu Abstract Numerous math benchmarks exist to evaluate LLMs’ mathematical capabilities. However, most involve extensive manual effort and are difficult to scale. Consequently, they cannot keep pace with LLM development or easily provide new instances to mitigate overfitting. Some researchers have proposed automatic benchmark generation methods, but few focus on identifying the specific math concepts and skills on which LLMs are error-prone, and most can only generate category-specific benchmarks. To address these limitations, we propose a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts and skills that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses. Experiments show that hypothesis accuracy positively correlates with the difficulty of the generated problems: problems generated from the most accurate hypotheses reduce Llama-3.3-70B-Instruct’s accuracy to as low as 45%, compared to 77% on the original MATH benchmark. Furthermore, our pipeline is highly adaptable and can be applied beyond math to explore a wide range of LLM capabilities, making it a valuable tool for investigating how LLMs perform across different domains. 1 Introduction Evaluating the mathematical reasoning of LLMs relies on benchmarks, yet most existing benchmarks are manually constructed and curated. This manual process does not scale, creating two persistent problems. First, because benchmark problems are static and publicly available, LLMs frequently encounter them during training—sometimes inadvertently, sometimes by design—leading to overfitted evaluations that overestimate true capability. Efforts to generate fresh instances that resemble the original benchmark ( Zhang et al. 2024a ) still rely on manual construction and therefore inherit the same bottleneck. Second, LLMs are evolving rapidly, but manual benchmark creation cannot keep pace: existing benchmarks quickly become obsolete, while new benchmarks targeting emerging capabilities are slow to appear. To overcome these limitations, recent work has explored LLM-based benchmark generation. While faster than manual construction, current automatic methods share several shortcomings: (i) they are often restricted to a single problem format such as multiple-choice or simple QA; (ii) they do not identify the specific mathematical concepts and skills on which LLMs are weakest, and therefore cannot target generated problems toward those weaknesses; and (iii) they tend to be domain-specific, covering only a narrow slice of mathematics rather than adapting across topics or beyond math entirely. We introduce a benchmark generation pipeline that addresses all three gaps. Our pipeline uses Hypogenic ( Zhou et al. 2024 ) , an LLM-based hypothesis generator, to analyze problems that a target LLM consistently fails and produce hypotheses about the mathematical concepts and skills underlying those failures. These hypotheses then guide the generation of new problems designed to probe the identified weaknesses. The pipeline is lightweight and adaptable: by modifying the hypothesis prompt, it can investigate non-mathematical factors that affect LLM performance (e.g., problem wording or solution length) or generate benchmarks in domains outside of mathematics. We evaluate the pipeline on the MATH benchmark ( Hendrycks et al. 2021 ) using Llama-3.3-70B-Instruct ( Meta 2024 ) as the target model, testing across five levels of mathematical concept granularity and three hypothesis-generating LLMs. Our experiments show a positive correlation between hypothesis accuracy and the difficulty of the resulting problems: problems generated from the most accurate hypotheses reduce the target model’s accuracy to as low as 45%, compared to 77% on the original MATH benchmark. We also find that granularity matters—redundant or overlapping concept categories degrade both hypothesis quality and downstream problem quality. Our contributions are as follows: 1. A generation pipeline that identifies mathematical concepts and skills on which LLMs are weak and produces targeted benchmark problems that are substantially more challenging than general-purpose benchmarks. 2. An empirical analysis showing that the granularity of concept categorization significantly affects both hypothesis accuracy and generated problem quality, with redundant categories degrading performance. 2 Related Works Most existing math benchmarks, such as GSM8k ( Cobbe et al. 2021 ) and MATH ( Hendrycks et al. 2021 ) , require extensive manual effort to construct. Even recent benchmarks rely on substantial human involvement: Glazer et al. 2024 enlist expert mathematicians to create and verify a dataset of challenging problems, and Chernyshev et al. 2024 manually curate a university-level benchmark. Because human-in-the-loop construction is slow and difficult to scale, it struggles to keep pace with LLM progress or to refresh problems quickly enough to prevent overfitting. This has motivated a growing body of work on partially or fully automatic benchmark generation. Extending existing benchmarks. Several methods generate new problems by transforming or mimicking existing ones. Huang et al. 2024 train an LLM-based pipeline to produce new instances that match the length, semantic embedding, and diversity of a source benchmark. O’Brien et al. 2025 modify problems with counterfactual rules to test induction, deduction, and overfitting tendencies. Other work focuses on increasing difficulty: Wang et al. 2024 evolve the complexity of existing benchmarks through multi-agent interaction, and Zhang et al. 2024b extract and perturb reasoning graphs to produce higher-complexity samples. Prompt-based generation. Rather than extending existing data, some approaches generate benchmarks directly from prompts. Yuan et al. 2025 develop BenchMaker, which automatically elaborates a user-provided prompt into detailed multiple-choice questions. Shashidhar et al. 2025 introduce YourBench, a framework that generates QA or MCQ problems using an input document as reference. A limitation of these prompt-based approaches is that they require either seed data or reference documents, making them difficult to apply in domains that lack such resources. Tool-integrated generation. A third line of work combines multiple LLM capabilities in the generation process. Shah et al. 2025 extract and categorize mathematical skills from the MATH benchmark, then prompt an LLM to generate new problems by combining these skills. Peng et al. 2025 build an automatic proof-benchmark generator for algebraic geometry that produces problems resilient to guessing and superficial pattern matching. Hypothesis generation with LLMs. Our pipeline builds on Hypogenic ( Zhou et al. 2024 ) , a framework for generating and evaluating natural-language hypotheses from labeled data using LLMs. Hypogenic iteratively proposes hypotheses, scores them against held-out examples, and returns the most accurate ones. We repurpose this machinery to hypothesize which mathematical concepts and skills underlie an LLM’s failures, then use those hypotheses to guide targeted problem generation. Among existing automatic generation methods, few attempt to identify the specific concepts and skills on which LLMs are most error-prone, and none investigate how the granularity of concept categorization affects generation quality. Our work addresses both gaps. 3 Generation Pipeline Figure 1: Overview of the three-stage generation pipeline: (1) filter problems that the target LLM consistently fails, (2) generate hypotheses about the concepts and skills underlying those failures, and (3) generate new problems guided by the hypotheses. Our pipeline generates challenging math problems through three stages (Figure 1 ): 1. Failing Problem Filtering: Test the target LLM on an existing benchmark and extract problems it consistently answers incorrectly. 2. Hypothesis Generation: Use an LLM to generate hypotheses for which mathematical concepts and skills underlie the failures. 3. Challenging Problem Generation: Generate new problems that target the identified weaknesses, guided by the hypotheses from Stage 2. 3.1 Failing Problem Filtering We

Read the complete text