Verifier-Backed Hard Problem Generation for Mathematical Reasoning
Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao · 2026
Read Verifier-Backed Hard Problem Generation for Mathematical Reasoning by Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06660.txt.
Introduction
Yuhang Lai, Jiazhan Feng, Yee Whye Teh, and Ning Miao introduce VHG, a framework that generates hard new math problems for training language models. Models solve problems well but struggle to create ones that are valid, challenging, and novel. Naive self-play generation often yields invalid problems because the setter exploits weak rewards, a failure called reward hacking.
VHG adds an independent verifier to the usual setter-solver pair, so the setter's reward depends jointly on validity, judged by the verifier, and difficulty, judged by the solver. Two verifier types are tested, one symbolic and one model-based. On integral tasks and general math reasoning, VHG beats all baselines clearly.
Catalog record
| CREATOR | Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06660
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Verifier-Backed Hard Problem Generation for Mathematical Reasoning Yuhang Lai, Jiazhan Feng, Yee Whye Teh, Ning Miao Abstract Large Language Models (LLMs) demonstrate strong capability in solving scientific and mathematical problems, yet they struggle to produce valid and challenging novel problems—an essential component for advancing LLM training and enabling autonomous scientific research. Existing problem generation approaches either depend on expensive human expert involvement or adopt naive self-play paradigms, which frequently yield invalid problems due to reward hacking. This work introduces VHG, a verifier-enhanced hard problem generation framework built upon three-party self-play. By integrating an independent verifier into the conventional setter-solver duality, our design constrains the setter’s reward to be jointly determined by problem validity (evaluated by the verifier) and difficulty (assessed by the solver). We instantiate two verifier variants: a Hard symbolic verifier and a Soft LLM-based verifier, with evaluations conducted on indefinite integral tasks and general mathematical reasoning tasks. Experimental results show that VHG substantially outperforms all baseline methods by a clear margin. * * footnotetext: Equal contribution. $\dagger$ $\dagger$ footnotetext: Work done during Jiazhan’s visiting at the Department of Statistics, University of Oxford. 1 Introduction Large language model (LLM) have achieved expert-level ability in solving scientific problems. For example, OpenAI’s o1 surpassed PhD-expert baselines on GPQA-Diamond [ 21 , 18 ] , while AlphaGeometry [ 25 ] and AlphaProof [ 10 ] demonstrated olympiad-level mathematical reasoning, making them very useful in doing research under human supervision. In scientific research, raising a meaningful new question is at least as critical as solving an existing one. This raises a natural question, Can LLMs truly generate valid new questions, enabling them to complete the research cycle and achieve fully autonomous research? Generating valid new questions is not only pivotal to scientific research but also to the training of LLMs themselves. For example, 4 found that data difficulty is one of the primary factors influencing the performance of LLMs after post-training. However, current post-training paradigms mostly rely on static human-written datasets (e.g., MATH [ 7 ] ), or offline transformations and post-training recipes (e.g., MetaMath [ 27 ] , WizardMath [ 15 ] , ToRA [ 5 ] , DeepSeekMath [ 22 ] , and R-Zero [ 8 ] ). To continuously increase the difficulty of training data, we need to hire domain experts either to compose new problems or modify existing ones, which is not only slow and costly, but also limits LLMs’ ability to the human level. Numerous previous efforts have been made on generating difficult problems. For example, PromptCoT [ 31 ] , CHASE [ 19 ] , and MathSmith [ 29 ] use human-designed scaffolds, independently verifiable components, or predefined difficulty strategies to build new problems. MathFusion combines old problems through sequential, parallel, and conditional fusion strategies [ 20 ] . Though effective in certain tasks, these methods are still upper-bounded by human design. Self-play based methods almost eliminate reliance on human expertise. They use a setter LLM to propose new problems, and another solver LLM to measure the difficulty of the proposed problems. Then, the negative problem accuracy is used as a reward for the setter, so that it will learn to generate harder and harder problems. Though looks elegant, vanilla self-play has a critical flaw, that is the proxy reward of problem difficulty can be easily hacked by generating invalid problems, where the solver has zero accuracy, which provides high rewards to the setter. Figure 1: VHG framework. The setter proposes problem-reference pairs, the verifier gates validity, and accepted pairs are scored by solver difficulty for training and challenge construction. In this paper, we address this problem by adding an additional verifier to the two-party game in self-play. Specifically, we propose VHG: verifier-backed hard problem generation : a setter proposes both the problem and solution as a pair (x,y^{\star}) , and a verifier accepts or rejects the pair based on its correctness. This design achieves two key goals: On the solver side, it eliminates the noisy training signals introduced by invalid problem-reference pairs. More importantly, on the setter side, for a generated problem, a high reward (i.e., low solver accuracy) can only result from the problem’s true difficulty, making it impossible for the setter to hack the reward system by generating invalid problems. We explore two types of verifiers: Hard and Soft. Hard verifiers leverage symbolic verification mechanisms to provide nearly 100% reliable verifications. Soft verifiers use LLMs to check the correctness of the step-by-step problem generation process. Though less accurate than hard verifiers, soft verifiers enables the framework to work on broader domains. To validate the concept of VHG, we first focus on the task of indefinite integral, a self-contained environment, allowing us to examine the inner working mechanisms of VHG. We also test VHG with a soft verifier in the general math domain, where exact verification is impractical, demonstrating its generalization potential. Our experiments show that VHG produces problem-reference pairs that are both valid and challenging, proving useful for both solver training and challenge dataset construction. For indefinite integral, VHG improves pass@1 accuracy on AntiderivBench Qualifier/Competition and Integration Stress Test by 16.9%, 16.6%, and 21.4%, respectively, significantly outperforming baseline approaches, including R-Zero, which is the current state-of-the-art model. For general math, VHG enhances performance across a wide range of benchmarks including MATH, AMC, Minerva, Olympiad, and AIME24-26: raising overall pass@1 accuracy from 56.8% to 69.0% and also outperforming baselines by a substantial margin. Beyond performance gains, we find that even though both the setter and the solver are based on Qwen3-4B, VHG can generate problems that challenge larger models (Qwen3-8B, 14B, and 32B), indicating that stronger models can benefit from training with data generated by weaker models. In summary, the contributions of this paper are as follows: • We design a novel three-party self-play framework for generating challenging problem, incorporating an additional verifier module. This framework avoids the generation of invalid problems, which would otherwise lead to a collapse of the training pipeline. • We conduct an extensive empirical study of VHG using both hard and soft verifiers, demonstrating that VHG can generate novel, challenging problems that in turn boost performance across multiple benchmarks. • We perform a comprehensive analysis of VHG’s working mechanism, highlighting the critical role of the verifier in ensuring the quality of generated problems. 2 Method: The VHG Framework Verifier-backed hard problem generation (VHG) is a three-party self-play framework designed to generate mathematical problems that are valid, novel, and challenging. It extends the standard setter–solver self-play paradigm by introducing a verifier: the setter proposes a problem-reference pair, the solver provides feedback on its empirical difficulty, and the verifier validates the pair’s correctness. During training, generated pairs are first checked for validity, and only verifier-accepted pairs are scored for difficulty or used to train the solver. This ensures that invalid or underspecified pairs are excluded from downstream processes, mitigating reward hacking and ensuring the quality of generated problems. Setter \mathrm{Q} . The setter \mathrm{Q} is responsible for generating problem-reference pairs (x,y^{\star}) , where x denotes a proposed mathematical problem and y^{\star} is its reference solution. This extends the conventional self-play formulation, where the setter typically only needs to propose a problem without a reference solution. In our framework, the reference solution y^{\star}