MathDuels: Evaluating LLMs as Problem Posers and Solvers

Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik · 2026

Read MathDuels: Evaluating LLMs as Problem Posers and Solvers by Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21916.txt.

Introduction

Xu, Jin, Arya, and Naik introduce MathDuels, a benchmark where models author and solve problems. Each model authors problems under hostile prompting and solves others' problems. Problems pass a making pipeline and a checker that excludes ill-posed questions. A Rasch model rates solver skill and problem hardness, and author quality comes from hardness. Dual-role evaluation across 19 frontier models shows authoring and solving partly separate, showing gaps hidden in single-role benchmarks. Newer models produce problems beating top solvers, so difficulty co-evolves with player strength.

Catalog record

CREATORZhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41883
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MathDuels: Evaluating LLMs as Problem Posers and Solvers Zhiqiu Xu, Shibo Jin, Shreya Arya, Mayur Naik Abstract As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. We introduce MathDuels, a self-play benchmark in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. Problems are produced through a three-stage generation pipeline (meta-prompting, problem generation, and difficulty amplification), and validated by an independent verifier that excludes ill-posed questions. A Rasch model ( 39 ) jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model’s authored problems. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation reveals capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark’s difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. We host a public leaderboard that updates as new models are released. Leaderboard: mathduels.ai 1 Introduction Static benchmarks for mathematical reasoning in LLMs are saturating faster than new problems can be produced. Benchmarks such as MATH ( 18 ) and GSM8K ( 9 ) no longer provide the headroom they once did for separating frontier systems. The community therefore turned to live competition problems, but even these are becoming less durable: recent competition-based evaluation results report strong performance on newly released sets, including AIME 2026 ( 5 ) . Model capabilities now advance faster than the supply of fresh, discriminative problems, and no fixed or annually replenished problem pool can sustain meaningful separation for long. This has pushed evaluation toward newly authored, research-level problems designed to remain difficult even for strong models ( 13 ) . When benchmark builders must target the boundary of current model capability, constructing problems that remain discriminative becomes extraordinarily difficult. Humans, in short, find it increasingly hard to supply problems difficult enough to differentiate the strongest models. But this predicament is not new. The question of how to separate two practitioners who both operate at the frontier of a discipline is one that mathematicians confronted centuries ago, and they arrived at an elegant answer: let them test each other. In 1535, the Venetian mathematician Niccolò Tartaglia received a challenge from Antonio Maria Fior: each would deposit thirty problems with a notary, and whoever solved more of the other’s set within fifty days would win ( 45 ) . Victory required not just solving skill but the ability to construct problems that exceeded the opponent’s reach. Tartaglia solved all thirty of Fior’s problems in a short period; Fior could not solve a single one of Tartaglia’s. By forcing each to probe the boundary of the other’s knowledge, the duel exposed a gap that no static test could have surfaced. We bring this principle to LLM evaluation. We introduce MathDuels , a self-play benchmark in which models occupy both roles simultaneously. Each model authors math problems under a fixed budget and solves problems authored by every other model. A model is rewarded both for solving opponents’ problems and for authoring problems that resist solution. As stronger models enter the arena, they generate problems that defeat previously dominant solvers, so the benchmark’s difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. Concretely, a single evaluation round proceeds as follows. Each of N participating models authors K problems via a prompted generation pipeline that includes adversarial difficulty amplification. Every non-author model then attempts each problem. Answers are judged automatically through symbolic verification, and problems that defeat any solver undergo a validity check to filter ill-posed or ambiguous questions. The resulting outcome matrix feeds a Rasch model that jointly estimates solver abilities and problem difficulties, with author quality derived from the difficulties of each model’s authored problems. Figure 1: Solver rating vs. Author rating for 19 frontier models. Our experiments across 19 frontier models reveal two findings that motivate the self-play design. First, solving capability and authoring capability are partially decoupled: strong solvers are not necessarily strong authors, suggesting these are partially independent axes of mathematical competence that single-role benchmarks conflate or ignore entirely. Second, as stronger models enter the arena, they generate problems that defeat previously dominant solvers, so the benchmark’s difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. Contributions. We summarize the contributions of our work: • We propose a self-play benchmark protocol for mathematical LLM evaluation in which models are scored in dual roles, as both problem authors and solvers, with automated problem validity filtering. • We conduct experiments across 19 frontier models and show that dual-role evaluation reveals capability separations invisible in single-role benchmarks, that authoring and solving abilities are partially decoupled, and that the arena resists saturation by design. • We host a public leaderboard which will be updated when new models are released. 2 The MathDuels protocol We formalize an analogous protocol for language models. A MathDuels run takes a set of N models and subjects each to evaluation in two roles: as an author of math problems and as a solver of problems authored by others. The protocol proceeds in four phases (generation, solving, judging, and ranking), each designed to be fully automated and reproducible without human annotation in the loop. Algorithm 1 gives a high-level pseudocode summary; the subsections that follow expand each phase. Algorithm 1 The MathDuels protocol 1: Input: Set of models \mathcal{M}=\{m_{1},\dots,m_{N}\} ; problems per model K ; domains \{\delta_{k}\} 2: Output: Composite ratings \{R_{m}\}_{m=1}^{N} 3: for all m\in\mathcal{M},\;k=1,\dots,K : \triangleright Phase 1: Generation 4: p\coloneqq(q,g)\leftarrow\textsc{Generate}(m,\delta_{k}) 5: 6: for all (m,p) with m\neq\mathrm{author}(p) : \triangleright Phase 2: Solving 7: y_{mp}\leftarrow\mathbf{1}[\textsc{Solve}(m,q_{p})\equiv g_{p}] 8: 9: for all p where \exists\,m:y_{mp}=0 : \triangleright Phase 3: Verification 10: (v,a^{*})\leftarrow\textsc{Verify}(q_{p},g_{p},\{(a_{i},r_{i})\}) 11: if v{=}0 drop p , else g_{p}\leftarrow a^{*} 12: 13: Fit \{s_{m}\},\{d_{p}\} via \triangleright Phase 4: Ranking 14: \max\sum_{(m,p)}\bigl[y_{mp}\log\sigma(s_{m}{-}d_{p})+(1{-}y_{mp})\log\sigma(d_{p}{-}s_{m})\bigr]-\lambda\!\sum_{p}d_{p}^{2} 15: R_{m}^{\mathrm{solve}}\leftarrow R_{\mathrm{anchor}}+C_{\mathrm{elo}}(s_{m}-s_{\mathrm{anchor}}) ; R_{m}^{\mathrm{author}}\leftarrow\mathrm{mean}\!\big(\{C_{\mathrm{elo}}\,\tilde{d}_{p}:\mathrm{author}(p)=m\}\big) 16: R_{m}\leftarrow w_{\mathrm{solve}}\,R_{m}^{\mathrm{solve}}+w_{\mathrm{author}}\,R_{m}^{\mathrm{author}} 2.1 Arena setup Let \mathcal{M}=\{m_{1},\dots,m_{N}\} be the set of participating models and let K be the number of problems each model generates, giving NK problems in total. Every model attempts every problem it did not author, so for each solver–problem pair (m,p) with m\neq\mathrm{author}(p) we observe a binary outcome y_{mp}\in\{0,1\},\qquad y_{mp}=1\;\Longleftrightarrow\;\text{model }m\text{ solves problem }p\text{ correctly}. These outcomes populate a solve matrix Y\in\{0,1\}^{N\times NK} whose entries are defined only when m\neq\mathrm{author}(p) . The full evaluation pipeline is: \mathcal{M}\;\xrightarrow{\text{generate}}\;NK\text{ problems}\;\xrightarrow{\text{attempt}}\;Y\;\xrightarrow{\text{rank}}\;\{R_{m}\}_{m=1}^{N}, where R_{m} is the final composite rating of model m . Sections 2.2 – 2.5 describe each stage. 2.2 Problem generation Generating problems that are simultaneously hard, well-posed, and unambiguously verifiable is the central design challenge. We use a three-stage pipeline: meta-prompting, problem generation, and difficulty amplification. The guiding design principle is minimal human steering: beyond specifying that problems should be difficult and framing the task as a competitive arena, the prompts impose no constraints on mathematical content, style, or strategy, letting each model explore its own notion of what makes a problem hard. Stage 1: meta-prompting. We first ask the model to write a prompt for authoring a problem, given

Read the complete text