IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation

Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck, Jeremy Feusi, Tim Gehrunger · 2025

Read IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation by Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck, Jeremy Feusi, Tim Gehrunger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26076.txt.

Introduction

Johannes Schmitt and a large team of co-authors introduce IMProofBench, a test set for judging how well AI systems write research-level maths proofs, in a 2025 paper. Language models keep getting better at maths, so school-style problems no longer show what they can and cannot do. The benchmark instead uses problems at the edge of current knowledge, where a valid proof is a real research product. The paper sets out why such a hard yardstick is needed.

Catalog record

CREATORJohannes Schmitt, Gergely Bérczi, Jasper Dekoninck, Jeremy Feusi, Tim Gehrunger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-26076

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck, Jeremy Feusi, Tim Gehrunger Abstract As the mathematical capabilities of large language models (LLMs) improve, it becomes increasingly important to evaluate their performance on research-level tasks at the frontier of mathematical knowledge. However, existing benchmarks are limited, as they focus solely on final-answer questions or high-school competition problems. To address this gap, we introduce IMProofBench, a private benchmark consisting of 39 39 peer-reviewed problems developed by expert mathematicians. Each problem requires a detailed proof and is paired with subproblems that have final answers, supporting both an evaluation of mathematical reasoning capabilities by human experts and a large-scale quantitative analysis through automated grading. Furthermore, unlike prior benchmarks, the evaluation setup simulates a realistic research environment: models operate in an agentic framework with tools like web search for literature review and mathematical software such as SageMath. Our results show that current LLMs can succeed at the more accessible research-level questions, but still encounter significant difficulties on more challenging problems. Quantitatively, Grok-4 achieves the highest accuracy of 52 % 52\% on final-answer subproblems, while GPT-5 obtains the best performance for proof generation, achieving a fully correct solution for 22 % 22\% of problems. IMProofBench will continue to evolve as a dynamic benchmark in collaboration with the mathematical community, ensuring its relevance for evaluating the next generation of LLMs. Benchmark Contributors: Raphael Appenzeller, Jim Bryan, Niklas Canova, Timo de Wolff, Filippo Gaia, Michel van Garrel, Baran Hashemi, David Holmes, Aitor Iribar Lopez, Victor Jaeck, Martina Jørgensen, Steven Kelk, Stefan Kuhlmann, Adam Kurpisz, Chiara Meroni, Ingmar Metzler, Martin Möller, Samuel Muñoz-Echániz, Robert Nowak, Georg Oberdieck, Daniel Platt, Dylan Possamaï, Gabriel Ribeiro, Raúl Sánchez Galán, Zheming Sun, Josef Teichmann, Richard P Thomas, Charles Vial https://improofbench.math.ethz.ch 1 Introduction Large language models (LLMs) are making rapid progress on mathematical tasks, achieving strong results on challenging benchmarks like AIME ( Balunovic et al. 2025 ) and FrontierMath ( Glazer et al. 2024 ) . These improvements suggest that LLMs may soon support mathematical research by collaborating with professional mathematicians on open problems. However, to determine whether current systems are capable of contributing in such settings, benchmarks are needed that test capabilities at the frontier of mathematical research. Limitations of existing benchmarks Existing benchmarks fall short of this objective: most focus on high-school or undergraduate-level mathematics ( Balunovic et al. 2025 ; Frieder et al. 2023 ) , due to the difficulty associated with designing rigorous, research-level problems. The few benchmarks that do target more advanced mathematics, like FrontierMath ( Glazer et al. 2024 ) and HLE ( Phan et al. 2025 ) , focus exclusively on final-answer problems. As a result, they overlook proof-writing capabilities and allow models to apply shortcuts to reach the correct final answer without fully solving the problem ( EpochAI 2025 ) . This work: IMProofBench To fill this gap, we introduce IMProofBench, a private benchmark developed in collaboration with the mathematical research community to evaluate LLMs on research-level proof writing. IMProofBench is built on a custom platform and supported by initiatives that actively involve professional mathematicians. It includes tasks ranging from challenging oral exam questions in a graduate course to open research questions based on the contributors’ own work. Unlike static benchmarks, IMProofBench is designed as a platform for continuous evaluation: problems are added on a rolling basis, ensuring its continued relevance for evaluating the next generation of frontier LLMs. Currently, IMProofBench consists of 39 problems developed in collaboration with over 23 mathematicians, with 30 more questions in the latest stages of the problem creation pipeline. Problem creation pipeline Each problem in IMProofBench is authored by a research mathematician within their area of expertise. Submissions undergo a rigorous review process by a core team member and an additional mathematician with expertise in the relevant field. Reviewers provide feedback that allows authors to refine their problems before finalization. Alongside the main proof-writing tasks, authors are encouraged to add follow-up subquestions with final answers that can be automatically graded. These follow-ups enable a comparison between proof-writing and final-answer performance, while also supporting lower-cost evaluation across a broader range of models. Question: Isomorphism Classes of Stable Graphs Given an integer g\geq 2 , let N_{g} be the number of isomorphism classes of stable graphs of genus g with precisely 3 edges. Give a closed formula for N_{g} valid for all g\geq 2 . Follow-up subquestions : What are N_{3},N_{8} , and N_{10000} ? GPT-5 Reasoning Summary N_{g} is a sum of connected multigraph types limited by 3 edges, considering partitions of genera. Obtain experimental data for some values. Submits a closed formula for N_{g} as a degree-3 quasi-polynomial with a period of 6 . Full Grade : 3/3 Subquestions : Grok-4 Reasoning Summary Full Grade : 2/3 Subquestions : Finds a non-closed formula for N_{g} , requiring to compute a sum from 1 to g . Attempts to look up the sequence in OEIS. Submits a very concise sketch of the answer, with the non-closed formula for N_{g} . Figure 1: Example IMProofBench problem. Models are tested on research-level questions in an agentic framework with tool access. Grading of the main reasoning is done by a human expert, while follow-up subquestions are evaluated using an automated parser. For the question above, models other than GPT-5 and Grok-4 only made minor progress. For full details, see Appendix C . Evaluation process Evaluation is conducted in an agentic framework designed to mirror a research environment. Models have access to computational tools such as Python and SageMath ( sag 2025 ) , as well as web search and multi-turn reasoning. Each model is first tasked with solving the main problem, followed by the associated follow-up subquestions. The main solution is graded by the problem’s author, who assigns a score from zero to three. Graders also annotate the types of errors, such as logical mistakes, and identify specific areas of partial progress, such as correct intermediate insights. Follow-up answers are automatically evaluated by comparing them with ground-truth solutions. We illustrate this process in Fig. 1 , which shows a sample problem with two model solutions. Key results We evaluate 10 state-of-the-art LLMs on the current version of IMProofBench. Our results show that models can already solve a small but meaningful fraction of research-level problems: the best model, GPT-5 , produces complete solutions for 22\% of tasks, closely followed by Grok-4 at 19\% . Notably, Grok-4 achieves the highest final-answer accuracy at 52\% , surpassing GPT-5 ’s 42\% . Other models lag further behind, with Claude-Opus-4.1 scoring particularly poorly, only providing complete solutions in 3\% of tasks. Qualitative analysis Beyond aggregate scores, our analysis reveals that many models are prone to reasoning errors, ranging from simple logical mistakes to deep misconceptions unlikely to be exhibited by any professional mathematician in the relevant area. Indeed, almost half of the model solutions contain arguments revealing fundamental misunderstandings of mathematical concepts, as judged by human graders. Moreover, models frequently hallucinate existing results to obtain a (flawed) answer. Finally, models almost never abstain from providing a solution attempt, preferring to present convincing but incorrect proofs rather than admit they are stuck. At the same time, they also show a wide-ranging familiarity with

Read the complete text