Towards Robust Mathematical Reasoning

Thang Luong ⋄ \diamond , Dawsen Hwang * , Hoang H. Nguyen * † \dagger , Golnaz Ghiasi * , Yuri Chervonyi * , Insuk Seo * † \dagger , Junsu Kim * , Garrett Bingham, Jonathan Lee, Swaroop Mishra † \dagger , Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim † \dagger , Jeonghyun Ahn † \dagger , · 2025

Read Towards Robust Mathematical Reasoning by Thang Luong ⋄ \diamond , Dawsen Hwang * , Hoang H. Nguyen * † \dagger , Golnaz Ghiasi * , Yuri Chervonyi * , Insuk Seo * † \dagger , Junsu Kim * , Garrett Bingham, Jonathan Lee, Swaroop Mishra † \dagger , Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim † \dagger , Jeonghyun Ahn † \dagger , (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01846.txt.

Introduction

Thang Luong and a large team of collaborators introduce IMO-Bench. It is a benchmark suite pitched at the level of the International Mathematical Olympiad. It measures real mathematical reasoning in foundation models, rather than easy short answers. Every problem was vetted by a panel of top specialists.

IMO-AnswerBench holds 400 diverse Olympiad problems with verifiable short answers. IMO-ProofBench goes further, grading full proof writing at basic and advanced levels with detailed grading guidelines for automatic scoring. The benchmarks were central to the team's gold-level Gemini Deep Think result at IMO 2025. The model scored 80 percent on AnswerBench and about 66 percent on the advanced ProofBench. It beat the best non-Gemini models by wide margins. A companion GradingBench, with a thousand human-graded proofs, supports progress in automatic evaluation.

Catalog record

CREATORThang Luong ⋄ \diamond , Dawsen Hwang * , Hoang H. Nguyen * † \dagger , Golnaz Ghiasi * , Yuri Chervonyi * , Insuk Seo * † \dagger , Junsu Kim * , Garrett Bingham, Jonathan Lee, Swaroop Mishra † \dagger , Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim † \dagger , Jeonghyun Ahn † \dagger ,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Robust Mathematical Reasoning Thang Luong ⋄ \diamond , Dawsen Hwang * , Hoang H. Nguyen * † \dagger , Golnaz Ghiasi * , Yuri Chervonyi * , Insuk Seo * † \dagger , Junsu Kim * , Garrett Bingham, Jonathan Lee, Swaroop Mishra † \dagger , Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim † \dagger , Jeonghyun Ahn † \dagger , Junhwi Bae † \dagger , Xingyou Song, Trieu H. Trinh, Quoc V. Le, Junehyuk Jung ⋄ \diamond ⋄ \diamond Corresponding authors, * Core and equal contributors, † \dagger Work previously conducted under Google DeepMind Abstract Finding the right north-star metrics is highly critical for advancing the mathematical reasoning capabilities of foundation models, especially given that existing evaluations are either too easy or only focus on getting correct short answers. To address these issues, we present IMO-Bench , a suite of advanced reasoning benchmarks, vetted by a panel of top specialists and that specifically targets the level of the International Mathematical Olympiad (IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench first tests models on 400 diverse Olympiad problems with verifiable short answers. IMO-ProofBench is the next-level evaluation for proof-writing capabilities, which includes both basic and advanced IMO level problems as well as detailed grading guidelines to facilitate automatic grading. These benchmarks played a crucial role in our historic achievement of the gold-level performance at IMO 2025 with Gemini Deep Think Luong and Lockhart 2025 . Our model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-ProofBench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4% respectively. We also showed that autograders built with Gemini reasoning correlate well with human evaluations and construct IMO-GradingBench , with 1000 human gradings on proofs, to enable further progress in automatic evaluation of long-form answers. We hope that IMO-Bench will help the community towards advancing robust mathematical reasoning and release it at https://imobench.github.io . 1 Introduction The field of artificial intelligence, particularly large language or foundation models, has demonstrated remarkable progress in mathematical reasoning capabilities. Many popular benchmarks such as GSM8K Cobbe et al. 2021 , MATH Hendrycks et al. 2021 , and the recently popular AIME have approached saturation, limiting their usefulness in differentiating model performances. The problems in these datasets often rely on a limited set of techniques and do not always require the deep, multi-step reasoning needed to truly evaluate AI mathematical reasoning. Indeed, relying on final answer matching, even in recent benchmarks such as FrontierMath Glazer et al. 2024 and Humanity’s Last Exam Phan et al. 2025 , is not entirely reliable. It could lead to AI systems that are good at guessing answers but do not exhibit robust reasoning. Figure 1 : IMO-ProofBench , a benchmark in IMO-Bench , for measuring proof-writing capabilities. We demonstrated high correlations between human and automatic evaluations on a variety of public models, including our IMO-gold model. See \lx@sectionsign 3 and \lx@sectionsign 5.3 for more details. To address these shortcomings, we propose IMO-Bench , a suite of benchmarks that focus on robust reasoning at the level of the International Mathematical Olympiad (IMO), the world’s most celebrated arena for young mathematicians. The IMO is selected due to its notoriously difficult problems, which require not only rigorous multi-step reasoning but also a high degree of novelty, going beyond the simple application of known formulas. Such characteristics make IMO an excellent testbed for assessing reasoning capability. IMO-Bench covers three different tasks as summarized in Table 1 and all problems were vetted by a panel of IMO medalists 1 1 1 Together, they won 10 gold and 5 silver IMO medals. and mathematicians. Benchmark Size Task IMO-AnswerBench 400 Get the right answer IMO-ProofBench 60 Write a rigorous proof IMO-GradingBench 1000 Grade a proof Table 1 : Benchmarks in the IMO-Bench suite. Figure 2 : Topic distribution by category in IMO-AnswerBench . Number Theory and Combinatorics have the most topics which reflect the broad knowledge required to solve these problems while Geometry is mostly skewed towards angle and sidelength computation problems due to the nature of the short answer benchmark. The first benchmark, IMO-AnswerBench , consists of 400 problems with verifiable answers carefully chosen from past Olympiad competitions and then altered by experts to avoid memorization. Problems were chosen from a variety of topics whose solutions require different problem solving techniques to ensure a diverse representation of topics, ideas, and domain knowledge as illustrated in Figure 2 . The second benchmark, IMO-ProofBench , consists of 60 problems of varying difficulty levels, similar to those found at the IMO. While some problems have short answers, all require models to generate complete proofs. The benchmark is divided into two subsets, basic and advanced , each with 30 problems. While the basic set covers difficulty levels from pre-IMO up to IMO-Medium, problems in the advanced set are up to IMO-hard level and consist of 5 complete IMO sets, 3 of which are novel. We designed this benchmark to steer the community’s focus from final answers to proofs, enabling a more rigorous assessment of AI reasoning processes. To ensure consistent evaluation, we include detailed grading schemes suitable for both human experts and automated systems. Figure 1 provides an early look into the potential of automatic graders for proofs. These two benchmarks played a crucial role in the development of our Gemini Deep Think, leading to the historic achievement of the gold-level performance at IMO 2025 Luong and Lockhart 2025 . Our IMO-gold model achieved an accuracy of 80.0% on IMO-AnswerBench by automatic evaluation, surpassing the best non-Gemini model and the best open-weight model by a large margin of 6.9% and 19.2% respectively. The advanced IMO-ProofBench is much more challenging. Our IMO-gold model scored 65.7%, whereas the best non-Gemini and the best open-weight models performed poorly with only 23.3% and 7.1% accuracy according to human evaluations. Furthermore, we demonstrate that automated graders for both answers and proofs, built upon Gemini 2.5 Pro, achieve high correlation with expert human evaluations. Last but not least, we introduce IMO-GradingBench , a benchmark of 1000 solutions to problems in the advanced IMO-ProofBench, together with grades from human experts. This resource is designed to foster progress in the automatic evaluation of long-form answers. We release IMO-Bench to the community and hope that it will spur further research towards advancing robust mathematical reasoning. 2 IMO-AnswerBench 2.1 Problem Selection 400 math problems were handpicked from various national, regional, and international Olympiad contests, spanning across four categories ( Algebra, Combinatorics, Geometry, Number Theory ). For each category, the benchmark contains 100 problems across four levels of difficulty: pre-IMO (middle school or pre-Math Olympiad problems), IMO-Easy (equivalent to Problem 1 or Problem 4 at the IMO), IMO-Medium (equivalent to Problem 2 or Problem 5 at the IMO) and IMO-Hard (equivalent to Problem 3 or Problem 6 at the IMO or post-Math Olympiad problems). The difficulty breakdown for each category is listed in Table 2 . Category Pre-IMO IMO-Easy IMO-Medium IMO-Hard Algebra 11 46 32 11 Combinatorics 4 19 31 46 Geometry 13 44 32 11 Number Theory 2 20 31 47 Table 2 : Difficulty statistics for IMO-AnswerBench . Problems with short answers were chosen so the correctness of a model’s output can be quickly and

Read the complete text