MathArena: Evaluating LLMs on Uncontaminated Math Competitions

Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, Martin Vechev · 2025

Read MathArena: Evaluating LLMs on Uncontaminated Math Competitions by Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, Martin Vechev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23281.txt.

Introduction

Mislav Balunović and co-authors build a math contest benchmark that cannot leak in a 2025 paper. Popular test sets sit online, so models may have memorized them rather than learning to reason. Their platform, MathArena, grades models on fresh problems the moment recurring contests release them. Contamination then has no time to happen.

Over fifty models faced 162 problems across seven contests. Signs of memorization showed in one famous older set. On a harder 2025 contest, top models still reasoned impressively. The platform also grades proof writing, a first for such benchmarks. On the 2025 olympiad set, the best models scored under forty percent. Fresh problems keep the measure honest as models grow stronger.

Catalog record

CREATORMislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, Martin Vechev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MathArena: Evaluating LLMs on Uncontaminated Math Competitions Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, Martin Vechev Abstract The rapid advancement of reasoning capabilities in large language models (LLMs) has led to notable improvements on mathematical benchmarks. However, many of the most commonly used evaluation datasets (e.g., AIME 2024) are widely available online, making it difficult to disentangle genuine reasoning from potential memorization. Furthermore, these benchmarks do not evaluate proof-writing capabilities, which are crucial for many mathematical tasks. To address this, we introduce MathArena , a new benchmark based on the following key insight: recurring math competitions provide a stream of high-quality, challenging problems that can be used for real-time evaluation of LLMs. By evaluating models as soon as new problems are released, we effectively eliminate the risk of contamination. Using this framework, we find strong signs of contamination in AIME 2024. Nonetheless, evaluations on harder competitions, such as CMIMC 2025, demonstrate impressive reasoning capabilities in top-performing models. MathArena is also the first benchmark for proof-writing capabilities. On IMO 2025, top models achieve slightly less than 40%, demonstrating both notable progress and significant room for improvement. So far, we have evaluated over 50 50 models across seven competitions, totaling 162 162 problems. As an evolving benchmark, MathArena will continue to track the progress of LLMs on newly released competitions, ensuring rigorous and up-to-date evaluation of mathematical reasoning. 1 Introduction Recent advances in the mathematical reasoning capabilities of large language models (LLMs) [ 20 , 7 ] have raised the following three concerns about the adequacy of existing mathematics benchmarks: 1. Contamination risks : Many benchmarks are sourced from publicly available math competitions, which are accessible online and often used to train LLMs. This leaves them susceptible to data contamination, making it difficult to measure progress accurately. Data contamination can occur either through indirect inclusion of benchmark problems in training data or by using benchmark performance for hyperparameter tuning or model selection. For instance, we find that the popular AIME 2024 dataset is significantly contaminated by most leading LLMs, making the benchmark unsuitable for evaluating the models’ capabilities. 2. High-cost, private benchmarks : To mitigate contamination, some leading benchmarks—such as FrontierMath [ 15 ] and HLE [ 29 ] —adopt a private, human-curated approach. While effective in avoiding data leakage, these datasets pose several major issues. First, their private nature raises concerns about reproducibility and transparency, making it impossible to verify the results accurately. Moreover, the benchmark creators may selectively grant access to certain organizations [ 10 ] , creating an uneven playing field. And finally, the high cost of developing such datasets is prohibitive. For instance, HLE required a $500,000 prize pool to incentivize contributions. 3. Emphasis on final answers : Most existing benchmarks, including HLE and FrontierMath, primarily evaluate problems with single final answers. This can be misleading, as models may arrive at the correct answer through pattern recognition or brute-force enumeration, rather than genuine mathematical reasoning. Such benchmarks fall short of capturing the depth and rigor of problems found in mathematical olympiads, which often require detailed proofs and multi-step logic. Furthermore, most practical applications of LLMs in mathematics involve generating proofs or explanations, rather than simply providing final answers. Figure 1 : MathArena leaderboard. The picture shows the CMIMC competition held in March 2025. Cell color denotes the pass rate of the model on the problem out of 4 attempts. Warning signs show a possible contamination risk due to the model being released after the competition. MathArena : A new benchmark for mathematical reasoning We introduce MathArena , a dynamic publicly available benchmark which addresses the above limitations by evaluating on newly released math competitions (see Fig. 1 ). Our core insight is that recurring math competitions produce a rich source of high-quality, uncontaminated problems. These problems are pre-vetted by the competition organizers for originality, ensuring that similar problems have not appeared previously and thereby reducing contamination risk. By evaluating models on competitions occurring after model release, MathArena eliminates the risk of contamination and offers a clean, forward-looking measure of progress. Furthermore, some of the included competitions (e.g., IMO 2025) have proof-based problems that are absent in other benchmarks. Unlike private or static benchmarks, MathArena is fully transparent, reproducible, and continuously updated throughout the year as new problems become available. This enables continuous adaptation to the evolving landscape of mathematical reasoning, ensuring that the included competitions remain relevant and challenging. We implement the entire MathArena pipeline for parsing, solving, and verifying problem solutions and release the code, data, and model responses as open source. So far, we have evaluated over 50 models across seven competitions, totaling 162 problems. Our results indicate that GPT-5 , Grok 4 , and Gemini-2.5-Pro are the top-performing models on the included competitions, outperforming the top 1\% of human participants. However, we also find room for improvement on proof-based competitions, with models scoring below 40\% on the IMO 2025. This highlights the need for further research in this area. Key contributions In summary, our key contributions are as follows: • We introduce MathArena , a benchmark that leverages newly released competitions to evaluate LLMs, eliminating contamination while being fully transparent and reproducible. • We propose a scalable evaluation pipeline for parsing, solving, and verifying problems from diverse competition formats, including final-answer and proof-based problems. • We compare and thoroughly analyze the performance of state-of-the-art models on these competitions, highlighting significant progress made within the past year. 2 Related Work In this section, we discuss the key prior approaches for evaluating mathematical reasoning. Public answer-based benchmarks The most widely used benchmarks evaluate models by comparing their outputs to fixed ground-truth answers—typically numerical values or closed-form expressions. Early benchmarks such as GSM8K [ 6 ] and MATH [ 17 ] have largely been saturated by recent language models. Even more challenging competitions, like AIME 2024, have seen similar progress and are close to saturation. Omni-MATH [ 14 ] , OlympiadBench [ 16 ] , HARP [ 36 ] , and OlymMATH [ 30 ] increase difficulty by incorporating harder problems from olympiad competitions. However, sourcing problems from past competitions, which have been available online for years, makes it difficult to track progress due to data contamination risks. This concern is supported by evidence in the case of GSM8K [ 37 ] , and we confirm contamination of AIME 2024 in Section 4 . Private answer-based benchmarks FrontierMath [ 15 ] is a recently introduced private benchmark designed to be significantly more challenging, with problems that demand mathematical reasoning combined with a deep background in research-level mathematics. Similarly, Humanity’s Last Exam [ 29 ] has collected a large number of private challenging problems across dozens of subjects. While their extreme difficulty makes an interesting target for frontier models, the private nature of these benchmarks makes standardized evaluation and fair model comparison difficult. Furthermore, this difficulty level makes tracking progress challenging, particularly for open source models and models on the Pareto frontier of cost-performance. Finally, the private nature of the benchmark raises concerns about reproducibility and transparency, as access to these benchmarks has been selectively granted to certain organizations [ 10 ] . Proof-based benchmarks Another line of evaluation focuses on verifying the correctness of reasoning traces rather

Read the complete text