IndiMathBench : Autoformalizing Mathematical Reasoning Problems with a Human Touch

Param Biyani, Shashank Kirtania, Yasharth Bajpai, Sumit Gulwani, Ashish Tiwari · 2025

Read IndiMathBench : Autoformalizing Mathematical Reasoning Problems with a Human Touch by Param Biyani, Shashank Kirtania, Yasharth Bajpai, Sumit Gulwani, Ashish Tiwari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00997.txt.

Introduction

Param Biyani and four coauthors introduce IndiMathBench in a 2025 paper on machine-checked math. Autoformalization turns worded math problems into formal statements a theorem prover can check. Even with strong language models, doing this reliably is hard, and good training data is a major bottleneck.

Expert annotation takes deep skill in both math and theorem proving. IndiMathBench is a human-verified benchmark for evaluating mathematical theorem proving. It was curated with an AI-powered, human-assisted pipeline that formalizes natural language problems in the Lean language. A trusted benchmark shows whether autoformalization tools are ready for real mathematical work.

Catalog record

CREATORParam Biyani, Shashank Kirtania, Yasharth Bajpai, Sumit Gulwani, Ashish Tiwari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IndiMathBench : Autoformalizing Mathematical Reasoning Problems with a Human Touch Param Biyani, Shashank Kirtania, Yasharth Bajpai, Sumit Gulwani, Ashish Tiwari Abstract Reliable autoformalization remains challenging even in the era of large language models (LLMs). The scarcity of high-quality training data is a major bottleneck. Expert annotation requires substantial time and deep expertise in both mathematics and theorem proving. We introduce IndiMathBench , a human-verified benchmark designed to evaluate mathematical theorem proving, curated using an AI-powered human-assisted pipeline for formalizing natural language problems in Lean. IndiMathBench is composed of 312 formal Lean 4 theorems paired with their corresponding informal problem statements, sourced from Indian Mathematics Olympiads. Through category-based retrieval, iterative compiler feedback, and multi-model ensembles, our pipeline generates candidate formalizations that experts efficiently validate via an interactive dashboard with automated quality summaries. Evaluation across multiple frontier models demonstrates that autoformalization remains challenging, with substantial gaps between syntactic validity and semantic correctness, while theorem proving success rates remain low even with iterative refinement, demonstrating that IndiMathBench presents a challenging testbed for mathematical reasoning. IndiMathBench is available at https://github.com/prmbiy/IndiMathBench . 1 Introduction The formalization of mathematics, expressing informal reasoning in precise, machine-verifiable logic, has been a long-standing goal in computer science and mathematics. A related task, autoformalization, seeks to automatically translate informal mathematical statements and proofs into formal representations ( 29 ; 1 ; 6 ) . Despite advances in large language models and theorem-proving frameworks such as Lean ( 19 ) , progress remains limited by the scarcity of paired informal–formal data. Let n be a natural number. Prove that: \left\lfloor\frac{n}{1}\right\rfloor+\left\lfloor\frac{n}{2}\right\rfloor+\left\lfloor\frac{n}{3}\right\rfloor+\cdots+\left\lfloor\frac{n}{n}\right\rfloor+\left\lfloor\sqrt{n}\right\rfloor is even. ⬇ theorem inmo_2014_2 ( n : \mathbb{N} ) : Even (( Finset . sum ( Finset . range n ) fun i => \lfloor ( n : \mathbb{R} ) / (( i + 1) : \mathbb{R} ) \rfloor ) + \lfloor Real . sqrt n \rfloor ) := by sorry Figure 1: A sample problem from INMO 2014 with its formalization in Lean 4 Existing benchmarks for formal theorem proving are few and narrow in scope. The largest Lean 4 benchmarks with Olympiad-level problems, miniF2F ( 32 ) , drawn from AIME, AMC, and IMO exams, and PutnamBench ( 24 ) , from the Putnam Competition, cover only a small fraction of the available competition mathematics, totaling roughly a thousand problems. This limited scale restricts comprehensive evaluation of model generalization and reasoning capabilities. Moreover, these popular sources increasingly appear in large-scale pretraining corpora ( 11 ) , creating contamination that obscures genuine reasoning ability. Producing new benchmarks is further constrained by the high manual effort required: experts must formalize, annotate, and verify each problem in Lean, a process that is both time-consuming and resource-intensive ( 31 ) . As a result, progress in ATP evaluation depends on developing scalable, high-quality, human-verified formal benchmarks. Also, despite LLMs’ improved syntax fidelity, their semantic alignment with mathematical intent remains poor. We hypothesize that human-AI collaborative formalization can bridge this gap. To address these challenges, we introduce IndiMathBench , a human-verified benchmark for automated theorem proving built from Indian Mathematical Olympiad problems, built using a human-AI pipeline. Our benchmark contains 312 problems spanning diverse mathematical domains, geometry, algebra, number theory, and combinatorics, each paired with human-verified Lean 4 formalizations. A sample problem from IndiMathBench is shown in Figure 1 . We conducted systematic human verification of all formalizations, ensuring high-quality ground truth for reliable evaluation. Our key contributions are: • A Lean 4 benchmark for formal theorem proving, created using LLM-assisted formalization and human verification. • A systematic formalization pipeline employing category-based retrieval, a self-debug loop, with an ensemble of LLMs, and cross comparison of formalizations and a controlled study on its efficacy gains. • A VS Code Extension for improved human-AI collaboration for Lean annotations. • Evaluation of frontier general purpose and fine-tuned models on their autoformalization and proving capabilities. 2 Related Work Formal Theorem Proving Benchmarks. The evaluation of automated theorem proving systems relies critically on curated formal benchmarks, yet existing resources remain limited in scale, diversity, and representativeness. The miniF2F benchmark ( 32 ) , based on AMC, AIME, and IMO competitions, and PutnamBench ( 24 ) , derived from the Putnam Competition, have become standard datasets for assessing Olympiad competition-level mathematical reasoning in Lean 4. More recent benchmarks such as ProofNet ( 3 ) and FormalMath ( 31 ) focus on undergraduate or textbook-style mathematics, providing complementary but less challenging problem domains. While these resources have advanced systematic evaluation, they remain narrow in both mathematical coverage and cultural scope, largely reflecting Western curricula and competition traditions, where the focus is relatively less on geometry and combinatorics style problems, and more towards analysis, algebra and number theory. Recent efforts like FrontierMath ( 7 ) extend evaluation toward research-level problems but reveal extremely low model success rates, underscoring the substantial gap between current benchmarks and the diversity of human mathematical reasoning, but doesn’t support Lean. In particular, Olympiad-style geometry and combinatorics remain strikingly underrepresented across all existing Lean 4 benchmarks. Even the most widely used Olympiad datasets contain only a small number of high-difficulty geometry problems, despite these being among the hardest domains for both ATPs and LLM-guided provers. Recent efforts such as LeanGeo ( 23 ) and CombiBench ( 15 ) were motivated by this same gap, indicating that it is a well-recognized limitation in the formalization ecosystem for geometry and combinatorics respectively. This imbalance limits the ability of current benchmarks to stress-test systems on categories that are disproportionately challenging. Autoformalization. Autoformalization aims to translate informal mathematical language into machine-verifiable formal logic (like Lean). Recent advances in large language models have substantially improved the ability to generate structured formal statements directly from natural language ( 29 ; 2 ) . Despite these advances, fully automated approaches continue to struggle with semantic consistency, complex mathematical reasoning, and the domain gap between natural language and formal specifications ( 32 ; 28 ; 12 ) . Current methods struggle to preserve precise semantic details when translating natural-language problems into Lean, particularly in advanced domains, limiting the availability of reliable formal reasoning benchmarks. Another challenge is the evaluation of these autoformalization systems. Syntactic approaches like BLEU score ( 20 ) do not consider the semantic distance between generations. Recent methods like Generalized Tree Edit Distance (GTED) ( 16 ) compute structural similarity through operator trees, while Bidirectional Extended Definitional Equivalence (BEq) ( 17 ) provides neural-symbolic equivalence checking. Both report high correlation with human evaluations. Other methods include combining certainty and similarity scores through contrastive learning ( 18 ) or providing a standardized quality assessment based on human repair effort on a correction effort scale (0-4) ( 10 ) . However, scalable evaluation remains challenging, with manual expert verification being costly but necessary for establishing reliable ground truth. Human-AI Collaboration in Formalization. As human verification remains a necessity, other Lean 4 benchmarks like miniF2F , PutnamBench, and ProofNet use completely manual annotation for robust formalizations; the process is costly and non-scalable. However, the formalization of research-level mathematics increasingly relies on human-AI collaboration that combines automated translation capabilities with human expertise in both competition mathematics and theorem-proving languages ( 14 ; 5 ; 31 ) . Process-supervised approaches use formal system feedback to improve translation

Read the complete text