MathAtlas: A Benchmark for Autoformalization in the Wild

Nilay Patel [email protected] Noah Arias [email protected] Davit Babayan [email protected] Victoria Cochran [email protected] Timothy Libman [email protected] Hafsah Mahmood [email protected] Liam McCarty [email protected] Soli Munoz [email protected] Laurel Willey [email protected], Jeffrey Flanigan [email protected] University of California, Santa Cruz · 2026

Read MathAtlas: A Benchmark for Autoformalization in the Wild by Nilay Patel [email protected] Noah Arias [email protected] Davit Babayan [email protected] Victoria Cochran [email protected] Timothy Libman [email protected] Hafsah Mahmood [email protected] Liam McCarty [email protected] Soli Munoz [email protected] Laurel Willey [email protected], Jeffrey Flanigan [email protected] University of California, Santa Cruz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14061.txt.

Introduction

Patel and nine co-authors present MathAtlas, the first large-scale autoformalization benchmark drawn from real graduate-level mathematics. Past benchmarks center on olympiad or undergraduate problems, leaving graduate and research math largely untested. MathAtlas holds about 52,000 theorems, definitions, exercises, examples and proofs pulled from 103 graduate textbooks. A math dependency graph with about 178,000 relations makes it the first benchmark of its kind to support dependency-aware systems, yet strong baselines top out near ten percent on theorem statements.

Catalog record

CREATORNilay Patel [email protected] Noah Arias [email protected] Davit Babayan [email protected] Victoria Cochran [email protected] Timothy Libman [email protected] Hafsah Mahmood [email protected] Liam McCarty [email protected] Soli Munoz [email protected] Laurel Willey [email protected], Jeffrey Flanigan [email protected] University of California, Santa Cruz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52552
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MathAtlas: A Benchmark for Autoformalization in the Wild Nilay Patel [email protected] Noah Arias [email protected] Davit Babayan [email protected] Victoria Cochran [email protected] Timothy Libman [email protected] Hafsah Mahmood [email protected] Liam McCarty [email protected] Soli Munoz [email protected] Laurel Willey [email protected], Jeffrey Flanigan [email protected] University of California, Santa Cruz Abstract Current autoformalization benchmarks are largely focused on olympiad or undergraduate mathematics, while graduate and research-level mathematics remains underexplored. In this paper, we introduce MathAtlas , the first large-scale autoformalization benchmark of in the wild graduate-level mathematics, containing ∼ 52 ​ k \sim 52k theorems, definitions, exercises, examples, and proofs extracted from 103 graduate mathematics textbooks. MathAtlas is enriched with a mathematical dependency graph containing ∼ 178 ​ k \sim 178k relations, and is the first autoformalization benchmark to include such relations, facilitating evaluation and development of dependency-aware autoformalization systems. Our extensive experiments show that MathAtlas is high quality but extremely challenging: strong baselines achieve at most 9.8% correctness on theorem statements and 16.7% on definitions. Furthermore, we find performance of state-of-the-art models degrades substantially with dependency depth: on MA-Hard , a subset of 700 entities with the deepest dependency trees, the best model achieves only 2.6% correctness for autoformalization on this challenging dataset. We release MathAtlas to the community as a benchmark set for large-scale autoformalization of graduate-level mathematics in the wild . 1 Introduction The popularity of formal mathematics (computer verifiable mathematics) within the mathematics community has grown dramatically over recent years [ 30 , 32 , 15 ] . Together with advances in AI, this has caused a surge of interest in autoformalization [ 10 , 36 , 13 , 2 , 18 ] , the task of automatically converting natural language math to a formal verifiable language such as Lean [ 24 ] . Figure 1 : An entity from MathAtlas together with a subgraph of its dependency tree. In yellow (bottom), we have a target theorem statement to be formalized (Proposition 15). Proposition 15 contains three “object references” to previously defined mathematical concepts: Dedeking ring, prime ideals, and principal ideal ring, which are dependencies needed to correctly formalize the theorem. These objects each have their own dependency tree (shown with a red border), and so on. Formalizations of objects possibly already exist (e.g., in Mathlib), or they need to be synthesized before the target statement can be formalized. Until recently, autoformalization work has been largely focused on simpler settings such as olympiad or undergraduate mathematics [ 2 , 36 , 41 , 9 , 1 , 12 , 19 ] , where mathematical statements require limited prerequisite material to properly formalize, much of which already has been formalized by humans in libraries such as Mathlib [ 22 ] . However, in more advanced mathematics, autoformalization becomes increasingly complex. Advanced mathematics has a large amount of prerequisite theory, much of which has yet to be formalized. This means an autoformalization system must first retrieve or produce the necessary dependent theory, correctly formalize it, and then formalize the original target statement. Additionally, existing benchmarks typically focus on theorem statements, and ignore formalizations of mathematical definitions . Systems which can only autoformalize theorem statements are limited in their scope by the level of existing theory because they cannot formalize theorems using math concepts they do not already have definitions for. Thus, creating scalable autoformalization systems necessitates the study of definition autoformalization, for which large-scale real-world benchmarks are lacking [ 39 , 34 ] . To comprehensively evaluate the performance of autoformalization methods in higher-level mathematics, we present MathAtlas 1 1 1 https://huggingface.co/datasets/MathAtlas/MathAtlas , a large-scale, graduate-level benchmark designed to measure autoformalization performance on mathematics in the wild. MathAtlas contains \sim 51k informal entities, divided into statements (theorems, examples, and exercises), definitions, and proofs, which are extracted from 103 graduate math textbooks. Entities in MathAtlas span 87 areas of mathematics, including real analysis, differential geometry, category theory, topology, quantum groups, among others. Some entities in MathAtlas are formalizable on top of Mathlib, but many require additional formal theory which may not currently exist. To facilitate access to these required dependencies, we enrich each entity with long-range dependency relations, local variable references, and local context (see Figure 1 ). MathAtlas is the first large-scale, in the wild autoformalization benchmark of graduate-level mathematics, the first to include rich contextual and dependency data, and the first to annotate their inter-entity dependencies. Our extensive experiments show that MathAtlas is very challenging: strong baseline methods have low overall autoformalization correctness (10.5% on statements and 20.3% on definitions) on MathAtlas . Furthermore, we find this performance is correlated with an entity’s dependency depth : our strongest baseline has only 2.6% overall correctness on MA-Hard , a subset of 700 entities with the deepest dependency trees. Finally, we investigate how well existing semantic faithfulness metrics for autoformalization generalize to MathAtlas . We present MA-Align 2 2 2 https://huggingface.co/datasets/MathAtlas/MA-Align , a binary classification benchmark of 200 entities from MathAtlas . Unlike prior benchmarks such as ConsistencyCheck [ 6 ] and CriticLeanBench [ 28 ] , MA-Align contains both labeled examples of theorem statements and definitions, and covers a wide variety of mathematics at the graduate level. Our experiments show that prior baselines have significant room for improvement on MA-Align . We organize the rest of the paper as follows: In § 2 , we present MathAtlas , its construction, contents and quality. We then discuss the benchmark’s autoformalization metrics ( § 3 ), before moving on to our autoformalization experiments and an analysis of the results ( § 4 ). Finally, we discuss related works ( § 5 ), and conclude ( § 6 ). 2 MathAtlas Benchmark Definition 2.7.10 (Invariant bilinear form).: Let V=V_{\bar{0}}\oplus V_{\bar{1}} be a superspace and carrying a representation of a Lie algebra \mathfrak{g} . A bilinear form (\cdot|\cdot) is said invariant if, for any homogeneous element X\in\mathfrak{g} and v,w\in V , (Xv|w)+p(v,X)(v|Xw)=0 . Moreover, if V is the representation space of the adjoint representation, then the bilinear form is said adjoint invariant. Corollary 3.A.24 : Let X be a compact Kahler manifold. Then the inclusion i:(\mathcal{A}^{*}(X)^{c},d)\subset(\mathcal{A}^{*}(X),d) is a dga-quasi-isomorphism. Example 5.1.14 .: Find two fundamental matrices for the linear homogeneous system in Example 5.1.11. Figure 2: Several entities from MathAtlas of various types: (1) a definition entity which defines two terms: “invariant” and “adjoint invariant”. (2) a theorem statement of a corollary from complex geometry. (3) an example from an ODE textbook. MathAtlas is constructed automatically from textbook documents. We discuss the entity and relation types in MathAtlas , and then discuss our entity and relation extraction pipeline. We also measure the quality of MathAtlas with a human evaluation. 2.1 Structure of MathAtlas The structure of MathAtlas is as follows, with some relevant statistics. Entities We use the term entity to mean a span of informal text which can be formalized, such as a theorem in a textbook. In MathAtlas , entities are either definitions, theorems, proofs, examples, or exercises (see Figure 2 ). Entities can, but don’t always, have identifiers , such as “Theorem 4.3.” These identifiers are sometimes referenced by other entities (see Figure 1 ). Importantly, entities sometimes contain multiple formalizable items. For example, in Figure 2 , Definition 2.7.10 actually defines two objects, and so

Read the complete text