TaoBench : Do Automated Theorem Prover LLMs Generalize Beyond MathLib?
Alexander K Taylor, Junyi Zhang, Ethan Ji, Vigyan Sahai, Haikang Deng, Yuanzhou Chen, Yifan Yuan, Di Wu, Jia-Chen Gu, Kai-Wei Chang, Nanyun Peng, Amit Sahai, Wei Wang · 2026
Read TaoBench : Do Automated Theorem Prover LLMs Generalize Beyond MathLib? by Alexander K Taylor, Junyi Zhang, Ethan Ji, Vigyan Sahai, Haikang Deng, Yuanzhou Chen, Yifan Yuan, Di Wu, Jia-Chen Gu, Kai-Wei Chang, Nanyun Peng, Amit Sahai, Wei Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12744.txt.
Introduction
Alexander Taylor and co-authors test whether theorem prover language models generalize beyond MathLib. Most prover benchmarks use MathLib definitions, which biases training and scores. Their TaoBench comes from Tao's Analysis I, where core concepts are built from scratch. Each problem also gets an equivalent MathLib version for direct comparison. Top provers did well in MathLib form but dropped by roughly 26% on the Tao form. The bottleneck is thus framework shift, not task difficulty.
Catalog record
| CREATOR | Alexander K Taylor, Junyi Zhang, Ethan Ji, Vigyan Sahai, Haikang Deng, Yuanzhou Chen, Yifan Yuan, Di Wu, Jia-Chen Gu, Kai-Wei Chang, Nanyun Peng, Amit Sahai, Wei Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62515 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12744
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TaoBench : Do Automated Theorem Prover LLMs Generalize Beyond MathLib? Alexander K Taylor, Junyi Zhang, Ethan Ji, Vigyan Sahai, Haikang Deng, Yuanzhou Chen, Yifan Yuan, Di Wu, Jia-Chen Gu, Kai-Wei Chang, Nanyun Peng, Amit Sahai, Wei Wang Abstract Automated theorem proving (ATP) benchmarks largely consist of problems formalized in MathLib, so current ATP training and evaluation are heavily biased toward MathLib’s definitional framework. However, frontier mathematics is often exploratory and prototype-heavy, relying on bespoke constructions that deviate from standard libraries. In this work, we evaluate the robustness of current ATP systems when applied to a novel definitional framework, specifically examining the performance gap between standard library problems and bespoke mathematical constructions. We introduce TaoBench , an undergraduate-level benchmark derived from Terence Tao’s Analysis I , which formalizes analysis by constructing core mathematical concepts from scratch, without relying on standard Mathlib definitions, as well as by mixing from-scratch and MathLib constructions. For fair evaluation, we build an agentic pipeline that automatically extracts a compilable, self-contained local environment for each problem. To isolate the effect of definitional frameworks, we additionally translate every problem into a mathematically equivalent Mathlib formulation, yielding paired TaoBenchMathLib statements for direct comparison. While state-of-the-art ATP models perform capably within the MathLib framework, performance drops by an average of roughly 26% on the definitionally equivalent Tao formulation. This indicates that the main bottleneck is limited generalization across definitional frameworks rather than task difficulty. TaoBench thus highlights a gap between benchmark performance and applicability, and provides a concrete foundation for developing and testing provers better aligned with research mathematics. † † affiliationnotice: * Equal contribution. \dagger Senior authors served as equal advisors and are listed alphabetically. Website: https://TaoBench.github.io 1 Introduction Figure 1 : Tao’s Analysis I constructs several mathematical structures in LEAN from scratch, often using formulations that diverge significantly from their equivalents in MathLib. TaoBench provides mathematically equivalent MathLib counterparts, and reveals a substantial degradation in model performance when navigating Tao’s specific definitional framework. Figure 2 : Agentic framework for automatically extracting a compilable, self-contained local environment from a formalized textbook. The construction of references and dependencies for each exercise is performed using the JiXia tool. An agentic harness, equipped with a file-lookup tool and a Lean verifier, then iteratively builds a compilable, self-contained local environment. Applying large language reasoning models (LLMs) to formally verifiable programming languages like Lean 4 has emerged as a critical frontier for advancing the mathematical reasoning abilities of AI. Accordingly, several recent works have focused on training specialized automated theorem proving (ATP) LLMs to reason in Lean and have achieved high solve rates on formalized benchmarks, such as MiniF2F ( Lin et al. 2025c ; Ren et al. 2025 ; Wang et al. 2025 ; Zheng et al. 2022 ) . Current datasets for training and evaluating ATP models ( Wang et al. 2025 ; Xin et al. 2024 ; Ren et al. 2025 ; Lin et al. 2025b ; Lin et al. 2025c ; Xin et al. 2025 ) are built almost entirely within the definitional framework of MathLib ( Zheng et al. 2022 ; Tsoukalas et al. 2024 ; Azerbayev et al. 2023 ; AI-MO 2025 ; Gao et al. 2025 ) , the largest and de facto standard mathematical library for Lean 4 ( Community 2025 ) . However, exploratory mathematical research often ventures beyond the boundaries of MathLib and requires problem-specific definitions and constructions to prototype concepts and guide proofs ( Tao 2023 ; Degenne et al. 2025 ; Harbor-Collins & Hulse 2025 ) . This introduces a distribution shift between the definitional framework used in public SOTA ATP model training and the frameworks to which they are applied. As such, current benchmarks offer limited insight into whether ATP models can generalize across such definitional frameworks. We introduce TaoBench , the first benchmark designed to measure ATP robustness beyond MathLib by targeting generalization to a novel definitional framework. TaoBench is grounded in 150 exercises from Terence Tao’s Lean formalization of Analysis I ( Tao 2025a ) . This formalization develops core concepts of analysis from first principles rather than relying on standard MathLib definitions, and in some cases combines from-scratch constructions with MathLib components. To ensure fair evaluation, we develop an agentic extraction pipeline that automatically packages each problem with a self-contained, compilable local environment extracted from the formalized textbook, requiring no external imports or references. Moreover, we construct a paired control: for each Tao-style problem, we construct a mathematically equivalent MathLib formulation, which we title TaoBenchMathLib . This paired representation isolates the effect of definitional frameworks from the difficulty of each problem, enabling a direct evaluation of impact on model performance. We evaluate several state-of-the-art ATP models and find that, while these models solve analysis exercises reliably on TaoBenchMathLib , performance drops by approximately 26% on average on the same problems in TaoBench . The observed gap implies that training in the MathLib definitional framework does not reliably transfer to unfamiliar definitional formalisms, even when the underlying mathematics is equivalent. We believe TaoBench exposes a key hurdle towards applying public SOTA ATP models to research mathematics, and provides a concrete testbed for aligning future prover models with real-world mathematical workflows. Our main contributions are as follows: (1) We propose TaoBench , the first Lean benchmark targeting generalization beyond MathLib via a new definitional framework. (2) We provide TaoBenchMathLib , mathematically equivalent MathLib translations of the problems in TaoBench and identify a substantial generalization gap in existing ATP models. (3) We introduce two pipelines that together automatically extract self-contained, compilable local environments for problems from a large, formalized project and provide equivalent formalizations in MathLib syntax, which enables scalable, high-fidelity generation of mathematically grounded training data for future models. 2 Benchmark Construction Figure 3 : Translation pipeline for converting exercises from Tao’s definitional framework to MathLib’s. The rewriting and equivalence checking stages ensure the compilability and mathematical equivalence of TaoBenchMathLib , followed by expert verification for additional assurance. Research in mathematics often requires introducing novel axioms, definitions, and constructs that are not yet in formalized Lean libraries. Characterizing the efficacy of public ATP models in an exploratory research environment requires formalized definitional frameworks distinct from MathLib, but with enough conceptual overlap to allow models to transfer their proving capacity. Thus, evaluating the robustness of ATP models to these novel environments requires grounding evaluation in a formal framework distinct from MathLib, while preserving enough conceptual overlap to test the transfer of existing proving capabilities. Tao’s Lean formalization of Analysis I defines many standard mathematical objects using custom inductive types or structures for pedagogical clarity, adopts naming schemes that differ from MathLib conventions, and organizes definitions within section- and subsection-level namespaces. Additionally, some concepts are defined as fields of larger structures rather than as standalone definitions. As a result, it becomes possible that even mathematically elementary statements could become out-of-distribution to models trained primarily on MathLib-style Lean code. To enable a fair and interpretable evaluation of model capabilities on undergraduate-level analysis, we construct twin versions of the benchmark. The first version, TaoBench , evaluates models directly on formalized textbook problems and provides all relevant definitions and references necessary to correctly state the problem in a fully compilable, in-context environment. This setting is