Mathesis: Towards Formal Theorem Proving from Natural Languages

Xuejun Yu, Jianyuan Zhong, Zijin Feng, Pengyi Zhai, Roozbeh Yousefzadeh, Wei Chong Ng, Haoxiong Liu, Ziyi Shou, Jing Xiong, Yudong Zhou, Claudia Beth Ong, Austen Jeremy Sugiarto, Yaoxi Zhang, Wai Ming Tai, Huan Cao, Dongcai Lu, Jiacheng Sun, Qiang Xu, Xin Shen, Zhenguo Li · 2025

Read Mathesis: Towards Formal Theorem Proving from Natural Languages by Xuejun Yu, Jianyuan Zhong, Zijin Feng, Pengyi Zhai, Roozbeh Yousefzadeh, Wei Chong Ng, Haoxiong Liu, Ziyi Shou, Jing Xiong, Yudong Zhou, Claudia Beth Ong, Austen Jeremy Sugiarto, Yaoxi Zhang, Wai Ming Tai, Huan Cao, Dongcai Lu, Jiacheng Sun, Qiang Xu, Xin Shen, Zhenguo Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07047.txt.

Introduction

Xuejun Yu and co-authors present a 2025 paper on a system called Mathesis. Formal theorem proving with language models usually needs the problem already written in formal code. Real problems arrive as plain words, which blocks most users. Mathesis is an end-to-end pipeline from natural language to finished proof. Its first part, the Autoformalizer, translates word problems into formal statements. Reinforcement learning trains that translator, graded by a scoring framework called LeanScorer.

The second part, the Prover, generates the formal proof itself. Together they remove the expert middleman from the pipeline's front end. AI researchers get a full-stack answer to a known bottleneck. Autoformalization quality is the hinge: a wrong formal statement makes any proof worthless. A nuanced scorer for translations is thus as important as the prover. The work targets the Lean ecosystem, where verified proofs carry lasting value.

Catalog record

CREATORXuejun Yu, Jianyuan Zhong, Zijin Feng, Pengyi Zhai, Roozbeh Yousefzadeh, Wei Chong Ng, Haoxiong Liu, Ziyi Shou, Jing Xiong, Yudong Zhou, Claudia Beth Ong, Austen Jeremy Sugiarto, Yaoxi Zhang, Wai Ming Tai, Huan Cao, Dongcai Lu, Jiacheng Sun, Qiang Xu, Xin Shen, Zhenguo Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mathesis: Towards Formal Theorem Proving from Natural Languages Xuejun Yu, Jianyuan Zhong, Zijin Feng, Pengyi Zhai, Roozbeh Yousefzadeh, Wei Chong Ng, Haoxiong Liu, Ziyi Shou, Jing Xiong, Yudong Zhou, Claudia Beth Ong, Austen Jeremy Sugiarto, Yaoxi Zhang, Wai Ming Tai, Huan Cao, Dongcai Lu, Jiacheng Sun, Qiang Xu, Xin Shen, Zhenguo Li Abstract Recent advances in large language models show strong promise for formal reasoning. However, most LLM-based theorem provers have long been constrained by the need for expert-written formal statements as inputs, limiting their applicability to real-world problems expressed in natural language. We tackle this gap with Mathesis, the first end-to-end theorem proving pipeline processing informal problem statements. It contributes Mathesis-Autoformalizer, the first autoformalizer using reinforcement learning to enhance the formalization ability of natural language problems, aided by our novel LeanScorer framework for nuanced formalization quality assessment. It also proposes a Mathesis-Prover, which generates formal proofs from the formalized statements. To evaluate the real-world applicability of end-to-end formal theorem proving, we introduce Gaokao-Formal, a benchmark of 488 complex problems from China’s national college entrance exam. Our approach is carefully designed, with a thorough study of each component. Experiments demonstrate Mathesis’s effectiveness, with the autoformalizer outperforming the best baseline by 22% in pass-rate on Gaokao-Formal. The full system surpasses other model combinations, achieving 64% accuracy on MiniF2F with pass @ @ 32 and a state-of-the-art 18% on Gaokao-Formal. 0 0 footnotetext: * Equal Contribution 0 0 footnotetext: {\dagger} Correspondence to: Zhenguo Li <[email protected]> and Xin Shen <[email protected]>. Figure 1: Performance of end-to-end theorem proving on the MiniF2F test set. Each configuration pairs an autoformalizer, which formalizes informal statements to formal ones, with a prover that generates formal proofs. Results reports pass @ 32 accuracy for all autoformalizer-prover combinations. 1 Introduction The emergence of reasoning abilities in large language models (LLMs) has opened new frontiers in automated mathematics [ 1 ] . Recent automatic theorem provers (ATPs) leverage formal verification systems, such as Lean [ 2 ] , Isabelle [ 3 ] , and Coq [ 4 ] , to enable formal reasoning. Formal reasoning starts with a clear formal problem statement, followed by the generation of mechanically verifiable proofs in formal languages. This approach ensures greater reliability and verifiability. Notable models in this field, including Deepseek-Prover-V2 [ 5 ] , Kimina-Prover [ 6 ] , and Goedel-Prover [ 7 ] , have advanced the state-of-the-art on benchmarks: MiniF2F [ 8 ] , proofNet [ 9 ] , and putnamBench [ 10 ] , where the input problem statements are already formalized by human experts. However, real-world mathematical problems are typically written in natural language, which prohibits the direct use by ATPs. Traditionally, manual formalization ensures faithful translation of problems but requires significant effort and expertise before ATPs can solve them. In this paper, we study automatic end-to-end theorem proving from natural languages. The task begins with a given natural language (NL) problem statement and entails its automatic translation into formal language (FL), followed by the generation of a formal proof. A critical step in the end-to-end task is autoformalization–the process of automatically translating informal mathematics into formal language [ 11 ] –which can significantly impact the success of proving due to errors introduced during formalization. Figure B.4 illustrates two common examples, highlighting how improper formalization can yield misleading proof successes or render problems unprovable. Despite its importance, the task remains underdeveloped in terms of benchmarking, evaluation, and the availability of a powerful autoformalizer. Existing benchmarks such as MiniF2F are not designed to evaluate autoformalization. Some human-authored formal statements in existing benchmarks subtly deviate from the original NL intent by simplification, while others exclude problems that are hard to formalize, such as those involving geometry, combinatorics [ 8 ] , and word problems [ 9 ] . A fine-grained method to assess autoformalization quality also remains lacking, hindering systematic assessment. Existing works often use MiniF2F, proofNet [ 9 ] , and Numina [ 12 ] , as benchmark datasets, relying primarily on binary compilation checks in Lean 4 and basic LLM judgments [ 7 ] , which fail to capture nuanced errors. In terms of LLM-based autoformalization training, recent methods [ 13 , 11 , 14 , 7 ] fine-tune LLMs on paired informal and formal statements (referred to as parallel statements [ 13 , 14 ] ) for higher quality, with Kimina-Autoformalizer [ 6 ] achieving state-of-the-art performance via expert iteration. However, these training approaches lack dynamic learning from direct feedback on both syntactic and semantic correctness. In this paper, we present Mathesis-Autoformalizer ( M ulti-domain A utoformalization T hrough He uristic-guided S yntact i c and S emantic Learning), the first autoformalization framework trained via online reinforcement learning (RL) combined with a novel Hierarchical Preference Optimization ( HPO ) mechanism. By incorporating Lean compilation and semantic verification into the RL reward function while learning prover preferences through HPO, Mathesis significantly enhances formalization quality and achieves state-of-the-art performance. To advance end-to-end theorem proving and establish a robust evaluation paradigm for autoformalization, we introduce Gaokao-Formal , a benchmark comprising 488 proof problems from China’s National Higher Education Entrance Examination (Gaokao) spanning diverse mathematical domains, alongside LeanScorer , a novel evaluation framework for nuanced formalization assessment. Furthermore, we develop Mathesis-Prover to support end-to-end theorem proving. Our key contributions are as follows. • We release Mathesis-Autoformalizer, pioneered through online reinforcement learning with rewards for both syntactic validity and semantic correctness, as well as a novel hierarchical preference optimization (HPO) process. It achieves a 22% improvement in pass rate on Gaokao-Formal and 5% on MiniF2F compared to the state-of-the-art baseline Kimina-Autoformalizer. • We introduce Gaokao-Formal benchmark, a challenging set of 488 Gaokao problems with parallel English and human-verified formal statements, aimed at advancing end-to-end formal reasoning research. • We propose LeanScorer, a novel evaluation method combining LLM-based analysis with Sugeno Fuzzy Integral for nuanced formalization assessment. It achieves 0.92 F1 score on Gaokao-Formal, outperforming the prior methods LLM-as-a-Judge by 7 percentage points and re-informalization by 27 points. • We study end-to-end theorem proving, a critical yet underexplored task for LLM-based formal theorem provers to solve practical mathematical problems directly from natural language. Extensive experiments shows the effectiveness of our end-to-end pipeline, achieving state-of-the-art performance 18\% on Gaokao-Formal. Informal Statement: Let S_{n} denote the sum of the first n terms of the sequence a_{n} . Given that \frac{2S_{n}}{n}+n=2a_{n}+1 , prove that \{a_{n}\} is an arithmetic sequence. Two Formal Statement Cases: theorem case_goal_as_assumption_error (a : \mathbb{N}\rightarrow\mathbb{R} ) (ha : \exists d, \forall n, a (n + 1) = a n + d) \Rightarrow Erroneously includes the desired goal in assumptions (h : \forall n, 2 * ( \sum i in Finset.range n, a i) / n + n = 2 * a n + 1) : \exists d, \forall n, a (n + 1) = a n + d := by sorry theorem case_definition_error (a : \mathbb{N}\rightarrow\mathbb{R} ) (S : \mathbb{N}\rightarrow\mathbb{R} ) (hS : \forall (n : \mathbb{N} ), n \geq 1, S n = \sum k in Finset.range n, a k) \Rightarrow \sum k in Finset.Icc 1 n, a k (h : \forall (n : \mathbb{N} ), n \geq 1, 2 * S n / (n : \mathbb{R} ) + (n : \mathbb{R} ) =

Read the complete text