Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions

Lan Zhang, Marco Valentino, André Freitas · 2025

Read Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions by Lan Zhang, Marco Valentino, André Freitas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12065.txt.

Introduction

Lan Zhang, Marco Valentino, and André Freitas test language models at turning plain math into formal code, in a paper published in 2025. The task, autoformalization, would let computers check proofs, but models are usually tested on tidy contest problems. Real definitions from papers and encyclopedias are messier and may expose weaker spots.

The team built two new test sets of definitions, drawn from Wikipedia and from arXiv papers, and asked models to render them in the Isabelle proof system. Definitions proved harder than older benchmarks. Models struggled to fix their own errors when a proof tool pushed back, and to anchor terms in formal libraries. Two aids helped: structured rounds of feedback raised self-correction, and supplying library context cut undefined-term errors sharply.

Catalog record

CREATORLan Zhang, Marco Valentino, André Freitas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions Lan Zhang, Marco Valentino, André Freitas Abstract Thanks to their linguistic capabilities, LLMs offer an opportunity to bridge the gap between informal mathematics and formal languages through autoformalization . However, it is still unclear how well LLMs generalize to sophisticated and naturally occurring mathematical statements. To address this gap, we investigate the task of autoformalizing real-world mathematical definitions : a critical component of mathematical discourse. Specifically, we introduce two novel resources for autoformalization, collecting definitions from Wikipedia (Def_Wiki) and arXiv papers (Def_ArXiv). We then systematically evaluate a range of LLMs, analyzing their ability to formalize definitions into Isabelle/HOL. Furthermore, we investigate strategies to enhance LLMs’ performance including refinement through external feedback from Proof Assistants, and formal definition grounding , where we augment LLMs’ formalizations through relevant contextual elements from formal mathematical libraries. Our findings reveal that definitions present a greater challenge compared to existing benchmarks, such as miniF2F. In particular, we found that LLMs still struggle with self-correction, and aligning with relevant mathematical libraries. At the same time, structured refinement methods and definition grounding strategies yield notable improvements of up to 16% on self-correction capabilities and 43% on the reduction of undefined errors, highlighting promising directions for enhancing LLM-based autoformalization in real-world scenarios. 1 1 1 Code and datasets are available at https://github.com/lanzhang128/definition_autoformalization 1 Introduction Large Language Models (LLMs) have demonstrated remarkable potential in assisting with mathematical reasoning on different downstream tasks ( Wei et al., 2022 ; Meadows et al., 2023 ; Meadows et al., 2024 ; Valentino et al., 2022 ; Lu et al., 2023 ; Meadows and Freitas, 2023 ; Mishra et al., 2022a ; Ferreira et al., 2022 ; Ferreira and Freitas, 2020 ; Welleck et al., 2021 ; Mishra et al., 2022b ; Petersen et al., 2023 ) . In the context of mathematics, formal languages play a crucial role by providing a precise, logic-based framework for verifying the correctness and logical validity of mathematical statements and proofs ( Kaliszyk and Rabe, 2020 ) . Figure 1: Can LLMs formalize complex mathematical statements? This paper investigates the task of translating real-world mathematical definitions into a formal language. We introduce a new resource collecting definitions from Wikipedia and ArXiv papers, exploring different strategies for autoformalization through the interaction between LLMs and Proof Assistants . Consequently, one promising application of LLMs is autoformalization , the task of translating informal statements into formal languages ( Wu et al., 2022 ) . Given their advanced linguistic and inferential capabilities, LLMs offer an opportunity to bridge the gap between informal mathematics, natural language, and machine-verifiable logic, potentially streamlining and scaling the process of formal mathematical reasoning ( Jiang et al., 2023 ; Tarrach et al., 2024 ) . The task of autoformalization has collected increasing attention in recent years, leading to the development of benchmarks and evaluation methodologies ( Azerbayev et al., 2023 ; Zhang et al., 2024 ; Li et al., 2024 ) . Despite this progress, however, existing benchmarks for autoformalization often focus on relatively simple mathematical problems, limiting our understanding of how well LLMs generalize to more sophisticated and naturally occurring mathematical statements. To address this gap, this paper investigates the task of autoformalizing mathematical definitions , a critical component of mathematical discourse ( Moschkovich, 2003 ) . Definitions serve as foundational building blocks in mathematical reasoning, yet they are often intricate, context-dependent, and thus difficult to formalize. Evaluating LLMs on this subset of mathematical statements, therefore, allows for assessing their ability to formally represent fine-grained mathematical concepts, highlighting persisting challenges and limitations for real-world applications. Specifically, this paper introduces two new benchmarks for autoformalization by collecting real-world mathematical definitions into two distinct resources: (1) Def_Wiki , including definitions extracted from Wikipedia articles, and (2) Def_ArXiv , including definitions collected from machine learning research papers. Using these resources, we first evaluate LLMs in a zero-shot setting, analyzing their ability to translate definitions into Isabelle/HOL ( Nipkow et al., 2002 ) . Furthermore, to address observed limitations, we investigate two key strategies to enhance performance: (1) Refinement via external feedback , investigating the self-correction capabilities of LLMs by incorporating errors found by the supporting Proof Assistant. In particular, we show that while LLMs exhibit limited ability to refine outputs based on binary feedback (error vs. non-error), a more structured categorical refinement implemented via additional instructional constraints can improve performance. (2) Formal definition grounding . Many mathematical definitions require references to formal objects in external mathematical libraries. To augment autoformalization from LLMs, we investigate the impact of introducing additional contextual control mechanisms, which add contextual elements from formal mathematical libraries as auxiliary premises. Overall, our findings reveal that the proposed benchmarks present a greater challenge compared to existing autoformalization datasets, such as miniF2F ( Zheng et al., 2022 ) . In particular, LLMs struggle with self-correction and particularly with incorporating relevant mathematical libraries as preambles. Proposed structured refinement methods and definition grounding strategies both deliver notable improvements, highlighting promising directions for enhancing LLM-based autoformalization in real-world scenarios. Our contributions can be summarized as follows: 1. We introduce and release two novel datasets for autoformalization: Def_Wiki (definitions from Wikipedia) and Def_ArXiv (definitions from research papers on arXiv), designed to assess LLMs performance on complex, real-world mathematical definitions. 2. We perform a comprehensive error analysis on Isabelle/HOL, identifying key failures in formalizations generated by LLMs spanning across different families, including GPT-4o ( OpenAI, 2024b ) , Llama3 ( Llama Team, 2024 ) and DeepSeekMath ( Shao et al., 2024 ) . 3. We investigate refinement-based strategies, including structured feedback mechanisms from Proof Assistants and instruction-based categorical refinements. 4. We explore the role of formal definition grounding, investigating how the inclusion of relevant mathematical libraries impacts the ability of LLMs to connect the formalized statements with contextual mathematical elements and relevant premises. miniF2F Def_Wiki Def_ArXiv 1. Suppose that \sec x+\tan x=\frac{22}{7} and that \csc x+\cot x=\frac{m}{n}, where \frac{m}{n} is in lowest terms. Find m+n. Show that it is 044. 2. What is the sum of the two values of x for which (x+3)^{2}=121 ? Show that it is -6. 3. The product of two positive whole numbers is 2005. If neither number is 1, what is the sum of the two numbers? Show that it is 406. 4. The expression 10x^{2}-x-24 can be written as (Ax-8)(Bx+3), where A and B are integers. What is AB+B ? Show that it is 12. 1. Definition of Rademacher Complexity: Given a set A\subseteq\mathbb{R}^{m} , the Rademacher complexity of A is defined as follows: \operatorname{Rad}(A):=\frac{1}{m}\mathbb{E}_{\sigma}\left[\sup_{a\in A}\sum_{i=1}^{m}\sigma_{i}a_{i}\right] where \sigma_{1},\sigma_{2},\dots,\sigma_{m} are independent random variables drawn from the Rademacher distribution (i.e. \Pr(\sigma_{i}=+1)=\Pr(\sigma_{i}=-1)=1/2 for i=1,2,\dots,m ), and a=(a_{1},\dots,a_{m}) . 2. Definition of Polynomial Kernel: For degree- d polynomials, the polynomial kernel is defined as K(\mathbf{x},\mathbf{y})=(\mathbf{x}^{\mathsf{T}}\mathbf{y}+c)^{d} where \mathbf{x} and \mathbf{y} are vectors of size n in the input space, i.e. vectors of features computed from training or test samples and c\geq 0 is a free parameter trading off the influence of higher-order versus lower-order terms in the polynomial. 1. Definition of Covering Number: Given a metric space (\mathcal{S},\rho) , and a subset \tilde{\mathcal{S}}\subset\mathcal{S} , we say that a subset \hat{\mathcal{S}} of \tilde{\mathcal{S}} is a \epsilon -cover of \tilde{\mathcal{S}} , if \forall\tilde{s}\in\tilde{\mathcal{S}}

Read the complete text